First verify the background and stand firm on “I want to stay in a group of five.”
If it were me, I’d first quietly ask the side that’s excluding others, “Recently I’ve only invited ◯◯, why is that?” If the reason is weak, I’d say, “I like being in a group of five, so let’s stop doing that,” and put a stop to it there.