Only the five main characters are actual party members - the others are uncontrollable guest stars.
The party is currently on a stealthy mission, where moving around in such large numbers would draw too much undue attention. Bob would then have to determine who would be best to come along.
Inverted: There is an arbitrary minimum number of characters you must have with you at all times.
You are told in the tutorial to put 5 units on the field and no more, but you actually have no limit...
The heroes are purposely holding back, only bringing along everyone who is absolutely necessary for the quest. Under special circumstances, however, more people can be brought into the active party.
Double Subverted: ...However, you only get a handful of units to begin with, so it's functionally the same as a headcount limit.
Parodied: Bob hires a bureaucrat to join the party so as to accurately determine the most efficient makeup... but the otherwise-worthless bureaucrat can only do this when in the party.
Zig Zagged: The game allows the player to use all of their characters at once; however, several characters enter and leave the party due to various plot-related events.
Averted: The game allows you to use all your units at once.
All attacks are loaded with special effects, and characters are extremely well-detailed. Having all characters in battle at once would put a serious strain on hardware.
The game is being played on a portable unit and only three characters can fit on the screen.
Lampshaded: "You'd think that 5 men fighting 5 men wouldn't count as a war, would you?"
Invoked: Bob shoos away most of the party, because it's traditional to only use a handful.
Defied: "I'll be damned if a single one of us doesn't help in the battle! Alice, get back here!"
Discussed: "This isn't a game, soldier. You don't need to spawn more overlords. Just send in the whole army."
Conversed: "Why do you suppose they always go in four at a time?" "Game balance, probably. Maybe it's just less complicated that way?"
The army is weak, due to using no more than 4 men, and is defeated quickly.
The 4 man army is easily capable of wiping out an entire enemy regiment, but other regiments take different routes and destroy the poorly defended capital while the 4 man army is out.
The four-man army is insanely powerful thanks to Applied Phlebotinum that the enemy does not have access to, but is in very limited supply.
While Bob and his team are off doing their thing, the unused party members have their own jobs to do in the meantime. By the time Bob gets back to base, they'll have gathered intel or useful equipment that they wouldn't have had the chance to if they went with Bob.
Bob's mission is centered around infiltration and sabotage, rather than outright confrontation. The four of them are enough to handle the handful of patrols they encounter, and they are able to do massive damage to the enemy's lair and military capacity.
Played For Laughs: During battles, unused characters are on the sidelines, having tea and playing badminton. In fact, they're always a little disappointed when duty calls.
Played For Drama: Bob lost a comrade once when he committed too many of his troops to one fight. He's learning to fight more conservatively, and to trust himself to heavier responsibilities vis a vis human life.