True, the Total war games and even some more modern RTS games do use entire formations as a single "unit", as well as allow you to divide or meld them together. I suppose what I actually meant was how in reality you had little control over said formations or other large groups in that era, so they weren't very tactical or even used often. Instead, tactics such as raiding parties, small groups of skirmishers, and total clusterfucks were more common that actual, proper formation units in that era.
How about this for an RTS idea? Instead of making an epic battle, focus the game on small raiding parties, no bigger than 20-40 people, maybe even less. The battles would take place in forests, towns, castles, and rocky valleys: bascially, places where there's lots of cover, and lots of hiding places. The units would be much smaller, such as 3-4 men per team, and each would carry different weapons or specialize in certain tasks, such as archery, swordsmanship, jousting, blunt weapons, fire-based weapons, etc. The idea would be that the raiding party would have to either destroy or capture the defenders objective, or wipe all of them out first.
The combat would be similar to Dawn of War II, but instead of HP or mana, units would have stamina, which gradually depletes when in combat, running, being demoralized, or taking hits. When drained, the unit becomes vunerable to being killed, but as long as they have some stamina left they can defend against attacks, use abilities, and survive injuries. Pinning in enemies and surrounding them causes them to lose stamina faster, since they have to block and focus on enemies from multiple directions, and getting attacked from behind while focused on another target means an instant kill. Thus, players would be encouraged to try and trap enemy units into areas where it's easy to flank and tire the enemy units while at the same time make sure the enemy doesn't sneak up behind your attacking troops either.
Would you believe I never fully watched the original Indiana Jones trilogy? I gotta correct that someday.