El Sandifer of Tardis Eruditorum has an explanation as to why only Ben was brainwashed. Ben's working class background meant that he would have been the only one familiar with the holiday camp atmosphere of the colony. He would likely have been to holiday camps as a child. Polly, on the other hand, was probably too middle-class for holiday camps.
Characters repeatedly refer to the Macra as "insects" when they're clearly giant crabs. But considering how long these colonists may have been gone from Earth, it's entirely possible most people there aren't familiar with crabs, only insects. They're just using their only available frame of reference to describe "large, multi-legged arthropods".
