Oddly enough, Mineta's expulsion in this fic actually has a tangible effect on Izuku's fight with Muscular. In canon, Izuku is the only one without a partner for the test of courage, as there is an odd number of students participating. In this fic, however, Mineta's expulsion makes that number even, and as such Izuku ends up paired with Yaoyorozu. This puts him in the forest during the gas attack, and allows him to ask Momo to not only make him a mask, but to also make one for Kouta. In turn, this is what allows Izuku to lead Muscular into the gas after blinding him, allowing Izuku to end the fight with all of his limbs intact. It's not the only reason the fight with Muscular ends that way, but it acts as an interesting point of divergence from canon as a result - and also shows that the expulsion wasn't simply a throwaway act to get Mineta out of the fic.
So when All for One says he could take Izuku's hidden Quirk if he actually did it, it would have let to a bunch of angry ghosts who could now affect him. The would not need Izuku's Quirk to be boosted by One of All to let them have their revenge on All for One.
How is this Fridge Logic? All For One has no idea what Izuku's hidden quirk actually is. The only outward sign of Izuku's Ghost Sense is his reflective eyeshine. And they didn't need Izuku's quirk to get their revenge on him in the first place - they're all waiting for him to die. Ghosts can hurt other ghosts without Izuku's assistance.
Poltergeists are ghosts that suffer from a great deal of emotional stress (usually related to their deaths) and are often the most powerful and violent ghosts around. The most striking feature is that poltergeists have pitch-black eyes, while normal ghosts have dead white eyes. That's not the Fridge Horror part, however. The Fridge Horror part is when Izuku realizes that all the ghosts of former Noumu are permanently stuck in black-eyed poltergeist mode. Including Rei, who remains black-eyed even when feeling calm and cheerful. Just how traumatizing were those experiments?
Izuku knows the most common phrase that kidnappers say to other adults so as to not raise alarms. He even calls Mirio out when the older student nearly falls for Overhaul's story, and then explains to Nighteye that he doesn't want to listen to another child tell him where to find their body. How many children has Izuku, a 16-year-old, already done this for?
Better question: When was his first time? It's clear he's done it multiple times, enough to be weary of it by the time he meets Eri, but how young and relatively innocent was he when he met that first ghost kid?