"Imagine if you suddenly learned that the people, the places, the moments most important to you were not gone, not dead, but
worse: had never been. What kind of hell would that be?"
What isn't remembered never happened. Memory is merely a record. You just need to re-write that record.
Lister: We don't exist here anymore! Kryten: Actually sir, we don't ever have existed here anymore, but this is hardly the time to be conjugating temporal verbs in the past impossible never tense.
When the shadow people take you, they take all of you. Past, present, future. They reach back and rip you out of the past, like tearing up a plant by the roots. If a man kills you, you're gone. If a shadow man kills you, you never were.
Per Degaton: His name was Jason Huntington. He was your best friend in grade school. He was the only other boy you could relate to. Jason loved the pulp magazines and the science-fiction pictures. He kept you from feeling isolated. From being without friends. I think I'll take him too.
Per Degaton: Thanks to the Red Morgue, your friendship with Jason Huntington never existed.
Hourman: ARRGH! (spasms and collapses)
Jason died when he fell - I mean when he was pushed - from his crib. Your childhood is becoming quite bleak. Do you even remember him? Or do you just recall the loneliness? The empty void during your school years. Who else, Hourman? Perhaps the first girl you kissed? Shall I erase her from your timeline too?
: Computer, read the entire crew roster for the Enterprise
: Dr. Beverly Crusher. Crusher
: Have I always been the only member of the crew on the Starship Enterprise
: Affirmative. Crusher
: If this were a bad dream, would you tell me? Computer
: That is not a valid question. Crusher
: Like hell it's not.
: This is what it means to be a torch
Yuji: And I will vanish in the same way?
Shana: Yes. The remaining scum will eventually burn out. Nobody remembers it existed, and nothing remains. That's why all continues smoothly.
If you return your memories to him, you'll disappear. And since everything about you was built on those memories...no one will remember you when you're gone. There won't be any "you" to remember. I can't save you, Xion—even a memory of you.