Around here, however, we don’t look backwards for very long. We keep moving forward, opening up new doors and doing new things… and curiosity keeps leading us down new paths.
That moment when Lewis's hat comes off and the Robinsons realize that they can't keep him. They also can't tell him why. So Lewis gets rejected by the first people who have accepted him without question and he doesn't understand why.
Wilbur blurting out to his mother that he never planned to take Lewis back to find his mom, right in front of him no less. The opening music during that part is so heartbreaking that it makes one want to give Lewis a hug when he hides behind a topiary and begins crying.
Wilbur fading from existence because of Doris tampering with the past.
Also the scene early on when he's angrily explaining to Mildred why he's a lost cause: he's already failed over 40 adoption interviews (and was keeping track) due to prospective couples being put off by his zealous enthusiasm for science. To make matters worse, he's almost thirteen, and he knows it's damn near impossible for teenagers to get adopted. At this point, he's given up on finding a new family and has resigned to just aging out of the system alone.