The scene in the second season episode "Jimmy" in which Jimmy's brother, who loves him and has fought with everyone, even his own wife, to allow Jimmy to have a normal life, is forced to break down and agree to have him institutionalized. This development especially alarms Al, who later explains why.
Al's monologue about midway through the same episode:
There was a girl named "Trudy." She was retarded, Sam! Her IQ was lower than Jimmy's. And all the kids in the neighborhood, they used to tease her. Kids can be cruel. They'd call her names, like "dummy" and "monkey face." And I hated it. And I used to get in fights all the time over this. But that's what big brothers are for, right? My mother couldn't handle it. That's probably why she ran off with this stupid encyclopedia salesman. But my dad tried to keep us all together. He was a construction worker. He went from job to job, and then when it took him to the Middle East, I wound up in an orphanage and she wound up in an institution. When I was old enough, I went back there for her, but it was too late—she was gone, Sam. Pneumonia, they said. How does a sixteen-year-old girl die from pneumonia in 1953, Sam?!
The Season 3 episode "Black on White on Fire," in which Sam manages to talk down his host's older brother only for him to be shot anyway because of a misunderstanding.
The Season 2 finale "M.I.A." with Al's wife Beth, the only woman he ever truly loved. He was M.I.A. at the time, and Beth thinks he is dead and has given up hope of him being alive. He tries to get Sam to stop Beth from falling in love with another man she meets, but they can't alter their own timelines. And if that isn't sad enough, the very end has Al, a hologram, talking to and "dancing" with Beth in her home. She seems to reply to something he said, but it turns out that she was just reacting to the record that was playing. The change in Al's face from hope that she could hear his voice to pure sadness when he realized she couldn't...
"The Leap Home" two-parter. In Part 1, Sam leaps into his younger self to win a championship basketball game. However, he desperately wants to change his family's future (prolonging his father's life, stopping his brother Tom from dying in Vietnam, and preventing his sister Katie from marrying an abusive drunk). Despite his efforts, however, he is forced to realize that there are some things he just can't change.
Sam: It's not fair, Al. I mean, c'mon, it's not fair.
Al: Well, I think, uh, I think it's damn fair.
Al: I'd give anything to see my father and my sister for a few days, be able to talk with them again, laugh with them, tell 'em how much I love them. I'd give anything to have what you have, Sam. Anything.
The acting in the above exchange really bears mentioning: Sam's rage and despair is expertly portrayed by Scott Bakula, while Dean Stockwell gives Al a quiet wisdom. It's arguably one of the best-acted scenes in the entire series.
Part 2, meanwhile, provides Sam the opportunity to save Tom, but at an unforeseen price. Maggie, a war photographer who survived in the original history, dies during a mission to rescue some POWs. Additionally, her last photograph reveals that one of the POWs was Al, who won't be set free for another five years.
In "Return of the Evil Leaper," Sam leaps into a college student named Arnold, who has a habit of dressing up as a costumed hero and risking death to save innocent people. It turns out that Arnold is an orphan, losing his parents in an ex-cop's shooting rampage and only surviving because he left to get his jacket out of the car. As Al talks to him about this, Arnold breaks down ("I should've died, too").
From the episode "Raped," the young woman whose life Sam has leaped into is forced to temporarily come out via Al's gateway to help Sam give testimony over what happened to her. Slowly, the camera shifts to her instead of Sam and, in a touching effect, Bakula stops talking, letting the young actress (Sheryl Pollak) give the testimony in her own voice. When she shakily says "I was raped," you believe her.
And Al's silent reactions in the background. You can tell he's trying not to cry.
In "Disco Inferno," Sam leaps into an aspiring musician's older brother, only to remember that he has an older brother of his own. He can't remember everything, though, and has a lingering sense of dread about the whole thing.
Al: Sometimes it's tough bein' a big brother. Sam: Yeah, now I know how Tom felt. I always thought he was putting me down; y'know, trying to tell me what to do. But by the time I figured out he was just lookin' out for me, it was too late because he was... (realizing) Tom's dead, isn't he?
"Future Boy" goes to lengths to make it seem like actor Moe Stein's desire to build a time machine is caused by senility and that he's not mentally fit to live on his own. The ending reveals the true reason for his actions: To change a good review he received when his wife was pregnant with their daughter, Irene.