The scene in the second season episode "Jimmy" in which Jimmy's brother, who loves him and has fought with everyone, even his own wife, to allow Jimmy to have a normal life, is forced to break down and agree to have him institutionalized.
From "Jimmy" is Al's monologue about midway through the episode:
"There was a girl named Trudy. She was retarded, Sam! Her IQ was lower than Jimmy's. And all the kids in the neighborhood, they used to tease her. Kids can be cruel. They'd call her names, like "dummy" and "monkey face." And I hated it. And I used to get in fights all the time over this. But that's what big brothers are for, right? My mother couldn’t handle it. That's probably why she ran off with this stupid encyclopedia salesman. But my dad tried to keep us all together. He was a construction worker. He went from job to job, and then when it took him to the Middle East, I wound up in an orphanage and she wound up in an institution. When I was old enough, I went back there for her, but it was too late—she was gone, Sam. Pneumonia, they said… How does a sixteen year old girl die from pneumonia in 1953, Sam?"
The season three episode "Black on White on Fire," in which Sam manages to talk down his host's older brother only for him to be shot anyway because of a misunderstanding.
The season two finale "M.I.A." with Al's wife Beth, the only woman he ever truly loved. He was M.I.A. at the time, and Beth thinks he is dead and has given up hope of him being alive. He tries to get Sam to stop Beth from falling in love with another man she meets, but they can't alter their own timelines. And if that isn't sad enough, the very end has hologram Al talking to and "dancing" with Beth in her home. She seems to reply to something he said, but it turns out that she was just reacting to the record that was playing. The change in Al's face from hope that she could hear his voice to pure sadness when he realized she couldn't...
"The Leap Home" two-parter. In Part 1, Sam leaps into his younger self to win a championship basketball game. However, he desperately wants to change his family's future (prolonging his father's life, stopping his brother Tom from dying in Vietnam and preventing his sister Katie from marrying an abusive drunk). Despite his efforts, however, he is forced to realize that there are some things he just can't change.
Sam: It's not fair, Al. I mean, c'mon, it's not fair.
Al: Well, I think, uh, I think it's damn fair.
Al: I'd give anything to see my father and my sister for a few days, be able to talk with them again; laugh with them; tell 'em how much I love them. I'd give anything to have what you have, Sam. Anything.
Part 2, meanwhile, provides Sam the opportunity to save Tom, but at an unforeseen price. Maggie, a war photographer that survived in the original history, dies during a mission to rescue some POWs. Additionally, her last photograph reveals that one of the POWs was Al, who won't be set free for another five years.
In "Return of the Evil Leaper," Sam leaps into a college student named Arnold, who has a habit of dressing up as a costumed hero and risking death to save innocent people. It turns out that Arnold is an orphan, losing his parents in an ex-cop's shooting rampage and only surviving because he left to get his jacket out of the car. As Al talks to him about this, Arnold breaks down. ("I should've died, too.")
From the episode "Raped," the young woman whose life Sam has leaped into is forced to temporarily come out via Al's gateway to help Sam give testimony over what happened to her. Slowly, the camera shifts to her instead of Sam and in a touching effect Bakula stops talking letting the young actress (Sheryl Pollak) give the testimony in her own voice. When she shakily says "I was raped," you believe her.
And Al's silent reactions in the background. You can tell he's trying not to cry.