Cullen Bohannon told Elam about finding his boy's body in the hayloft...and the body of the slave Cullen had freed before the war, embracing the boy and trying to protect him. He looks over and assumes Elam isn't listening...but we cut to Elam, still wide awake.
Bohannon's flashbacks to the day he found his wife dead. They are all very brief, but ever so painful.
For the more of an asshole he was (and sometimes slides back now), you gotta feel sorry for Mr. Toole having such a big, dumb grin on his face when Eva tells him she's pregnant....but then seeing that smile vanish when she tells him it isn't his. You can see the sheer look of pain on his eyes as he stays in utter silence he doesn't even lay a finger on her, which for the period he's in is something of a minor miracle.
Reverend Cole's final dialogue with his daughter (and Joseph) is this. After two entire seasons of a broken relationship, he looks at her, breaking in tears, and says he's been a horrible father, says she's better than he could ever be. and begs for her forgiveness. She forgives him. Then he talks with Joseph, saying he accepts his impending death. Then Joseph stabs him in the gut, and he dies with a big grin on his face: knowing he's been forgiven by his daughter makes him die in peace. Damn.
In the same episode, Doc Whitehead faces his charges of robbery. He asks Bohannon to finish him off, so he can die with a friend by his side instead of common soldiers. For the first time in the entire series, Bohannon seems incapable of pulling the trigger on someone, but he eventually agrees. Doc says he had a good life, and that this is a good death for him. Bohannon shoots him in the head, and the last shot we see is Bohannon, utterly alone in the middle of the woods, representing how alone he is spiritually after his last remaining friend has died.
In the season two finale, the death of Lily. The look on Cullen's face when he finds her body is heartbreaking, and you can see just how dejected he is as he carries her lifeless body to the church.
In the same episode Mr.Toole has too much and is Driven to Suicide. It's heartbreaking to watch the man try his best to be a good husband and redeem himself, only to see his beloved loves someone else. Swallowed in grief, he ends his own life.
Season three premiere, we have the death of Dick Barlow, new chief of police. Dick Barlow is introduced to us as the one genuinely entirely good human being in this railroad, and the only member of this entire cast to treat Fergurson as an equal, not even mentioning the race factor when he meets him, instead happily chatting about their daughters and their wives like true friends even though they just met. But then, Dick Barlow is fatally wounded in the gut, and in his final moments, asks to see Fergurson's baby daughter, delivering some tearjerking Famous Last Words:
Dick Barlow: She's beautiful, Mr.Fergurson, just like you said...
In season four, there's the discovery of Ezra's corpse. Bohannon is left in stunned silence, Ruth weeps for hours, Mickey just stares in the horizon. Nobody can vocalize the sheer level of that tragedy, and quite possibly neither can the audience.
Similarly, when Ruth herself is hanged. Literally everyone in Hell on Wheels loves and cherishes her existance, and nobody wants to see her hang, making her the only person in the entirety of Hell on Wheels with no enemies. Despite that, she refuses every attempt to be saved, even a pardon from the Governor himself. And thus she finally hangs, trying to contain her tears, the last good soul of Hell on Wheels just...gone.