The cover of "Go To Sleep You Little Baby" performed over the first episode's end credits as a Shout-Out to O Brother, Where Art Thou?. There's also "Children of the Sun" playing as Rye follows Judge Mundt to the Waffle Hut.
"Yama Yama" by Yamasuki kicking off "The Myth of Sisyphus." The song itself is an odd yet fitting introduction to an episode where shit is about to go down.
"Down in the Willow Garden" as performed by The Chieftains playing over the credits to "Fear and Trembling" immediately after seeing Lou sitting on his back porch to protect his family for the night. It sweetly hearkens to the way we see Lou do this back in season one.
Blitzen Trapper's cover of "Man of Constant Sorrow", which plays over the end of "Rhinoceros" and pays tribute to The Coen Brothers' other works in a beautiful fashion.
Lisa Hannigan's very soft cover of "Danny Boy" in "Did You Do This? No, You Did It!" as Bear proceeds to kill Simone offscreen and have a breakdown about it directly after.
"Oskus Urug" by Radik Tyulush, played while a shotgun-toting Action Mom Gloria clears her stepfather's house room by room, with a cutaway to the murderer exhaling a vast cloud of steam like some sort of dragon. The result is quite a dramatic and otherworldly scene.