When Zouken throws Byakuya's mother to the worm pit, forcing the young boy to watch her mother being assaulted, raped and eaten alive by the worms. Even worse as this happened months after Kariya was born.
Chapter 2: Role
The situation that Kariya is in — after being chosen by Zouken as the new Matou heir, he trains him in the worm pit every night, where the poor kid is raped by the worms. Byakuya can't do anything but acclimating him to the worms and wash him after the training is done for the day.
Just to twist the knife even further, the description that Byakuya has trouble carrying Kariya from the basement to the bathroom... Reminding you personally that this is a 6-year-old boy that is forced to watch his toddler brother to be trained and violated every day and can do nothing but console him after the torture is over.
Chapter 3: Changes
Kariya telling his brother that he likes Aoi's long hair, making Byakuya realize that Kariya has never interacted with another female in his life, and he was too young to remember their mother.
The casual mention that the Matou brothers hate summer vacations, because that means more time in the worm pit.
When Kariya reaches puberty, he starts being more conscious of his body and starts thinking of himself as unworthy of Aoi due to have been repeatedly violated through his life. He breaks down, calling out Byakuya and everyone else for letting him suffer.
Kariya: "No one does anything. Not grandfather, not you, not our dead mother, not even the maids. Nobody ever helps me!"
Byakuya: "Don't say that!"
Kariya: "Please no more! I hate doing these dirty things! [...] No....I don't like this! Save me, Aoi!"
The kicker? Byakuya starts freaking out at the thought of his brother abandoning the house, leaving him at Zouken's mercy. He runs away to his bedroom, leaving Kariya in the basement crying alone.
Chapter 4: Time of Separation
After their confrontation in the last chapter, Kariya becomes more distant with Byakuya, ignoring him at school and going to the bathroom on his own, after every night of training.
Realizing that Aoi loves Tokiomi, Kariya decides to let her be happy with the one she wants. It doesn't help that he still thinks of himself as "dirty".
Kariya leaving the Matou household the day after Aoi and Tokiomi's engagement party.
Chapter 5: Sacrifice
Byakuya breaking down at the thought of Zouken throwing him to the worm pit now that Kariya is gone. Especially with the reveal he's still traumatized by his mother's death.
"I don't want to die! I don't want to be eaten alive like mom. I don't want to be raped and eaten from the inside! I hate it!"
Chapter 7: Makiri's Blood
Yuri being thrown to the worm pit after Shinji is revealed to be incapable of magecraft. Not to mention, Byakuya's apathetic reaction to it (compared to his mother's death), showing that he's starting to break at the pressure of how dysfunctional the Matous are. And the next chapter reveals he started drinking and is starting to blame Kariya for all the misery in his life....
Chapter 8: Tohsaka's Daughter
Traumatized by Yuri's death and treated as a failure by Zouken, Byakuya slowly becomes alcoholic and starts blaming Kariya for all his misery, just like how we see him in Fate/Zero.
"At least Kariya was talented. He was born with magic circuits that Byakuya couldn't have. But even though he had them, he ran away. From this house. From Zouken. From this cursed family....And from Byakuya. If only Kariya hadn't ran away."
Byakuya is horrified when he finds out that Aoi's daughter, Sakura, was adopted by Zouken. When he tries to oppose to it, Zouken forces him to choose: leave Sakura at his mercy or else Shinji will be feed to the crest worms. Byakuya chooses his son over Sakura.
And the worst part of it all for Byakuya? All was this was started by his desire to protect his son from Zouken and for him to live a life away from magecraft. What he hadn't counted on was that Shinji will become obsessed with becoming a magus in the future. Not to mention this happens in the Heaven's Feel timeline, where Shinji was eventually killed by Dark Sakura. Making Byakuya's actions all for nothing. Talk about shooting the shaggy dog.
Sakura crying for her parents and Rin in her first day at the worm pit.
The Matou family eating dinner at the end of the chapter. Sakura starts coughing blood due to Zouken putting poison in her food. Being a brat and locked out of the loop about Sakura's situation, Shinji laughs at her suffering.
"Stupid girl!"
To make things worse, Zouken encourages Byakuya to laugh as well. Byakuya clearly doesn't want to do, but prioritizing his and his son's safety, he compels.
"He couldn't understand half of what father was saying. But he knew something. Byakuya had to laugh at Sakura's suffering until Zouken was satisfied. If he didn't, he won't be allowed to live in the house."
Chapter 9: Consideration
Kariya returns to the Matou household to save Sakura and his interactions with Byakuya are quite hostile, a far cry to how well they got along at the beginning.
Compare and contrast Byakuya's reaction to Kariya in the worm pit. When they were children, he was devastated, shocked and heartbroken at seeing his brother violated day after day. Now that they're adults, Byakuya can't bring himself to care for Kariya's suffering and eventual Body Horror and condems him as a fool for returning.
"I wish we weren't born..."
Chapter 10: Hope
Even if you think he deserves it for his cowardice and his hand in Sakura's abuse, Byakuya's death in the end. The night before he dies he has a nightmare with his mother, Kariya, Aoi and Yuri silently judging him for his cowardice. It concludes with him being thrown in the worm pit and eaten alive. Even though he was scared of the worms for years, he's been past the Despair Event Horizon to the point he's not afraid anymore, he welcomes it.
Another kicker? Byakuya was left pondering about his mother's last words for years, wondering why she called out to her children in her last moments even though the worms were eating her alive. In his own last moments he laments leaving Shinji (and Sakura) in the dysfunctional Matou family and muses about how he finally understands his mother...
Chapter 11: End of the Cycle
Sakura finding Byakuya's diary and finally understanding why Kariya sacrificed himself for her.
"I'm running out of time. Being born in this house was a mistake, I understand it now. I continued to live at the expense of others. So many dead Matous lay down at my feet. And I will follow them soon. All I could do was write everything down. Hopefully somewhere in the future, the family's endless curse will be lifted. Shinji. Sakura. Please be happy."
Kariya wanted to free Sakura from the Matous while Byakuya pitied her situation but couldn't do anything because of his apathy and fear of Zouken. In the end, their dream was realized and Sakura can live happily now with Shirou at her side....but they never got to see it.
