There's also the part in the movie where Pi makes up the second version of the story, for the insurance people. He just looks so tired and broken. The entire time he tells the story, he stares straight ahead into space, talking with no emotion, as tears run down his face.
Pi crying as he's being taken to civilization after washing up on shore, because Richard Parker left before he could say thank you and goodbye.
The look of despair on Pi's face when the boat sinks is truly heartbreaking, especially knowing that his family and everything else he loved was on the boat.
Let's not forget Pi losing his entire family when the severe storm sinks their ship, leaving him as the only survivor (and whether you believe the story that you see or the alternative one that he makes up, this is still the result either way). Sure, it's kind of a Foregone Conclusion, but it doesn't make it any less heartbreaking. They had a much less significant part in the book, but in the movie, you very well do get the chance to become emotionally attached to them, making Pi's loss much more heavily felt.
Really, the fate of everybody save Pi and Richard Parker from the boat.