The "Singing in the Rain" dance sequence, the most pure and unpretentious expression of what it feels like to be in love ever committed to film.
One at the very end of the number: as Don walks away happily in the downpour, he gives a soaked passerby his umbrella.
Cosmo's continued friendship with Don. Don refuses to let stardom get to his head; he treasures his friends. And Cosmo knows that even though he doesn't get the fame that being the best friend of the biggest actor in Hollywood would imply, it's still worthwhile to be with Don.
Cosmo is possibly the only secondary character in a movie that will be completely fine as a single guy. He sets up his friend with the girl of his dream and doesn't once try to make a move on Kathy.
An example which doesn't occur in the movie, but is heartwarming nonetheless: Debbie Reynolds was not a dancer at the time she made Singin' in the Rain, and Gene Kelly worked her so hard, she began having breakdowns during rehearsal. Fred Astaire was hanging around the studio and found Reynolds crying under a piano. Hearing what had happened, Astaire let her witness one of his rehearsal sessions - normally he never let anyone watch him rehearse - in order to help her understand that dancing can be very hard work, even for the best performers, but that hard work is necessary and, in the end, worth it.
Another meta moment: Gene Kelley was apparently a rather tyrannical director, and neither Debbie Reynolds nor Donald O'Connor enjoyed working with him because of his perfectionism and tendency to yell at people. Debbie never did another movie with Kelley, but she did do another one with Donald O'Connor, I Love Melvin, because he treated her better.
Don's smile during Kathy's performance of "All I Do Is Dream Of You." It's clear that he's not mocking her, he's actually enjoying the show and seeing that Kathy does in fact have talent.