After he returns from juvenile prison, Joe is met with a welcome back party and a newly built gym just for him and Nishi. He doesn't even know how to react, only staring as everybody laughs and dances in front of him. Later that night, while everyone else is asleep, he's crying quietly under the sheets. "This was the first time in his life he had ever felt loved by other people. That was the reason for his tears".
Yoko deciding she won't run away anymore (as a Call-Back to Joe and Rikiishi's first match) and ordering her driver to return to the stadium. And when everyone keeps telling Joe to give up the match (and with good reason), Yoko, showing how much she has come to understand him in these last minutes (possibly even better than anyone else alive), is the only one to cheer him on and tell him to fight without regrets... And he does. Which might have played a big part in what made him decide to give his gloves to her, even if you take it as his declaration of love.