When you discover that Yuto's cards erase the memories of anyone he meets, every friend he ever had will never remember him. Ryotaro being the only person who wasn't effected, because one of Sakurai's cards already did.
Yuuto's family is never mentioned, implying that they've already forgotten him.
The last episode, oh boy! After the Climatic final fight, all of the Imagin disappear, including Momotaros, Urataros, Kintaros, Ryuutaros, Deneb and Sieg. Both Ryotaro and Yuto are saddened, you can simply feel the emotion. When Yuto heads back to his Liner, there's no one there aside from himself, and he finds a feast, which is a goodbye present from his Imagin partner, goes to show while Yuto is annoyed by Deneb, he cares about him. Ryotaro didn't take it well, saying he never got a chance to say goodbye. Hana, despite putting up this barrier that she hates the Taros, also cries.
Kotaro badmouthing Ryotaro stuck out to this troper; granted, no kid would exactly be squealing with joy if their grandparent gave them a country-sounding name along with terrible luck, but the lengths Kotaro goes... harsh man.