From an emotional point of view, it's even worse when you consider that Scrooge does not view the contents of his vault as monetarily valuable. As he admitted to others in related media, the value of the treasures in the Money Bin is the memories of his adventures throughout his life, not the artifacts and treasures that he gained from them. Having what amounts to his entire life's hard work and courage being stripped from him with a single wish is probably the heaviest blow to Scrooge's morale and ego than anything else.
It only gets worse when Merlock gets the lamp and goes a step further by outright destroying the bin and covering the money before Scrooge's eyes with stones and spikes as he creates his castle.
His stay in prison is also depressing. He is there having a mourning monologue in a rainy night until all his employees and family come to get him out of prison. At least Mrs. Beakley manages to make the scene funnier with all her hysterical sobbing.