How about having your entire life taken away from you because you put on the wrong clothes?
This is a binding magical contract with no less than three ironclad clauses in it that are activated by putting on a coat, right? OK. Now...who's enforcing this? What all-powerful being created this? Who demanded it? Is it God? Assuming there is a good, why would they need to create the Clauses? Moreover, if it's not God, or any god at all, then how long has this been going on exactly? Judy said it took her twelve centuries just to perfect a cup of cocoa, so it's at least that. If that's the case, it could have started with St. Nicholas (a fourth-century bishop), but that would mean that "god" is just an allegory for "magic(ks)". That would explain how all the elves are basically just children that can live for thousands of years.
Back to the original point of who is enforcing this contract - in the first film, Scott questions what would happen if he just choose not to follow the Santa Clause, and Bernard just guilts him into doing it, but what if that didn't work? In the third film, Jack Frost just turns Christmas into a parody, but what if there's a guy who just hates Christmas, puts on the coat, and refuses to deliver the toys, who doesn't care about how disappointed the children of the world feel, and just wants the holiday to die? What's stopping him?
In the first film, Laura realizes that Scott really is Santa Claus because he gave her (back) the thing she wanted most of all for Christmas that year—he returned Charlie to their home. (That he took him in the first place is beside the point.)
All of the rest of the elves are cheerful, so why is Bernard so uptight? Well, given his high position and the legacy status of Santa, he's likely gotten impatient with giving the same spiel over and over again, in addition to becoming more and more jaded every time another Santa goes missing, which would no doubt be worse if he thought of the Santa in question as a personal friend.
A gay man, or one who'd simply be happy single, wouldn't be able to stay as Santa Claus for more than a few years unless they entered into a marriage they didn't want.
How many people before Scott has played the role of the Santa Clause, and how many times has someone been replaced after fallen off a roof or gotten seriously injured in other ways that tragically ended their clause? Assuming there was more than just one Santa Clause beforehand. The elves are pretty quick to accept that he's the new one, right after the last one's death. How many times has he passed that they're that used to it?
A lighter explanation: perhaps any Santa who "died" were just warped back to their homes without any memories of taking on the clause?
If being Santa makes Scott immortal doesn't that mean he'll eventually outlive his son Charlie? Charlie's younger half brother Buddy Clause would end up watching his older brother grow old and die while he presumably lives on to take over as Santa one day.
What happened to the previous Santa's wife? Did she lose her immortality and just crumble to dust when the previous Santa died? Risking your own life is one thing, but the realization that, after a certain number of years, your wife is doomed to die with you would eat at you and make you paranoid.