I think scariness has almost nothing to do with whether the Monster is sympathetic, and relies almost exclusively on the protagonists. The typical ghost horror, for example, often has a tragic and sympathetic origin/motivation for the ghost. That doesn't mean it's not scary, as long as (as you said) the audience can relate to the protagonists in some way.
In a word: Gaara.
"Any campaign world where an orc samurai can leap off a landcruiser to fight a herd of Bulbasaurs will always have my vote of confidence"
In a setting of mine, The Dreaded Eldritch Abomination is a Proud Warrior Race Guy who eventually turns out to be a Last of His Kind Fallen Hero. He has a rather unique take on Not Worth Killing: not only he spares you if you flee from him, he deliberately radiates an aura of fear to make weak-willed enemies flee without fighting so that he won't have to kill them.
He eventually leads a minority faction of his side into a Heel–Face Turn - although he was never quite a heel to begin with. Doesn't detract any less from his scariness, considering that he's a Lightning Bruiser who favors One-Hit Kill if you're not a Worthy Opponent.
Blapper of Water
There can be something scary about watching two people you like kill each other. Not that I would know personally.
Seriously, though, I think structure of presentation is very important to horror, and I think that can affect whether the monster is scary and if the monster is sympathetic. Frankenstein presents two characters who you can sympathize with while being terrified of both Victor Frankenstein and his creation. Simultaneously. I suggest you read (or re-read) it and look at its structure. A huge question the text asks is "Who's the monster?" and you could argue that they both are.
Look at all that shiny stuff ain't they pretty
So in order for horror to be scary you need to care about the protagonists. This causes the audience to worry for them when they are put in danger. However, there's this trend of trying to make the villains in horror and slasher stories sympathetic. I think when this is done wrong this can undermine horror, as the audience is meant to feel sorry for and even root for the monster. Nothing's scary about watching characters you hate getting killed by someone you like. So I want to know if there's a way to avoid this. Can I make a monster that's sympathetic, but also still scares the audience.