Was Oliver ALWAYS going to kill Felix or did he only do it once Felix rejected him?
Was that always in his plan and solely the house/money was the goal or was there a point where he actually wanted to be with Felix?
YMMV, but I assumed he wanted to be close to Felix and all of Felix's people. When he'd realized that he'd gotten caught lying about his background, and wouldn't only be pushed out of Saltburn but would never be welcomed back (judging from Felix's reaction), he killed Felix so as not to get caught or ever lose him. Then he took Saltburn so he could permanently possess Felix postmortem.
Oliver says that he "got there in the end, somehow", after orchestrating Elspeth's death, which suggests that his original plan to gain Saltburn went wrong at some point.
Why do the police never suspect Oliver?
Felix's death is one thing: Oliver may have poisoned him, but we can assume he did it in a way to make himself untraceable. And as far as we know, Oliver had nothing to do with James' death. But Venetia's apparent suicide so soon after Felix's murder would have at the very least triggered another police investigation, right? It looks like Oliver slashed her wrists, something that could conceivably be deduced by forensics. And the way Elspeth dies (with the air hose torn out of her throat) seems like it would be noticed. It's also hinted that the staff might know what's going on, though they may not care and even be happy that Oliver is bumping off their employers.
Oliver didn't kill Venetia. He left the razor there to push her to commit suicide in her grief over Felix's death, so it would've looked like suicide and no forensics would've detected Oliver's role in killing her as it was purely psychological. As for Elspeth, that's definitely a hanging plot thread, but I guess we're not technically aware of what Oliver told the rest of the world or how much the outside world knows about Elspeth's "illness."
Why does Venetia kill herself after Felix's death?
She's the closest to figuring out Oliver was at least involved, and she sees his true nature more clearly than anyone. Why kill herself after that conversation? Why do that instead of exposing or resisting Oliver?
She doesn't know Oliver was involved, and she doesn't really seem to know his 'true nature'; she suggests he's a moth crashing into the light of a world he doesn't belong in, which is absolutely not Oliver. She's also mentally broken and devastated after her brother's death, and - as Oliver keeps on pointing out - she's incredibly drunk. None of that adds up to excellent decision making; Oliver placing the razor blades by the bath likely gave her the thought and she ran with it.
It should also be pointed out that Venetia is the Un Favorite of her siblings, with Elsbeth especially deriding her for being attention-seeking and promiscuous. Considering how Elsbeth clung onto Oliver in her grief, she probably wouldn’t believe Venetia, especially since she has no actual proof that Oliver did it. Also, the family has no reason to suspect Oliver: we as the audience know his motivation is his obsession with Felix, but the rest of the family sees him as a friend of his with no suggestion that Oliver’s feelings were romantic in nature.
