Emily(/Amanda) and Nolan really are Like Brother and Sister: Amanda was taken from her father when she was nine, and then Nolan visited him in jail every day for nine years. But they met for the first time when Amanda was eighteen, just after David's supposed death.
Possibly a reference to Classical Mythology, specifically Odysseus's dog Argos recognizing his master after 20 years of separation.
In fairness, Sammy's apparently been at the end of his life this whole time and he does eventually die of old age.
Conrad and Lydia's affair (according to the Whole Episode Flashback) went on for almost ten years without Victoria noticing?
It's possible that Conrad and Lydia only ever got together when they were in the Hamptons (which gives Conrad a whole array of excuses: "I'm going golfing with some potential investors," "I need to go into the office for a few hours," etc). At one point, he claims to be flying to another state for business, when in reality he's visiting Lydia.
Also worth noting Victoria was cheating herself. She probably was aware and simply didn't bother with it. The Graysons are all about maintaining their public image.
Furthermore, IIRC when Victoria confronted Conrad about at the end of the pilot, she said 'You could have had anyone but you knew she was my friend", suggesting she didn't mind that Conrad had cheated on her, but that he had done it with one of the few people she trusted.
Something that always bugged me: How could Emily (the real one) have fallen off a storey-or-so tall banister while heavily pregnant and survive? Her unborn child should not have been able to withstand such an impact!