Lady Grantham: Why does every day involve a fight with an American?
Right after Matthew and Sir Richard have a fight, following Mary ending the engagement, and deciding to leave Downton Abbey first thing in the morning:
Sir Richard: (to Lady Grantham) I doubt we'll meet again.
Lady Grantham: Do you promise?
In the same exchange discussing the lamp broken in the fight. Matthew apologizes for breaking it.
Lady Grantham: Oh no need, it was a wedding present from a wretched aunt. I have hated it for half a century.
Let's not forget this gem, after Pamuk's death:
Lady Grantham: Of course it would happen to a foreigner. It's typical. No Englishman would dream of dying in someone else's house - especially somebody they didn't even know.
And when discussing the scandal with Cora
Lady Grantham: We can't have him assassinated. . .I suppose.
About Sir Richard
Lady Grantham: It's not that I dislike him, I just don't like him.
The Dowager Countess' mini-rivalry with Mrs. Crawley is almost too much fun to watch. Their constant one-upping of each other often provides much-needed comic relief in an otherwise very dramatic series.
Violet: You are quite wonderful the way you see room for improvement wherever you look. I never knew such reforming zeal.
Isobel: I take that as a compliment.
Violet: I must have said it wrong.
After Mrs. Crawley accuses the Dowager of telling Mary to refuse Matthew until Cora's baby is born.
Dowager Countess: Your quarrel is with my daughter Rosamund. I told her to accept him, so put that in your pipe and smoke it.
Made even better by the quick shot of Carson's rather pleased reaction.
When Coras' mother is visiting, Lady Grantham has this gem.
Lady Grantham: I'm so looking forward to seeing your mother again. When I'm with her, I'm reminded of the virtues of the English.
Matthew: But isn't she American?
Lady Grantham: Exactly.
Mr. Carson's reveal of his shameful past as a stage performer, which to him seems to be on the same level as murdering someone.
That said, Carson's stuffy nature can become so hilarious that it's an inside joke among some of Downton's residents. Among one of his more memorable moments is his epic rivalry with the new telephone.
This exchange between Lady Grantham and Sybil, right before Edith's wedding:
Lady Grantham: I really think you should go to bed. No bride wants to look tired on her wedding, that means she’s anxious or up to no good.
Edith: I won't sleep a wink.
Sybil: Tonight or tomorrow?
Lady Grantham: Sybil, vulgarity is no substitute for wit.
Sybil: You started it.
Although this is a little sad in hindsight, when one realizes that Edith is once again the least fortunate sister even in this regard. Consider that Sybil's marriage to Tom, a handsome and fiery rebel is clearly passionate, considering how quickly she got pregnant, Mary has married the dashing and romantic Matthew and the first two episodes of season 3 made it clear that their relationship certainly does not lack passion. This impending marriage, on the other hand, does not seem likely to yield much in that department.
Even better, you can see Matthew facepalming at the suggestion.
This exchange (about Thomas after he was outed)
Mrs Hughes: Do you think he's the first of. . .that type I've met?
Carson: I should hope so!
Molseley dancing like a maniac after being hit by the liquor meant for O'Brien.
While it happens in the middle of a very serious scene, this troper found it hilarious that Robert was commended for being smart enough to use modern methods to handle the finances at Downton, thus saving it from future financial ruin, after he spent several episodes fighting tooth and nail whenever Matthew tried to implement said methods.
About Cora's pregnancy:
Robert: How could this happen?
Cora: The doctor will tell you the details by a glass of whiskey.
New footman Jimmy is irritated when Carson introduces him to the Crawleys as James:
Jimmy: I was Jimmy to Lady Anstruther.
Carson: I don't care if you were Father Christmas to Lady Anstruther!
The clash between The Dowager Countess and Cora's mother, Martha Levinson. Played to absolute perfection by Dame Maggie Smith and Shirley MacLane. A propper, aristocratic British Matriarch + An eccentric rich American matriarch = fun for everyone!
It's not strictly from the show itself, but Graham Norton decided to share some Tweets with Dan Steven on his show after Matthew's death. Watch the segment here.
"Matthew had BETTER be in a coma and not dead. And don't use the doctor that killed off the sister".
When Mrs. Hughes reveals that a person standing in the airing cupboard can listen to a conversation in the kitchen through a vent, so she knows why Bates suddenly left with his wife, Vera.
Carson: If I were a gentleman, I would not wish to know. Mrs. Hughes: But you're not. Carson: Fortunately. [closes door]
Daisy the Deadpan Snarker when Ivy and Daisy discussing Ivy's plans for the evening.
Ivy: What do I wear?
Violet is outraged to hear Isobel describe her new life after Matthew's death, and tells her there's no need to eat off a tray!
Matthew notes in a letter how stupid he was to not get his will in order after getting married. Violet responds "I'm afraid I have to agree with him there," which is filmed almost like a Shakespearian aside.
After Violet comments on Branson's lack of conversational skill:
Robert: Not everyone can be Oscar Wilde.
Violet: Thank heavens for that.
Robert managing to say without a hint of irony how great it would be to have people serve your every need, while Carson is serving him dinner.
Carson's biting Sarcasm Mode throughout negotiating with Molseley to replace Alfred.
The militaristic soundtrack during Alfred's cooking class.
Carson getting Squicked out by Mrs. Patmore talking about how hot Rudolph Valentino is.
Mary casually discussing how Thomas will love scoping out the hot ship's crew if Robert takes him to America.
The final shot of series 4 (minus the Christmas special): Violet, Rosamund, and Isobel all surreptitiously leaning forward to watch Mary's suitors all driving off together.
Violet's gobsmacked face when Isobel and Lord Merton start getting along.
When Mary meets with Jack Ross to convince him to break things off with Rose, the entire scene is more or less very bittersweet from start to finish. There is one amusing quip from Mary, though. When Jack guesses that Lord Grantham wouldn't be very happy to have a black in-law, Mary says, "To be fair to Papa, I think he'd be more shocked by the fact that you're a singer in a nightclub."
This exchange, as Carson gets the servants to help setting up the bazaar:
Jimmy: Will we be paid extra?
Carson: No, but if you play your cards right, you'll avoid being clipped around the ears.
Slade constantly getting on Carson's nerves with his American service style.