A Song of Ice and Fire features lots of The Clan, and most of the Great Noble Houses have intermarried repeatedly over the past 300 years. The Southern Houses have a tendency towards big families and bundles of cousins in a bid to have enough redundancy to keep the family holdings within the family, come what may. Rather more so than Northern Houses tend to do... The vast majority manage some form of Big, Screwed-Up Family, however; be they from the North or South.
House Frey's elderly patriarch, Lord Walder Frey, is on his eighth official wife (don't ask about the affairs, one night stands, or what have you — also, you might want a gallon of free Brain Bleach with that thought) and has over a hundred known and living descendants of varying degrees at the start of the series. For added confusion (for not-Freys), a large chunk of the family are also named Walder or Walda (Waltyr just confuses everybody) in a bid to suck up. Now, if some rumours about what Black Walder has been up to are even remotely true, the paternity of a fair few of the younger Freys may be in question...
Tracing the descent of the Heir to the Eyrie requires some work, as the Arryn family has had trouble producing surviving sons for a couple of generations now. As a result, the various female-line claims have to be sorted out — with a bit of head-scratching. This becomes important to the plot in later books. Littlefinger also steps in to make it sound more tangled than it is for his own reasons. The relationship is simple enough: the heir to the Eyrie, should little Robert Arryn "unexpectedly" die without issue, is Jon Arryn's (Robert's dad) great-nephew Harry Hardying (the son of Jon's sister's daughter). But, for the right price, somebody could, with varying degrees of ease, get that called into question as there are other options out there. Lady Anya Waynwood, Harry's fostermother and cousin, is very well aware of this, even if he may not be.
Craster marries his daughters — of which he has many — and sacrifices his sons to the White Walkers. By the time we meet him, he's got granddaughters (who are also his daughters) old enough that he's now marrying them.
The Targaryen dynasty had a tradition of brother-sister, uncle-niece, or, failing these, first-cousin marriages — repeated six times throughout the generations for brother-sister alone (including 2 times in consecutive generations)— to reduce the number of rival claims to the throne and maintain their pure Valyrian blood. An unfortunate side effect of this was their own special brand of Royally Screwed Up: at least one character comments that with the Targaryens you have a 50% chance of Crazy Awesome and a 50% chance of Ax-Crazy; although the fact that only three of the seventeen Targaryen monarchs were crazy makes that a significant exaggeration. Just see this.
House Baratheon is distantly related to the Targaryens since their founder was Aegon the Conqueror's illegitimate half-brother Orys. And Robert, Stannis and Renly's grandmother was a Targaryen princess, daughter of Aegon V, which is how they had a claim on the Iron Throne.
The Targaryen and Velaryon family tree around the Dance of the Dragons gets quite confusing to follow, Rhaenyra marrying her second cousin Laenor Velaryon, Laenor's illegitimate son Alyn (possibly in fact his illegitimate brother) marrying their cousin (or half-niece) Baela Targaryen, and Rhaenyra marrying Baela's father, her father's brother Daemon, who was previously married to her husband's sister.
Joffrey, Tommen, and Myrcella are brother and sister, first cousins, and third cousins all at once, thanks to their grandparents being cousins and their parents being brother and sister.
The Flowers in the Attic novels by V. C. Andrews (really, almost any series by Andrews) has a famously incestuous cast, further confused by multiple marriages, adoption, half-siblings, and fake siblings (how does one address the son you have by your mother's second husband when your mother is also your aunt?)
Robert A. Heinlein tends to have extremely meshed up family trees. Probably owing to the fact that most of his novels are set in a time where sexual stigma is all but gone and advances in genetics has erased the physical consequences of incest.
Lazarus Long did NOT make his mother pregnant with himself (as implied), but in the later books, he is the ancestor of about 99% of all humans. And he did have children with at least 2 of his female clones.
—All You Zombies— has a uniquely tangled family tree, considering that there's only one person in it.
Nanny Ogg is said to be matriarch of a family tree that makes a mangrove swamp look straightforward.
Whereas her cat Greebo's family tree is more like a single mangrove, considering he's managed to be the sole male ancestor of most of Lancre's cat population for at least seven generations.
The Nobbs family is described as being less of a tree than a briar patch. In Feet of Clay, Nobby is actually revealed to be descended from the Earl of Ankh. Of course, the source of this information is an obsessive-compulsive vampire whose hobby involves editing the Morporkian royal line for desirable genetic traits and was making a power play against the current non-royal government, so grain of salt recommended. Add to this the sheer number of artifacts the family has "acquired" over the years through compulsive thievery and Sam Vimes believed you could probably prove Nobby to be the Dowager Duchess of Quirm if you really wanted to.
Pyramids has Pteppic from the royal family of Djelibeybi. Because the "divine blood" must never be diluted, they had a long history of inbreeding and outright incest: Pteppic's marriage prospects are limited to his aunt and his sister. Also, anything spoken by a ruling monarch at any point in the last seven thousand years is automatically law and treated as such unless repealed... and Pteppic's grandmother declared herself to be a man in order to keep the throne. So at this point, Pteppic could easily be his own grandpa...or his own grandma...or anything really. And the prohibition on official out-marriage didn't prevent a lot of bastards over the millennia. Pteppic remarks that most of the country's population is probably related to the royal family in some degree, so just find someone who dreams about the seven cows to get a new heir. When Pteppic abdicates, he easily finds a successor. As a matter of fact, it's the person he's explaining this to: Ptraci, a courtesan he (rightly) suspects to be his half-sister.
Harry Potter: The "pureblood" movement believes that only those whose ancestors are all witches and wizards are "true" members of the wizarding population. They therefore refuse to marry any squibs, muggles, muggleborns, or magical non-humans (like giants or veelas), or anyone with an ancestor from one of those groups. By the time of the novels ALL the pureblood families in Britain are very closely interrelated, especially those aligned with the Death Eaters (who have even more insanely narrow standards for marriage partners).
The Peverell family tree◊, which reaches about as far back into wizarding family history as can be reliably traced, includes almost all the main characters in one way or another. Voldemort and Harry are even related as direct descendants of the middle and youngest Peverell brothers, respectively.
The Peverell tree includes as a sub-set The Ancient and Most Noble House of Black. Their family tree◊ makes them officially blood relatives of most of the other major Death Eater families. Through their disinherited descendants, they're also blood cousins to many muggle-friendly families. Molly and Arthur Weasley were third cousins through the Black family even before they married.
Given how small the general wizarding community is, all non-muggleborns are also probably related, if more distantly. Even then, some wizards born into what seem to be all-Muggle families are likely the descendants of squibs from pureblood families and part of the extended kindred. Ron calls the opposition to halfbloods and muggleborns completely illogical given that the wizard population has hit bottlenecks in the past that could have destroyed it if they hadn't married outside the community. Arthur Weasley suggests that purebloods by the current definition are headed to extinction because soon they'll all be first cousins and within the incest taboo. In the movies, Hagrid is skeptical that any modern wizard truly fits the pureblood definition...it's just a matter of how recently they had muggle or non-human ancestors.
In the Incarnations of Immortality, the woman Orlene has a very tangled relationship with almost all of the other Incarnations. Mars and Gaea are her father and mother, while Satan is Gaea's current lover. Chronos was Orlene's lover until Orlene's suicide, while an aspect of Fate was Orlene's grandmother. In the book "And Eternity", we find Orlene becomes God.
The Atreides family in the later Dune novels. Having the same genes reintroduced every few generations in the form of Duncan Idaho probably led to some strange folk...
The Kashpaw-Nanapush-Lamartine-Morrissey family from Louise Erdrich's Love Medicine. Lulu Napapush is responsible for at least half of the tangledness.
Looking at the Star Wars Expanded Universe, try working through Soontir Fel's family tree. The Fel family is pretty large even before it gets to him, and then it practically explodes; he marries the sister of Wedge Antilles, trying the Antilles family in, and has a lot of children with her. Presumably several of the ones who don't get killed off end up married. One of his sons ends up in a relationship with Jaina Solo, daughter of Leia and Han, tying in the Skywalker and Solo lines. It's implied by the Legacy comics that his son and Jaina got married. Soontir Fel's great-grandson is The Emperor. Oh, and in the Hand of Thrawn duology we found that Fel had clones. A lot of clones. Many of them in sleeper cells, with families of their own. For some reason, they almost never show up on family tree charts.
Add to that the insanity of the Skywalker family. Virgin birth gives rise to Anakin, his son marries the former Emperor's Hand, his daughter marries Han Solo... Who just so happened to be the last in line of the house of Solo, a dynasty of Kings who used to rule Corellia before it became a democracy. And so on. Then we get to Cade Skywalker. Cade's father was Kol Skywalker. His mother, really got around. She lived a double life as Nyna Calixte, an Imperial Grand Moff, and Morrigan Corde, an Imperial Spy. As Nyna, she'd sleep with anyone who would advance her career, (Moff Rulf Yage, Moff Morlish Veed) as Morrigan, she'd sleep with whoever she thought would help her mission (This includes Cade's father, and Cade's best friend Jariah Syn, whom he considers a surrogate brother), and at least one of her past relationships (Moff Rulf Yage) gave Cade a half sister, Gunn Yage, who Cade almost slept with as well. Cade must've inherited his mother's sluttiness, as he's slept with a Sith, an Imperial Knight, Delliah Blue (whom he for a time shared with Jariah Syn in a little 3-way relationship), countless Twi'lek, Human, and Zeltron one night stands in sleazy bars, and Princess Marasiah Fel, who, if the above statement about Sontir Fel's son marrying Jaina Solo is correct, is actually related to him with their most recent common ancestor being Anakin Skywalker.
The family tree of Ciri in Andrzej Sapkowski's The Witcher cycle is a major plot point, as her ancestry is intricately tied to The Prophecy predicting The End of the World as We Know It, and it gets discussed a lot at a certain point in the story, but... Well, just abandon any hope to get it without pen, paper and nice two-page wide flowchart, because with the number of mages, elves, Kissing Cousins, Brother–Sister Incest and whatever else (though, thank you, without Time Travel at least until Ciri herself saw to it) it is as convoluted as it gets.
Margit Sandemo's The Legend Of The Ice People has this. The 47 book long story begins in 1581, and ends sometime in the 1960's, and it follows a family/clan. The family branches all over Scandinavia, and the family tree rivals that of the European royal houses. Just about everyone can trace themselves back to the founders of the modern Isfolk in several ways, and the older, pre 1581, Isfolk were even worse with their inbreeding, though not willingly all the time. For more convolution throw in Lucifer, Lilith and Typhon as ancestors to some of the modern ones.
The Hitchhiker's Guide to the Galaxy plays this for laughs by combining it with My Own Grampa. While the family tree of Zaphod Beeblebrox (who calls himself "the First") isn't fully revealed in detail, his father is Zaphod Beeblebrox the Second, his grandfather is Zaphod Beeblebrox the Third, and his great-grandfather is Zaphod Beeblebrox the Fourth. In addition, Zaphod the Fourth says Zaphod the First is actually Zaphod the Nothingth. (Confused yet?) According to Zaphod the First/Nothingth, this was all caused by "an accident with a contraceptive and a time machine."
It's also known that Ford is his semi-cousin because they share two mothers.
The Vangers in Stieg Larsson's Millennium Trilogy are an incredibly screwed-up family with lots of members.
Due to their obsession with blood purity and the resulting intermarriages and incest, ALL Masters in The Stone Dance of the Chameleon are somehow related, although tracing the exact kinships can be a nightmare. The author's website helpfully provides some (complicated) family trees. (Beware of a spoiler in the House of Masks.)
The Pollard family in The Bad Place. The present generation (co-protagonist Frank, villain "Candy", and their sisters Violet and Verbena) have two great-grandparents instead of the usual eight, two grandparents (who were brother and sister) instead of the usual four, and one genetic parent (a hermaphrodite who self-impregnated).
Koontz seems to be fairly fond of this trope. In addition to the non-consensual Parental Incest that results in the births of major characters in Whispers and Life Expectancy, in What the Night Knows, a major character is the product of three generations of line-breeding in his family, starting with a brother-sister pairing, then the father/uncle impregnating his daughter/niece, then impregnating his twin granddaughters/grandnieces, one of whom is the mother of the character in question. The other twin and her daughter (also fathered by the family patriarch) state in their last documented conversation with their relative that they're both about a month pregnant. Guess who by?
In The Wheel of Time, the Damodred/Mantear/Mandragoran/Trakand/Al'Thor family tree. Rand is the half-brother of Galad by the same mother, Tigraine Mantear, although Rand and Luc/Isam are the only characters who actually know this. Galad is the half-brother of Elayne and Gawyn by the same father, Taringail Damodred. (Probably.) Rand is in a relationship with Elayne and as of the latest book, she's pregnant, expecting twins. Gawyn wants to kill Rand in revenge because he believes Rand killed Gawyn's mother Morgase, although she isn't actually dead. Gawyn is in love with Egwene. Egwene is a friend of Elayne's and used to be betrothed to Rand. Morgase is now working for Rand's childhood friend Perrin as a servant. Moiraine Damodred, Rand's mentor, is Taringail's younger half-sister, and so Galad, Gawyn, and Elayne's aunt; her Love Interest, Thom, is one of Morgase's ex-lovers, and another Morgase ex, Gareth Bryne, is, as of book twelve, engaged to Moiraine's formerlover Siuan. Tigraine's brother, Luc Mantear, is also alive and merged some way or other with Isam Mandragoran, first cousin of Lan Mandragoran, another mentor figure of Rand's and Moiraine's Warder. No characters know anything at all about Luc/Isam being alive or connected except for himself.
Lampshaded when Rand asks a particular noblewoman about how closely Tigraine and Morgase would be considered related. He is relieved when after being given the suggestion to compare them like farmers, the noblewoman says nobody would even bother thinking about it. Even better is that he was specifically asking about Elayne, though in an obfuscated way. As a messianic character, he may lack knowledge of lots of things a prince or general would know, but a farmer knows what interbreeding does to a herd.
Oberon from The Chronicles of Amber has so many children from different marriages that his family tree is too convoluted for anyone to decide who is the rightful heir to the throne. And he apparently did it on purpose to mess with his children. And then his bastard children starts pooping out in Merlin books. And if what Pattern Wraith based on Oberon says it's true, he has at last forty seven illegitimate children running around, that he knows about. It doesn't help that Corwin has a son with a woman from Courts Of Chaos Royal Family the same might be possible with Brand, which is exactly as much messed up, because their Oberon's equivalent, Swayvil, has the same damn idea how to mess with his kids. Not to mention that Oberon is son of Unicorn, avatar of Pattern (or vice versa) personification of order in The Multiverse, and Pattern's creator Dworkin, who comes from Courts Of Chaos. Luckily Oberon has forbidden incestous relationships or this would be even more messed up. And it wents that road pretty quickly in Merlin books, when Pattern forces Merlin to have sex with and possibly impregnating Coral, who is Oberon's illegitimate daughter and later gets in arranged marriage with Luke, who is Brand's son and therefore her newphew. And of course if Dara's words are true and she is Benedict's descedant, then the mess gets even bigger, as she gave a birth to her own uncle's son. Not to mention possibility that Jasra, Luke's mother, might be Dara's relative, which means he and Merlin are related from both sides... Thinking too much about it will make your head hurt.
The Count of Monte Cristo does have a handy chart. However, it also chronicles how characters interacted with each other throughout the novel. Hell if you can make heads or tails of it.
In Middlesex by Jeffrey Eugenides, Cal/Calliope's family tree is like this. His parents, Tessie and Milton, are 2nd cousins. His grandparents, Lefty and Desdemona, are actually brother and sister. Then there's the offhand mention early in the book that Lefty and Desdemona are also 3rd cousins...
In Richard Powell's Pioneer Go Home! the protagonists were from a New Jersey family so inbred that the narrator's father wasn't sure whether his relation to grade school-age twins Eddie and Teddy was grandfather or uncle. When a judge later commented that there was no real proof that he was directly related to the two of them, he replied that there was no real proof that he wasn't.
The Lord of the Rings: Arwen and her brothers have an extremely tangled family tree◊. Mostly due to the fact that their father Elrond is descended from all the important Elven and human families of the First age. Their maternal grandparents Galadriel and Celeborn are also related.
Arwen herself getting married to Aragorn, a descendant of her uncle Elros, arguably adds another knot to the tangle, but there are many, many generations between them.
Clary is raised by her mother Jocelyn, and thinks of her mother's devoted, if platonic, boyfriend Luke who is really a Shadowhunter named Lucian Graymark that was turned into a werewolf as her stepfather. He later gets a Relationship Upgrade with Jocelyn. She believes her biological father was a soldier killed in action, but he is really the Big Bad Valentine Morgenstern.
Jace is supposedly the son of Michael Wayland. But then it turns out that Michael was murdered and the man who supposedly fathered him was really Valentine Morgenstern, leading to a Brother–Sister Incest problem with Clary. It later turns out that his actual biological father was Valentine's right-hand man Stephen Herondale. Needless to say, Jace goes through a great many surname changes, although he often uses Lightwood after his adoptive parents and siblings. Worth noting that Stephen had previously been married to Luke's sister. Although after City of Heavenly Fire, he adopts his father's last name, Herondale.
Valentine did in fact have an actual son with Jocelyn, Jonathan, whom he raised in secret. Jonathan disguises himself as Sebastian Verlac, a cousin of the prominent Penhallow family. Later he magically coerces Jace into regarding him as a brother. Jonathan also has a Brother–Sister Incest vibe with Clary.
When they meet in Kushiel's Chosen, the second book in Jacqueline Carey's Kushiel's Legacy, Prince Severio Stregazza asked Phèdre, in an extremely annoyed tone, to which degree they were related, since every other nobleman/noblewoman he met at that ball seemed to actually be related to him, showing just how much tangled d'Angeline's nobility's bloodlines are.
Of course, when you see how some of them are horrified at the mere idea of the Queen marrying outside of their circle...
And let's not start on the Sharizai clan...
"A Tangled Web" by LM Montgomery Tells the story of 1 year in the lives of the Dark and Penhallow clans. It is said at one time they would have no-one else, now no-one else will have them. Aunt Becky is the matriarch she dies owning a heirloom jug. She leaves instructions on who is to get it, with the one person in the clan who can keep a secret. For one year - the family must wait, and the stress causes all sorts of problems.
The Rozes, a trio of giant/troll hybrids from the Garrett, P.I. series, claim to be "triplets with different mothers": a concept that Garrett doesn't even try asking them to explain. Not only that, but Garrett surmises that they're somehow distantly-related to his human/dark elf friend Morley.