"If enough data is collected, a board of inquiry can prove anything."
— Murphy's Law of Combat #75
This well-known saying is part of a phrase often attributed to Benjamin Disraeli and popularized in the U.S. by Mark Twain:
"There are three kinds of falsehoods: lies, damned lies, and statistics."
Numbers and formulas are supposed to represent "objective scientific data" you cannot deny and examined by intelligent and experienced experts. Now the Consummate Liar wants his forgeries to look undeniably "scientific", so why not use the magic of numbers that the not-so-math-literate masses could never deny? They say that statistic don't lie, and while that may be true, liars do use statistics.
This trope covers all instances of Artistic License - Statistics where statistics are used to deceive people as to the truth. The problem is, people do not pay attention to the context, just the numbers. For example, the statement "Brand X is 84% fat-free" sounds good until you realize that this means the food product is 16% fat by weight. Also, "fastest growing" could mean that there used to be one customer and then there were five more, making a five-hundred percent increase. You should also notice Absolute Comparatives: it's fastest growing, but specifically compared to when/what?
The whole business of throwing percentages at people in advertising, politics and other forms of propaganda is almost destined for this kind of abuse. Relative measures are more likely to be understood accurately, and thus are less likely to be used in advertising.
The bogus uses of statistics are intended to imply a causal link between two elements when they are not linked, the link is questionable, or the link is opposite to what is implied. A beautiful example? "Coca-Cola causes drowning". By looking at statistics on drowning and Coca-Cola sales, you can see a link — more people go swimming on hot days, and more people buy Coke on hot days. Likewise, birth rates per head of population are higher in areas where there are more storks — because birth rates are always higher in rural areas, which is where one finds the Delivery Stork. Correlation does not equal causation; if it does, then we might also conclude that global warming is caused by a decline in pirate population and that 100% of Homo Sapiens who consume dihydrogen monoxide will cease vital functions and decompose.
Also be aware of the Law of Very Large Numbers. Any fraction of a very large number is likely to be a large number, no matter how small the fraction is. It is estimated that 2,135,000 Americans have used cocaine (including crack) in the past month. But that's only 0.7% of the population! So, is this a lot of people, or not?
You should also be on the lookout for the related effect where things are made more remarkable than they really are. The odds that any given ticket will win a raffle may be very small, but it is certain that one will be a winner. You'd notice being dealt a royal flush in spades at poker, but the odds of it happening are exactly the same as those for being dealt any other hand of five specified cards.
Then you can get the kind of statistical abuse in which you are careful to define the question to get you answer you want. What is the most popular book in the world? Depends if you mean most copies in existence (Quotations from Chairman Mao), most copies ever sold (The Bible) or fastest selling ever (Harry Potter And The Deathly Hallows).
Statistics are like studies: who made them and who paid them matters a lot. Want to "prove" that video games cause violence? Get a group of scientists that are already savvy to this and don't mind the lack of ethics. Have them draw from a very small pool of test subjects that are known to display violent behavior. Mental hospitals, prisons, schools for children with behavior disorders, what have you. Do some generic tests that are guaranteed to show up positive, come up with numbers, and presto, instant headline. "Recent test shows 77% of subjects become more violent after playing Mortal Kombat." Most people won't bother with reading the article the whole way through and will just look at the headline. This works with anything from comic books, and rock to watching Brokeback Mountain or voting for specific parties, basically anything.
Confirmation Bias, or the tendency for people to search out statistics that support their preconceived notions and ignore statistics that don't, is the reason for many of the entries on this list. The forgery mentioned above is also the reason most scientific and medical studies are done double-blind (meaning it's all anonymous, neither the researchers nor the participants know who's in the experimental group nor who's in the control group) and should allow for a chance at being falsified by Real Life (see also The Scientific Method). But you should beware of any advertisements touting a "double-blind" study, especially late-night ads, because they tend to violate the truth-in-advertising laws.
In the end, statistics are not lies and statistics don't lie: people lie about the statistic itself or how it is interpreted. Some don't lie, they are simply ignorant, as are most members of the public in terms of statistical interpretation. See Logical Fallacies and Critical Research Failure.
Put another way, by baseball announcer Vin Scully:
See also Nine out of Ten Doctors Agree, which is much a sub trope to this, and Absolute Comparative, where use of statistics is avoided entirely by comparing the product to nothing.
When adding examples, please remember that this trope is not about amusing statistical fallacies, but about using statistics to deceive people as to the truth.
For politics-related examples, you can find an abundant amount at PolitiFact (search for any Mostly False or lower verdicts) and Washington Post Fact Checker (3 and above Pinocchios)
During World War I, helmets were almost withdrawn from British soldiers. When Britain started issuing steel helmets to all soldiers on the western front in 1916, generals began to call for their removal as they increased incidences of headwounds twelvefold and doubled total casualties. The reason? If someone gets hit in the head by some woolly bear or flying frog (German H.E. or rifle grenade) shrapnel and lives it's a "head wound" and if they are unable to fight, the person is a "casualty"; if they die from a bullet in the brain, then they are a "fatality" and so don't appear on casualty statistics. Since helmets let more people survive, the number of head wounds soared. A politician used the the number to support his position that "helmets are expensive and cause cowardice", and never explained what it really meant - doubly effective as most people don't know the difference between "casualty" and "fatality". The real justification behind the attempt to withdraw helmets narrowed down to, "all a dead soldier needs is a funeral." A wounded soldier gets dragged out of combat by at least one of his buddies, and then provided weeks, months or even years of medical attention. From a statistical standpoint, adopting helmets drastically increased the effectiveness of enemy weapons - and a lot of WW 1 generals genuinely believed in disposablepersonnel. Luckily, more ethical parties changed the way they recorded casualties, or the helmets would likely have been recalled.
Likewise, the number of cyclists being treated for head wounds have increased massively since wearing a helmet became more widespread. Of course this is because they previously wouldn't have survived the accident at all.
Situations like the cyclists mentioned above also can also be attributed to the Peltzman Effect, aka Risk Compensation. This is the effect of a person being aware of greater safety ("I don't have to worry about hitting my head, I'm wearing a helmet!"), and taking greater risks due to the perceived increase in safety. So there actually could be an increase in injuries, due to more people taking greater risks.
Sick as it was, WW 1 quickly devolved into a stalemate, at which point it was clear that the loser would be whichever side ran out of men or money first. With sufficiently large draft pools, conserving resources was more valued strategically than conserving men.
Something of a historical subversion: During World War II, the Royal Air Force wanted to add more armor to their planes, but because of weight limits they needed to know which places needed the armor most. So, they examined the planes after they came back and counted how often bullet holes were found in certain areas... and then placed armor in places that showed the fewest bullet holes. This is because, they assumed, that any place that did have bullet holes was a place that planes could be hit and still fly. Helped by the fact: No plane that ever came back had holes where the gas tank was. Because planes whose tank was hit would explode and not come back.
It's a bit like the "statistics" on shark shows. "You are more likely to die on the toilet than be eaten by a shark." When you compare how much time you spend around sharks versus how much time you spend around toilets ... really, the toilet has time to plan out its move in advance.
Same deal with most accidents occurring in the home. Considering that you spend the majority of your time in your home, this should come as no surprise to anyone.
The same for the example above about most vehicular accidents occurring near the home (some say "within 25 miles from your home"). This is because most people do most of their driving near their homes, not that the home or the surrounding area is more dangerous than areas distant from the home.
At some Reform Judaism synagogues, a popular "joke" to lead into the sermon is, "x% of deaths occur in a hospital, x% of deaths occur in a car, x% of deaths happen in the home...[continues on for a while] while there have been only three deaths in a synagogue, and no deaths ever reported while studying Torah! Clearly, the safest passion, therefore, is studying Torah."
One of Bothered About Dungeons & Dragons (BADD)'s favorite weapons was a list of over 500 Americans they claimed were gamers who had committed suicide in the same year. Thus role playing games somehow cause suicide. Except that even if you take this bogus statistic at face value, 500 suicides a year is still a lower percentage of suicides than clergy and a tiny fraction of the average.
Patricia Pulling, the leader of this organization, once said in an interview that "8% of the Richmond VA-area population is involved with Satanic worship at some level." When asked where that figure came from, she said that she estimated that 4% of the teenagers and 4% of the adults. She then added them together and got the 8%.
Another time, BADD cited an increase in suicides corresponding with a major Dungeons and Dragons release. Again, however, there's no evidence that's not simply a coincidence, as similar statistics can be used to prove that the release of a Britney Spears CD caused suicide numbers to jump.
Newspapers love doing this with drug related stories. It's almost impossible to see Ecstasy mentioned in a British newspaper without the qualifier "That killer drug", the supporting statistic is that a dozen people die per year from consuming it. Yet over the course of a year they will rack up an impressive body count in stories about fatal car accidents without ever devolving into calling cars "Those murderous rampaging kill bots" How many Britons drive? How many take E? And how many deaths are caused by legal drugs? - cigarettes, alcohol, over the counter medications, misused prescriptions...
A related strategy was used by US president Richard Nixon to portray marijuana as a gateway drug. His anti-drug team estimated that 80% of marijuana users go on to use cocaine; actually, 80% of cocaine users had started with pot, but only about one in 2,400 marijuana users go on to use cocaine. And related to that, most people willing to take a drug as dangerous as cocaine are willing to take a drug as relatively safe as marijuana. It's the same reason most marijuana smokers have drunk alcohol at some point, and why most alcohol drinkers have drunk something with caffeine in it. Water: the Gateway Liquid.
"Nine out of ten dentists recommend Trident for their patients who chew gum." The tenth dentist was insistent that his patients never chew gum at all, but surprisingly, Trident didn't want you to know about that.
In a similar vein, a commercial for Five Hour Energy states that 4 out of 5 doctors wish for their patients who use energy supplements to use low-calorie energy supplements. Think about that: They specify patients who already use energy supplements, meaning they didn't count any doctors who recommend that their patients not use energy supplements at all.
Darrell Huff's "How To Lie With Statistics" was printed in the '50s. It's usually available on eBaystill in print and is a very easy read that shows you all the basics.
An Ars Technica article discussed the statistics usually used by software developers to complain about piracy. Specifically, the article pointed out that the statistics most commonly cited are most likely not only bullshit, but old bullshit. Amusingly enough, the image used for the related post on Gamepolitics was a pie chart divided into three sections, marked "Lies/Damned Lies/Statistics".
A Trojan Condoms commercial claims that the United States ranks between two African nations in HIV cases. This means nothing, since the population of the USA is much higher than either of those countries!
The Column 8 column in the Sydney Morning Herald once referenced a statistical correlation between the difficulty of the sudoku on a given day and the price of petrol.
When Ronald Reagan's Attorney General Edwin Meese wanted "proof" that pornography was evil, he created the Attorney General's Commission on Pornography. The commission members were a preselected cohort of anti-pornography campaigners. Not surprisingly, they discovered that statistics "proved" that pornography caused crime. However, the 1970 report of the President's Commission on Obscenity and Pornography, which was done by honest researchers and was highly praised for accuracy and honesty, discovered that there was "no evidence to date that exposure to explicit sexual materials plays a significant role in the causation of delinquent or criminal behavior among youths or adults."
There is a book produced for people in radio every year that compiles countless statistics about all stations taken from polls. These are used to attract advertisers. The less successful stations who have very few listeners are often forced to hire people who read through the book to get as many favorable statistics as possible, no matter how convoluted they may be. With the huge amount of data in the book, it's possible to say, for instance, that 85% of married men aged some arbitrary amount with income in some arbitrary range and who own a ferret will love your show, even though they represent a tiny proportion of the population. If you're selling ferret food, that's exactly whom you want to reach.
The Justice League was asked "Maybe you'd care to explain why on your watch, 50% of marriages now end in divorce, and the other 50% end in death!" Aside from the fact that the same was true before the formation of the League, until the end of time, a significant portion of marriages will end in death, as people do have a tendency to die, married or not.
In the heated German censorship debate about blocking sites allegedly containing child pornography, an organization in favor of this censorship law ordered a survey at a market research institute with questions asking if the person taking the survey is against child pornography and in favor of blocking the websites containing it. Over 90% answered 'Yes'. Another survey ordered by an opposing NGO — at the same institute no less — used a slightly different phrasing: Do you agree with blocking the content despite the fact that this content still exists and is easily accessible after the censorship? Over 90% answered with 'No'.
An old advert for Guinness ran with the quote "88.2% of statistics are made up on the spot", attributed to Vic Reeves.
Fletcher Knebel was apparently responsible for "Smoking is the leading cause of statistics", the most famous of which is "100% of non-smokers die".
In Montreal, there was an ad campaign run by a gum company whose gum came in round shapes instead of the usual square shapes. The ad said, "100% of people who chew square gum die."
Many casinos like to advertise their slot machines with lines like "Up To 99% Payout!" to make it sound like the player has a good chance to win. First, "up to" means the payout could be 1% for all you know (although laws usually set a minimum). Secondly, even a 99% payout means that for every $100 you put in the machine, on average, you'll get $99 back, i.e. you still lose. That "99% payout" is also an average that is based on something like one million pulls (plays) on the machine. If you play 100 times in one slot machine, you're not getting a representative sample of that average. These machines work differently in the UK. UK Fun With Prizes are required by law to seek their set hold percentage within a certain number of spins (usually 10,000). To achieve this, they naturally cheat all the time. They also can be, and often are, programmed to go on a suck cycle and take in way more money then they need to, in order to save up for a large series of payouts later.
A machine may have one payoff rate if you bet a single coin per spin (which most casual gamblers do) and a completely different rate if you bet the maximum coins per spin. For example, the payout for a video poker game might be $1000 for a royal flush if you bet one coin, but $7500 if you bet five. The advertised payout rates assume the player is playing maximum coins, so a casino can have a slot machine that has a completely legal and legitimate payout ratio of one hundred percent and still manage to make money on it.
Truth in Advertising laws require that if a set of machines is advertised as "Up to 99% Payout," then at least one of them must have 99% payout.
A common problem encountered is Simpson's Paradox, best demonstrated by example: Suppose Hospitals 1 and 2 are nearby, but 1 is better equipped for treating people with severe injuries, so proportionally more of the people taken there are badly hurt. It does better at treating badly hurt people than hospital 2, and also does better at treating people who are not badly hurt. However, since people who're badly hurt are more likely to die than people who're not badly hurt whether or not they go to hospital 1 or hospital 2, hospital 1 may still have a higher overall death rate.
Simpson's Paradox is when data shows one trend, but dividing it into categories shows the opposite trend. In the example above, hospital 1 has a higher death rate, but if the patients are split into categories based on severity of injury, it has a lower death rate in each category.
The same goes with good doctors and bad doctors, as told in the book Super Freakonomics. Good doctors are generally given tougher causes while bad doctors are given easier cases. However, if you look at death rates you see that some doctors have higher death rates, but these are usually the good doctors. Patients with serious cases are more likely to die, so good doctors lose a lot of their patients than, say the doctor who cures hiccups. The lesson is that you can be fairly certain that the doctor you receive at a hospital is competent enough to be assigned to you.
Ever wonder how all car insurance companies manage to to advertise that "people who switch from <competing company> to <our company> save an average of <large amount of money or substantial percentage>"? It's because the sample population "people who switch" is almost entirely composed of "people who are going to save a big chunk of money doing so", or else why would they bother to switch? Since no record is kept of the percentage of people who would not save any money and therefore don't switch, the cited statistic has almost no meaning.
As the Great Cable TV/Internet/Voice wars began ramping up, the tended to fish from similarly Small Reference Pools for their commercials, typically along the lines of "Of all customers who switched from <company> to <company>, over 53% that switched back did so because they realized they were paying less before." Of course this is said in rapid-fire and/or low-key speech, while the 53% is emblazoned on the screen, making you think that half of the people that switched had come back, instead of that being a sub-statistic OF those that switched back: that figure is never stated.
Italy got SŁdtirol, which used to be a part of Austria, to be added to their territory after World War II by using this kind of statistics to convince the Americans that the area was mostly populated by Italians. Which it wasn't. To this day, most of Sudtirol's population speaks German for a first language and watches German and Austrian TV, rather than the Italian channels.
Wolf Blitzer on polling information about the health care debate in American politics:
We did that poll CNN Opinion Research Poll, that said, "You like this health care bill or not like it?" We just assumed, a lot of us, that the people who said they didn't like it didn't like it because it was too much interference, or too much taxes or whatever. But if you take a closer look at people who didnít like it, about 12% of those people who said they didnít like it thought it didn't go far enough. They wanted a single payer option, they wanted the so-called public option, they didnít like not from the right, they didnít like it because it wasnít left or liberal enough. Thatís how you got 50% of the American people who said, "We donít like this plan." But only about 40 or 38% were the ones who said it was too much government interference.
A banner ad right here on TV Tropes promotes The Church of Happyology, rhetorically asking, "Why is it the world's fastest growing religion?" Hmmm, maybe because this religion that cannot be named for legal reasons was founded less than 100 years ago, and most major religions are thousands of years old and likely have grown as much as they ever will? Or because they count ad clicks as memberships.
In the 2004 US Presidential Election, Dick Cheney and John Edwards stated conflicting numbers regarding the Iraq war's casualties... and both men were partially right! Skip down to "90% of casualties".
One statistic used to justify the creation of the Comics Code Authority was that a large percentage of criminals liked to read comic books, ergo, comic books influenced people to become criminals. Nobody pointed out that they were using the wrong statistic - they should have been asking what percentage of regular comic book readers became criminals.
The Victorian-era belief that masturbation could drive men insane was derived from a similar error, in that mental asylums reported frequent Dates With Rosie Palms among inmates. The fact that mental illness can impair inhibitions against such behavior wasn't considered, nor the fact that men locked up in an asylum had few other respites from misery, frustration or boredom.
When Anthrocon decided to move from Philadelphia to Pittsburgh, one blogger who protested the move cited that only a tiny number of people of people would be as likely or more likely to attend Anthrocon if it moved to Pittsburgh. The organizers, however, heeded a different statistic: those that lived so far away that the move made little difference.
ABC loves to put out press releases trying to make the previous night's viewership of their shows look good. With their big hits, this isn't so bad; winning the timeslot in total viewers or Adults 18-49 (the demographic used to set ad rates, and thus the figure used to determine whether or not a show gets renewed) is definitely something to be pleased about (unless there's a huge skew between total and A18-49 viewership, but that's another matter). No matter how poorly-watched a show is, however, the ABC PR department can find some figure that sounds good but doesn't actually mean the show' doing well. They frequently put up demographics that aren't really indicative of a show's survival (e.g. women 18-34, or the adults 25-54 demo that only some cable channels use for ad rates), give the amount that the show built on its lead-in (usually when the lead-in was a repeat or another low-rated show, or a repeat of another low-rated show), or claim the the show had the best performance in its timeslot among ABC shows since X weeks/months/years ago (when you look at the absolute ratings, all it says is that ABC's done even worse in the past; this is rarely used for hit new shows since there are usually better statistics for those).
Programs on Animal Planet are fond of citing how Americans spend more money annually on cat or dog food than on baby food. This is depicted as evidence that Americans pamper their pets like babies, but overlooks several facts: that pets eat pet food for their entire lives, whereas babies only eat baby food for about a year and a half, and that many families have more than one pet at a time, but relatively few have more than one child of an age to eat baby food at the same time.
badscience.net occasionally shows how statistics get misused. For example, here (on small samples it's quite possible that B isn't significantly different from A or C, but you can put it as "B isn't different from A, C is different from A, so we see that C is different from B", which is wrong) and here (limit the view to one of many multipliers which per se can't prove anything). Unsurprisingly, the areas with traditional relations to snake oil trade suffer most.
QI is well known for deliberately phrasing questions like this in order to confuse the participants. One question was "What is three times more dangerous than war?" The answer given was work, because three times as many people are killed each year in work-related accidents than die in wars. Now, consider how much time you spent working last year compared to how long you were in a warzone. This prompted unhelpful responses from the panelists: "What if you're a soldier?" "What if you work in a shoe shop, near a war?"
An article was about a movement to shut down hospitals because "despite rapid advancement in medical technology, the world death rate remains at 100%."
Another article said that children are universally opposed to children's health care, with responses to the question "Do you want to go to the doctor?" ranging from "NO!!!!" to "inconsolable crying," but no children in favor.
There is a semi-famous magazine article from 1958 called "The Dread Tomato Addiction" that correlates consumption of tomatoes with everything from death to communism. It can be found here. There is a similar article about bread that can be seen here.
Listen to this: 40% of people support the Prime Minister and 50% support the Opposition.
That's only 90%.
Yeah, there was one guy who said the samples weren't big enough to be statistically significant.
Spoofed by America: The Book, which included a graph on "Growth of Misleading Charts". Two different bar heights represent the same number.
An old Archie story had one of the characters becoming a statistics-obsessed nut for the duration of the story, only for Jughead to start citing statistics that horrified them and lead them to run away in fright, at which point Forsythe noted that some ridiculously high percentage of people who quote statistics "make 'em up on the spot!"
The Penguin says this quote word-for-word in Detective Comics #684, at the same time pulling a You Have Failed Me on a newly-acquired henchman who, through usage of statistics, "proved" that a broad daylight robbery had a 0% chance of being foiled by Batman.
A commonly-cited factoid about the American Revolution is that roughly 1/3 of the residents of the Thirteen Colonies favored independence from Britain, 1/3 opposed it, and 1/3 were undecided or apathetic. The comedy series History Bites (based on the premise: what if TV had been around for 5,000 years) parodied Tom Paine as a spin-doctoring pundit:
TOM PAINE: Only 1/3 of the colonists are opposed to independence. Now, you can't let a minority opinion like that influence public policy!
INTERVIEWER: But the same number are in favor of independence.
TOM PAINE: But now we're talking half of decided voters, which is essentially a majority. You can't ignore the wishes of half of decided voters!
The original "statistic" doesn't come from any actual poll anyway... it was an estimation made by John Adams, and he admitted he'd not done any research on that, just that it was his feeling on the matter.
Penn And Teller Bullshit point it by having a man who makes pool research for the Republicans show he can make someone give two different answers to the same question by first asking: "Do you think the government expends too much in health care for immigrants?" The bystander answers "Yes". When he asks: "Would you deny an immigrant the right to treat himself? To give birth in a hospital? Etc..." and other medical services that go well beyond what the governments expends with immigrant health care, the answer now is: "No". Also, they make fun of the guy with his own mathematical wizardry by pointing out: "In this scene, ten cars pass by behind him. One guy from one of the cars shouts saying he sucks. This means that AT LEAST 10% of the American population believes he sucks".
There's a billboard along a freeway in Michigan, advertising a dentist willing to do implants for only a couple hundred dollars. Next to it is a statement saying "Voted best dental office in Michigan". Just below that, in smaller text, is the disclaimer "by our dental staff".
One episode of The Boondocks had Uncle Ruckus, an Uncle Tom racist, declare about the (black) man who invented peanut butter and hundreds of other uses for peanuts: "George Washington Carver was the man responsible for more peanut allergy deaths than any man in history!"
An episode of The Simpsons featured Homer forming a vigilante group to fight crime. At one point he recruits Jimbo (who calls the group "the drunken posse") on the basis that he can swing a sack full of doorknobs. Homer later gives an interview to the local news:
Kent Brockman: Mr. Simpson, how do you respond to the charges that petty vandalism such as graffiti is down eighty percent, while heavy sack beatings are up a shocking nine hundred percent?
Homer: Aw, people can come up with statistics to prove anything, Kent. Forfty percent of all people know that.
Two sports teams each only play one game in a season, and that game is against each other. How does the losing team make themselves sound good and their opponents sound bad? They say they only lost one game all season, and that their opponents only won one game all season.
Similarly in a two-horse race, one comes second and the other next to last.
Shizuo in Durarara!! maintains that the series's troll, Izaya, is behind every single weird thing in the setting. Sure, Shizuo can get irrational when mad, and will even use statistics and percents to maintain points. ...Did we mention Izaya's a troll?
In Dilbert, Dogbert once sold "value-priced lottery tickets;" half price, odds of winning only one in ten million less.
Sucker: Hey! This is for yesterday's lottery! Dogbert: And your point is...?
A stand-up comedian once said, "There is a direct correlation between being a serial killer, and being born. Show me one murderer who was never born."
In an episode of Psych, young Shawn attempts to justify his fear of shark attacks with the statistic that most shark attacks happen in about three feet of water. His dad points out that most swimming in general happens in about three feet of water.