Personal Experience vs. Statistical Evidence

This flies around a lot in the debates here, and I thought we should discuss the matter. Pros and cons of each as I see them;

Statistical Evidence

pros:

• You can cite your source, so both sides can read the data and draw conclusions from it.
• If properly applied, scientific method is argument-neutral and as such provides a stable base for both sides of a debate to draw on, at least in theory.

Cons:

• Often how the data was collected is not shown/glossed over, so it is tough to know whether the data actually applies to an argument or not.
• Numbers can be twisted around to any purpose the user desires; this was one of the first things I was taught in an "intro to statistics" class, and I think fellow tropers who have been educated in math can back me on that.

Personal Experience

pros:

• Personal experience is more vivid and nuanced than simple numbers.
• It reminds us that exceptions exist for any rule, and not to over-generalize.
• It shows us the difference between theory and practice.

cons:

• Generally speaking the same source cannot be cited by opposing sides, so deadlocks often result unless the debaters are mature enough to "agree to disagree" when such comes up.
• Personal experience can be distorted by the individual's own preconceived notions on the matter, rendering the assertion biased.

My opinion on the matter: Personal experience gets irrationally discriminated against, and that attitude encourages the thinking of "I can learn everything I need to know about the world from the comfort of my computer chair". Attempts to bring in personal experience or knowledge into a debate is often met with scorn or derision, or a demand that you "cite your source."

As I said above, numbers can be twisted around to say anything you want. So can personal experience, but that doesn't make either any less valid.

Discuss.

If I were to write some of the strange things that come under my eyes they would not be believed. ~Cora M. Strayer~

^ That whole post is pretty much 100% spot on.

Welcome back

Blind Final Fantasy 6 Let's Play

@Tom: Glad to see you as well. So, anything I got wrong? And what's your stance on the matter?

If I were to write some of the strange things that come under my eyes they would not be believed. ~Cora M. Strayer~

IMHO, statistical evidence is only useful in its entirely when the results it provides can be mapped back without change to individuals. A statistic indicator that the average family has, say, found happiness with a minimum wage of $X currency is of no practical use if the indicator relies on another statistic that the average family has 2.13 children. Unfortunately, it is precisely such statistics that are used to give birth to terribly flawed social management politics.

On the other hand, personal (empirical?) evidence by definition can not have value in and of itself. "This happened to me" is of no practical use; it can only be rendered useful once it is contrasted with a large or distinct enough amount of personal information. "This happened to me, but that other guy did manage to catch the ball because he has longer arms" is pretty useful.

Fanfic Recs orwellianretcon'd: cutlocked for committee or for Google?

Is it possible for personal experience to outdo statistical evidence in an argument?

Anime geemu wo shinasai!

@Grain: Depends on who's involved with the argument. On the internet, not in the slightest, considering how it's generally held to be bad form to bring anecdote into anything.

"I don't know how I do it. I'm like the Mr. Bean of sex." -Drunkscriblerian

@Grain: If the science is bad enough or the connection tenuous enough in the statistical argument, then yes. I think the best arguments contain a mixture of both.

The weakness of personal experience is mitigated by realizing that your experience isn't the only one that exists, and your reactions to it aren't the only ones possible. Again, it is all in how maturely you apply it.

If I were to write some of the strange things that come under my eyes they would not be believed. ~Cora M. Strayer~

They're both equally important imo. Both of them can give lots of information in their own ways. And both methods can be incredibly biased. But if you do both of them enough time, eventually you find a pattern.

And bias isn't necessarily a bad thing. If you can tell which individuals are known to be biased, you can eventually get a good pattern going from them as well as long as you know what you are doing.

"Every opinion that isn't mine is subjected to Your Mileage May Vary."

There's also personal experience that isn't just one anecdote. Say, people who work in the field under discussion, and while they haven't measured effect exactly, will still notice some definite trends.

Be not afraid...

I would call such a thing a "statistic" not "statistical evidence". That's why checking the source is important - there's a reason researches output a side long paper and not just a three line summary. The same as above also applies here.
If only the raw data (maybe after some processing through well known mathematical methods) is used for the argument, and not the interpretation of that raw data, then it's no problem.
(Actual example I had some time ago:)
Wrong: "Only 2% of women make false rape accusations"
Less wrong: "2% of the women who make rape accusations were threatened with charges if they wouldn't withdraw the accusations"
The data doesn't actually support the conclusion that got used as an argument. In fact all accusations could be true, and those women were falsely forced to withdraw.

Of course, when people rely on statistical arguments they normally only refer to the first form, so...

After trying to defend statistical evidence, I'll try to defend anecdotal evidence:
There's a math joke that goes: "During a lecture a math student tells the professor: 'Sir, I got a counter-example for your statement' 'Doesn't matter, I have two proofs!' "
If someone makes a statement that wouldn't allow something to happen but you experienced it, it means that the other one was wrong - plain and simple. He can try to pull argument after argument, but a single counter-example is sufficient to disprove an absolute statement.

In an argument, statistical evidence and personal experience serve imho two different purposes and they don't really conflict. Using personal experience to "disprove" statistical evidence is missing the point just as much as using statistical evidence to "disprove" personal experience.

Pour y voir clair, il suffit souvent de changer la direction de son regard www.xkcd.com/386/

Pretty much what I was trying to get at. However, there does seem to be an irrational level of faith in statistics and published information as it pertains to arguing via the internet, for all that any idiot can post anything he wants to online and call it a "study".

That and (OMG personal experience talking!) I've had my personal experience marginalized a great deal, to the point where I've started returning the favor (e.g. "go get out and live a little before you make such claims, young'un")

If I were to write some of the strange things that come under my eyes they would not be believed. ~Cora M. Strayer~

Hm... I really would like to know the context of the events.
If you try to use personal experience "instead" of a statistic, it means you are just using a statistic but done badly. In general low sample size (In case of single event it's a sample size of ONE, which is horrible) and no or at least less controlling for different factors (For example your influence on the events, the locations where you tend to be etc etc).

In general you are doing everything a statistic does only worse.

That's why I would like to know the context of the criticism you faced. If you were using your experience as a substitute to a better done statistic, they were imho right on dismissing it. If you used it like "according to you this could never happen but it happened to me" they were wrong.

Pour y voir clair, il suffit souvent de changer la direction de son regard www.xkcd.com/386/

In general, I tend to find that bringing anecdote into a discussion results in people going "Well, that's not what the statistics say", even if the purpose of the anecdote wasn't to disprove or deny anything, just to illustrate.

"I don't know how I do it. I'm like the Mr. Bean of sex." -Drunkscriblerian

@U (sorry not even going to attempt to spell that): Basically, people were making sweeping generalizations regarding racism and white privilege and I said "No, that's not how it works everywhere because that's not how it works where I live (description)", and they basically said "exception does not prove anything".

Also claiming "sampling bias" when all was trying to say was "well, I don't know what the numbers say but this is my experience on the matter" regarding things I am intimately familiar with.

edited 26th Feb '11 6:18:44 PM by drunkscriblerian

If I were to write some of the strange things that come under my eyes they would not be believed. ~Cora M. Strayer~

Both sets of data are useful but have their flaws and limits. Neither should be fully discredited or dismissed either.

Who watches the watchmen?

Copy+Paste is your friend for my name ;) (I don't really care though, call me whatever you like ^^)

I think I might even remember that post, was it on tvtropes?
Well, you remember the personal experience = statistic thing I said earlier? If I'm not confusing it you were basically using a whole city or village for your anecdote, which is imho enough data that it shouldn't be dismissed easily. Even if it's a bad (or "not that good") statistic, a model that cannot explain this satisfactory is at least a flawed model.

If the theory doesn't equal the reality, it means the theory is flawed. They should basically have the same results.
But in general people (i.e. WE) like to dismiss anything that doesn't support our point of view - personal experience as a rather weak set of data is just an easy target. Though I would agree that many "studies" that get cited aren't worth their space on the screen, especially because normally only the (biased) interpretations get cited, not the actual data.

edited 26th Feb '11 6:58:37 PM by Uchuujinsan

Pour y voir clair, il suffit souvent de changer la direction de son regard www.xkcd.com/386/

@U: It was in the "Racial Olympics" thread or whatever, though its an argument I've gotten into several times on this forum.

I somewhat agree with you; I'm willing to admit that what's true for me isn't true for everyone (durr), but the complete and utter dismissal of personal experience...yeah, I take issue with that. As I said in the OP, it encourages people to just sit in front of their computers, look up things on the Internet and think they know every damn thing.

And as you said, when someone makes a generalization or absolute statement, popping to say "that isn't true for me" should not get shouted down as much as it does.

If I were to write some of the strange things that come under my eyes they would not be believed. ~Cora M. Strayer~

I remembered it because I noted how informative it was relative to "normal" anecdotes - I also disagreed with the reaction.
(On a completely irrevelant side note, I hate having to check some vocabulary just to ensure that my German isn't polluting my English too much -.- )

Pour y voir clair, il suffit souvent de changer la direction de son regard www.xkcd.com/386/

@U: I assumed you were not an English native, because your speech was intelligent despite your spelling. And no, you're perfectly understandable.

Yes, I grok the limitations of personal experience. I just wish others would recognize the utility.

If I were to write some of the strange things that come under my eyes they would not be believed. ~Cora M. Strayer~

Oh my god, "worse" instead of "worth", "the" instead of "that"...

But I have an excuse, it's 4 am here :>

(Ok, I'll stop now with this OT stuff)

Pour y voir clair, il suffit souvent de changer la direction de son regard www.xkcd.com/386/

This may not be the best example, as I'd bet the professor is probably right. All we really know is that at least one of the people made a mistake, and without extra context, it's much more likely to be the student than the professor.

edited 26th Feb '11 7:06:20 PM by storyyeller

Blind Final Fantasy 6 Let's Play

The professor is certainly wrong, because having "two proofs" does jack shit against a counter example. Which basically is the joke. :>

Pour y voir clair, il suffit souvent de changer la direction de son regard www.xkcd.com/386/

But a "counter example" also does jack shit against a proof.

All we really know is that at least one of them is mistaken. And I don't think it's the professor.

edited 26th Feb '11 7:08:57 PM by storyyeller

Blind Final Fantasy 6 Let's Play

All I remember from precalculus is how to 2+2 =/= 4.

"I don't know how I do it. I'm like the Mr. Bean of sex." -Drunkscriblerian