Vanity Fair asks the wrong questions; overreaches with results

The March 2012 60 Minutes/Vanity Fair Poll

The results from a series of questions published in the March issue of Vanity Fair should prompt some pauses from any critical thinkers.

The first question that caught my eye was this one:

Which mythical creature do you think is most likely to actually exist?

Results (total):

Bigfoot 40%
Loch Ness Monster 20%
Unicorns 7%
Vampires 6%
Fairies 4%

So, here are the problems I have with this:

They have assumed Bigfoot is mythical, that is an oversimplification. Then they assert that it exists. How confusing to the person answering the question.
The question is framed as if out of all of these creatures, which is most likely. So, why don’t the percentages add up to 100% if someone had to pick one?
What did the leftover 23% answer?
What’s the margin of error?
How do we know this sample is actually random?
What’s a unicorn? (Is it a horse with a horn? Does it have magical powers? Is it a rhinoceros?) For that matter, define a vampire.
Who the heck thinks fairies exist?

I could go on but it gets worse. The results are broken down into those with college degrees and those without. Vanity Fair notes:

…getting a college degree will also make it five times more likely that you will believe in fairies than if you, say, hadn’t done something about that “incomplete.”

What? An increase from 2 to 10%. What was the margin of error?

In consideration of the other questions (non cryptozoological), they are phrased just as unclearly. Worse, the authors make sweeping conclusions from the results regarding older Americans.

The lesson here is that survey questions must be VERY carefully constructed to give you a meaningful data. They are easily skewed (which is the case here because these questions are phrased terribly) and people can misconstrue what is even being asked. Second, we don’t really know how the survey was conducted at all. But people are given the results framed as “isn’t this interesting”.

All that together makes for a jumbled, useless pile of garbage. This poll is worthless. I suggest you discard the results. But, I wanted to make a point that you should be VERY VERY skeptical of such nonsensical data tables presented in popular media and the (unwarranted) conclusions that are drawn from them.

Tip: @bigfootevidence

  3 comments for “Vanity Fair asks the wrong questions; overreaches with results

  1. January 30, 2012 at 10:30 AM

    I haven’t got that edition (I usually get it a week or more after it’s released in the UK), but I’ll keep an eye out for this. Thanks!

  2. Massachusetts
    January 30, 2012 at 6:48 PM

    I strongly suspect that giving accurate information is the last thing on their agenda. They probably want a short, fun, entertaining article to fill a little space, and they are probably working on a tight deadline. They probably believe that the results won’t really matter to anyone in a serious way, so they don’t put a lot of effort into the piece or the research (if any) behind it.

    But more significantly, most journalists probably don’t have a good grasp of the validity and accuracy issues raised, and they are in danger of reporting poorly on topics that matter more, like vaccinations and other public health matters.

  3. Massachusetts
    January 30, 2012 at 6:49 PM

    By the way, I particularly like how you pointed out that the results don’t add up to 100%. Interesting point.

Comments are closed.