"Oh, people can come up with statistics to prove anything, Kent. 14% of people know that."
~Homer Simpson
Often when reading the newspaper I am flummoxed by the treatment of numbers. Here is a recent example from the New York Times/ International Herald Tribune:~Homer Simpson
"The polls, taken for a local newspaper, use small samples, 500 people, limiting their usefulness as a gauge of popular sentiment in a city of one million."
Although numbers are often cited to provide evidence, they are often taken out of context or, just as bad, not given a context. Although editors seem keen to check the accuracy of quotes or the reliability of sources, they seem to skim over any mention of numbers. This is true of both op-ed opinion pieces and front-page investigations.
Besides teaching a research methods and statistics course, I also once worked for a public opinion polling firm, so the above quote struck me as incorrect on more than one level. So I wrote to the journalist:
"Your article on the mosque in Cologne misrepresents what is scientifically acceptable regarding sample sizes and population size. The whole point of a sample is to gauge the opinion of the greater population by taking advantage of probability. A sample of 500 is more than sufficient to do this--one need not ask every single inhabitant to infer their opinions! A properly created sample that randomly selects members of the population and does not seem to choose accidentally one segment of the population (e.g., the 500 polled all happen to be Turkish) is an excellent way to gauge the opinion at large with a confidence interval of +/- 4% to 5%. Unfortunately you do not provide the poll's results, just a summary that a majority support the mosque, so it is hard to apply the confidence interval here. The Times really should consider having a 'numbers czar' (maybe a colleague from the Science Times?) available for consultation on this and other issues that involve statistics and scientific literacy. The offending quote: 'The polls, taken for a local newspaper, use small samples, 500 people, limiting their usefulness as a gauge of popular sentiment in a city of one million.'"
The journalist was kind of enough to run a correction and write back to let me know:
"Your point was legitimate, as is your suggestion that we do a better job of vetting these issues before they get into print. We ran a correction on Saturday on the size-of-sample issue.
Best, Mark"
Here is the correction, with a link to the original article:
"An article on Thursday about a German backlash against plans for a mosque in Cologne, known for its Gothic cathedral, referred incorrectly to the size of polls taken for a local newspaper there, assessing the popularity of the mosque. The sample of 500 people was sufficient for a scientific poll; that sample was not "small," nor did its size limit the poll's "usefulness as a gauge of popular sentiment in a city of one million." (Go to Article)"
First, my thanks go to Mark for being open to the comment, and for having a correction appended. Second, I really do hope this paper and others consider having a staff member edit articles for numerical clarity. The readership deserves to have all the news be fit to print.
Technorati Tags: New York Times, International Herald Tribune, science, scientific literacy, statistics, Mark Landler, sampling, public opinion, Homer Simpson, Cologne, mosque
4 comments:
I find that newspapers rarely report statistics correctly. The sampling methodology is hardly ever described nor are we told the actual questions asked and the options respondents had for responding.
As a statistics person, that must drive you nuts! I agree with your point about not knowing the questions the reporters ask, too--that would be so useful!
Lies, more lies and statistics - attributed to Benjamin Disreali, Mark Twain and a few other people.
80% of statistics are made up on the spot - advert for Guiness beer.
I don't follow statistics much, but I know most surveys are badly worded. They way they ask questions determines the outcome.
Sometimes the question itself is loaded without any intention.
On the shooting people list, they asked if March of The Penguins deserved the award for best documentary.
What that question didn't take into account is 1) that there are four other films to choose from, and anyone who said "no it did not" might have just liked one of the other films better 2) that some dramatic filmmakers might be biased against a film that's about animals instead of humans (or about science instead of social issues) 3) that most normal people with normal opinions don't have time to answer surveys. 3) the way the question was asked already questions a validity that many people might not have previously questioned. In other words, the survey puts thoughts into people's heads that weren't there before.
I think it's much easier to survey the unemployed, bored housewives, and retired people than workaholics. Some personality disorders may be more willing to be surveyed, others may be less willing.
As election poles routinely show us, there are people who are too busy to answer questions but who still take the time to vote.
To add to all that, many people aren't entirely honest with the survey taker. Once they took a survey of us at school, and I lied in it to mess up their results on purpose.
A few times, I've seen people who've been paid on commision to survey me change my answers to fit their quota. For instance, when I didn't have a tv some woman was coming around to find out about video purchasing habits and how they relate to what shows people watch. Because few people were actually doing the survey for her, she was making up my answers as we went along.
"let's say you buy two or three."
Udigrudi--That's funny about March of the Penguins. I heard that the version in non-English countries was very different. While Morgan Freeman provided a serious, educational voice-over for the English version, apparently the original French version was more humorous with a voice-over providing voices for the penguins rather than information on evolution. Would it still be the "best" documentary if they had done a direct translation?
Post a Comment