"Oh, people can come up with statistics to prove anything, Kent. 14% of people know that."
~Homer Simpson
Often when reading the newspaper I am flummoxed by the treatment of numbers. Here is a recent example from the New York Times/ International Herald Tribune:~Homer Simpson
"The polls, taken for a local newspaper, use small samples, 500 people, limiting their usefulness as a gauge of popular sentiment in a city of one million."
Although numbers are often cited to provide evidence, they are often taken out of context or, just as bad, not given a context. Although editors seem keen to check the accuracy of quotes or the reliability of sources, they seem to skim over any mention of numbers. This is true of both op-ed opinion pieces and front-page investigations.
Besides teaching a research methods and statistics course, I also once worked for a public opinion polling firm, so the above quote struck me as incorrect on more than one level. So I wrote to the journalist:
"Your article on the mosque in Cologne misrepresents what is scientifically acceptable regarding sample sizes and population size. The whole point of a sample is to gauge the opinion of the greater population by taking advantage of probability. A sample of 500 is more than sufficient to do this--one need not ask every single inhabitant to infer their opinions! A properly created sample that randomly selects members of the population and does not seem to choose accidentally one segment of the population (e.g., the 500 polled all happen to be Turkish) is an excellent way to gauge the opinion at large with a confidence interval of +/- 4% to 5%. Unfortunately you do not provide the poll's results, just a summary that a majority support the mosque, so it is hard to apply the confidence interval here. The Times really should consider having a 'numbers czar' (maybe a colleague from the Science Times?) available for consultation on this and other issues that involve statistics and scientific literacy. The offending quote: 'The polls, taken for a local newspaper, use small samples, 500 people, limiting their usefulness as a gauge of popular sentiment in a city of one million.'"
The journalist was kind of enough to run a correction and write back to let me know:
"Your point was legitimate, as is your suggestion that we do a better job of vetting these issues before they get into print. We ran a correction on Saturday on the size-of-sample issue.
Best, Mark"
Here is the correction, with a link to the original article:
"An article on Thursday about a German backlash against plans for a mosque in Cologne, known for its Gothic cathedral, referred incorrectly to the size of polls taken for a local newspaper there, assessing the popularity of the mosque. The sample of 500 people was sufficient for a scientific poll; that sample was not "small," nor did its size limit the poll's "usefulness as a gauge of popular sentiment in a city of one million." (Go to Article)"
First, my thanks go to Mark for being open to the comment, and for having a correction appended. Second, I really do hope this paper and others consider having a staff member edit articles for numerical clarity. The readership deserves to have all the news be fit to print.
Technorati Tags: New York Times, International Herald Tribune, science, scientific literacy, statistics, Mark Landler, sampling, public opinion, Homer Simpson, Cologne, mosque
