As the U.S. enters election year, the press will be reporting on polls. Sadly, it is still not mainstream to prominently display margins of error, or to draw conclusions based on interpreting these margins. So readers must take this matter into their own hands.
Polling is an exercise in generalization: the goal is to learn the average opinion of a large group of people without having to solicit each person's opinion. If one conducted the same poll twice on different random samples of people, the results would vary since different individuals would have answered the poll. Margins of error account for such variability, and provide a simulation of what the range of results might have been if the poll were run repeatedly.
If the difference between two candidates is less than twice the margin of error, then the two candidates are in a "dead heat". A dead heat just means that this particular poll can't tell those two candidates apart - with the required degree of confidence.
***
The media routinely makes a mess of poll numbers.
This last week, Reuters led with the headline "Sanders leads, with Warren, Buttigieg, Biden chasing in Iowa democratic poll" (link). The poll results:
Sanders - 20%
Warren - 17%
Buttigieg - 16%
Biden - 15%
More specifically, the reporter said "Sanders received support from 20% of respondents in the poll, with the next three candidates - U.S. Senator Elizabeth Warren, former South Bend, Indiana Mayor Pete Buttigieg and former Vice President Joe Biden - in a statistical tie behind him in the poll." (my bolding)
This is a rare example of news media declaring a difference as insignificant. But don't lose sight of the pairwise differences: Sanders-Biden is 5%, Sanders-Warren is 3% and Buttigieg-Biden is 1%.
Later in the article, the margin of error is reported as 3.7%. Twice the margin of error is about 8%. So the poll really can't distinguish between any of the top 4 candidates.
The candidates outside the top 4 are said to have scored below 10%. For a candidate at 10%, this poll gives Sanders a difference of 10% which is significant; against Biden, the difference is only 5% which is a statistical tie.
***
Compare the above report with an earlier report also by Reuters, from August 2019 (link). This article led with "The latest Reuters/Ipsos poll finds Biden leads Democrats as minorities favor most electable candidate versus Trump".
What are the poll numbers that prompted this headline?
Biden - 22%
Sanders - 18%
Margin of error? 5%. So twice the margin of error is 10%. A 4% difference against 10% is definitely a "statistical tie". And yet, unlike the first article, this one ignored the statistical tie, and declared Biden the front-runner.
***
It is preferable to interpret collections of polls instead of individual polls. Over at FiveThirtyEight, Nate Silver doesn't make headlines based on one poll. See his latest view on the Democratic race.
P.S. It's unclear to me whether the margin of error cited is for an estimate of single proportions or difference in proportions. If the former, then these polls are even less precise than indicated above. Perhaps we have readers who are polling experts who can clarify...
Comments