In the several prior posts (e.g. here and here), I went into the weeds to describe how a data scientist draws conclusions about specific questions using a data set. In this post, I want to take a step back and draw some high-level lessons from exit polling data. In particular, these exit polls are a perfect vehicle to learn about statistical biases.
What is a bias?
Bias is when you believe the results from your poll (or survey) do not match the unknowable true results of your population (ground truth). In an exit poll of a state primary election, one ground truth is the actual vote share for each candidate in the entire state.
In a perfect world, every poll is designed properly, all polls are executed as designed, and there will never be any bias. Statistical theory proves it so. Alas, the real world intrudes.
Sources of Biases
In the world we live in, polls suffer from a variety of biases, which cause poll results to deviate from the ground truth.
1) Who are selected for the poll?
No exit poll operation can cover the entire state. Iowa for example had over 2,000 caucus sites. So the pollster selects a subset of polling stations at which they field interviewers who approach voters as they exit to ask them to fill out the questionnaire.
The trouble is that polling stations are not created equal. The pollster uses a variety of assumptions to select a set of polling stations that when taken together may approximate the general population of voters. It goes without saying this is not an exact science.
Bias in the selection directly results in bias in the exit poll results. There is no perfect design so pollsters manipulate the raw data to correct any known (or presumed) biases. [See section on Statistical Adjustment below.]
Think about the youth vote. College-age voters might be highly concentrated at polling stations on college campuses. These sites have a monolithic voter type that is hard to work with. One must select campus sites to learn about college students but what proportion of these voters should be polled? It's a chicken-and-egg problem. The rate of selection should be based on an assumed turnout model but if the turnout model is wrong, then the sample becomes unbalanced. [This is precisely why you should not use exit polls to talk about turnout trends. More later.]
Unfortunately, the sole vendor for exit polls does not disclose data that allow an outside to assess its selection bias. We should demand better transparency.
2) Who refuses the poll?
Another unavoidable bias is technically known as non-response. Imagine you are an interviewer at one of the selected polling sites, handing the questionnaire to every 10 or 50 voters exiting. Some proportion refuse while others don't fill out the whole thing. According to past data, less than 50 percent of approached voters complete the questionnaires.
The non-response bias is also a type of selection bias, only that the respondents are doing the selecting, as opposed to the statisticians. If there are certain types of people (say, younger/older, male/female, certain racial groups) who tend to refuse exit polls, then their answers will be under-represented in exit poll data, causing inaccuracy.
As with all biases, if we can see the bias and can estimate its magnitude, we can correct for it. The pollsters attempt to adjust for non-response bias by having the interviewers "guess" the race, gender and age of the refusing voters. How accurate is such guessing? I'll leave you to decide. One point is clear, though: that adjustments are well-intentioned but can backfire; it's totally possible that the adjusted data are further from the ground truth than the raw data!
3) Who is reachable?
The traditional way of sampling voters exiting the polling stations fails to capture the growing proportion of voters who submit mail-in ballots, absentee ballots, online votes, etc. Pollsters compensate for this by augmenting the base sample with data from supplemental outreach via other channels such as phones. Polls by phone now have response rates below 10 percent.
Needless to say, the refusal and non-contact rates of such outreach are much higher, and it's hard to believe that any pollsters will nail this adjustment without a lot of on-the-ground learning over multiple elections cycles.
Even predicting what proportion of voters will vote virtually is a challenge, let alone trying to deliver them questionnaires, and hoping they will return them.
Reports of extremely long lines on college campuses, and in areas where the Party decided to close a large number of polling stations wreak havoc to any pre-designed exit polling plan. If the pollsters did not extend the interviewers' schedules, or re-deploy resources from closed sites to other sites, that's another source of bias.
Again, the entire exit poll operation is cloaked in secrecy so we don't know enough to trust these results.
4) What questions are asked?
Some questions in a poll are more knowable. Who did someone vote for has a truthful answer. Other questions are subjective. Are you liberal or moderate? That's one of the key questions in the past two election cycles.
Here is the question as it shows up in the exit polls:
This question is subjective since the terms "liberal" and "moderate" mean different things to different people. A case in point: opponents of Bernie Sanders and the media portray him and his policies as "extreme" and "radical" while his supporters believe they are middle of the road, representative of all advanced economies around the world.
Secondly, notice that "moderate" is not in the middle of the set of answers. An alternative wording for the choices is: "Very liberal", "liberal", "conservative", "very conservative". In designing answer choices, typically symmetry is preferred.
The design used by the exit polling firm can be interpreted as 2 levels of liberalism, moderate and one level of conservatism. It splits up liberals into two groups, possibly even three since "moderate" could be interpreted as neither liberal nor conservative.
Statistical Adjustments
Real-world polls are biased for a variety of reasons. If the raw data from these polls are directly analyzed, the results would not mirror the ground truth. So, pollsters have many techniques to adjust the results - such adjustments are intended to move the raw statistics closer to the ground truth, but if the wrong assumptions are used, it is definitely possible to spoil them further.
In order to correct for bias, we have to measure it. If bias is the gap between the ground truth and the poll data, wouldn't it be necessary to know the ground truth in order to measure the bias? But isn't the ground truth what the polls are designed to learn?
To make this concrete, let's say the exit poll raw data say the voters split 60% male, and 40% female. A naive conclusion is that the exit poll sample is representative of the population of voters, and so in the population, 6 out of 10 voters are men.
That would be the conclusion if the sample was unbiased. To even know that there is bias requires that we know something about the ground truth! If we knew the male-female split from past election cycles was 40%-60%, then we can say the exit poll's estimate of the male-female ratio in the population is biased, undercounting females and overcounting males.
The statistical adjustment is a set of weights applied to each respondent to the exit poll. In this case, we downweight males and upweight females so that the male-female split after applying weights becomes 40%-60%. In practice, the weight for a male is 40%/60% = 0.67 and the weight for a female is 60%/40% = 1.5.
You should immediately notice two effects of such weighting:
- Weighting is intended to move raw data toward the ground truth. It requires knowing the ground truth. Since we don't know the ground truth, we "guess" it. Guesses are usually based on historical norms. It could also be based on supplementary polling. For example, a poll conducted before the election might suggest more female voters than the historical norm.
- After the weighting, the male-female split is essentially an assumption, not a measurement. The assumption of the weighting formula is that the historical norms and/or prior polling are fine, that there are no surprises. For this very reason, you cannot use exit polls to comment on changes in male-female turnout that has not already been accounted for by the weights.
Again, let's be concrete. Say the historical norm for the male-female split is 50%-50%. Prior to running the exit polls, we learn from the latest polling that there might be a female voter surge. Accordingly, we set the weights so that after adjusting the raw exit poll data, the male-female ratio will be 40%-60%. The raw data on election day came out to be 35%-65%. The reported adjusted data show 40%-60%.
Yes, the exit poll says female voters increased as a proportion of total voters by 10% above historical norms. Notice that you don't need to run the exit poll to know this. The 10% increase is built into the weighting formula! The raw data from the exit poll show a 15% increase; the statistical adjustment brings it down to 10%. The 10% came from prior polls. It's like a game of telephone where the first person says a rumor to the second person, the second person repeats it to the third person, and the third person delivers the rumor to everyone as fact.
Being a statistician, I am not here to denigrate polling. The above process works so long as the assumptions built into the weighting formula are valid. In this case, since we will never find out the ground truth, it's impossible to know if those assumptions are valid. So, my advice is to ignore exit polls. (Nate Silver used to be a critic of exit polls, but appears to be a reformed consumer.)
The other reason to ignore them is because the process is so opaque. It's a black box. We don't know the weighting formulas, we are only provided the weighted data, we can't compare the weighted data to the raw data, we don't know which polling stations they selected, we don't know response rates, refusal rates, etc.
It doesn't help that exit polls have a horrible record of doing the one thing it was designed to do - give an early estimate of the vote shares at the aggregate level. According to some analysts, this year's Democratic primary exit polls reported huge discrepancies with the official vote shares. You either have to believe in election fraud or that exit polls can't be trusted.
The problem I have with mainstream media pundits is that they are simultaneously (a) denying there could be fraud and (b) building elaborate stories based on cherry-picked exit-poll data. Essentially, any story about voters that break them up by demographic or ideological segments is one that relies only on exit polls. There is no other way to know who voted for whom. But if these pundits actually believe in the exit polls, then they must also believe that the official election results are rigged.
***
Back to statistical adjustments. After addressing the gender bias, you look at the age distribution. Oops, the age groups seen in the exit poll data after adjusting for gender do not match the expectation based on historical norm. You then undergo the same process to move the age group distribution closer to the "ground truth".
(You may notice that any age adjustment you make now will cause the gender adjustment to deviate from the optimal.)
In practice, you identify all the distributions you want to match to ground truths, and simultaneously try to match them all. This will not be exactly possible but you can minimize the difference between the raw data and those norms by setting a weight for each respondent.
The more variables you use to re-weight the raw data, the more assumptions you're making. The adjusted poll data reflect the historical norms and/or prior polling expectations. If the raw data deviate from those assumptions, they are re-weighted to match them. Therefore, and this is the most important point, it is entirely circular to use exit poll data to comment on demographic or ideological trends.
Yes, those trends may exist but the evidence of those trends would not have come from the exit polls. The evidence would have existed at the time the weights were determined for the exit polls before they were even run.
***
Oh, there are elections today despite the CDC's warning to avoid crowds of more than 10 people. This means the pundits will be polluting the airwaves with their detailed analysis of demographics and ideologies, all based on exit polls. You now know why you should just mute the volume. Any such comments are based on past assumptions and not on today's data. Also, if there are discrepancies between the exit poll results and official vote shares, these pundits must disclose whether they trust the exit polls or not. They can't have it both ways.
Comments