Our dataviz blog neighbor Alberto reacted to this chart in the New York Times:
The title of his post makes two important points: "Don't just visualize uncertainty; explain it and don't let captions contradict it". The New York Times article is here.
***
Don't let your visual contradict your message.
This is a piece of advice embedded in the Trifecta Checkup (see the Guide here). Notice the green arrow between the Q and V corners. This arrow means to make sure that the answer to our Question and the Visual design are in sync with each other. The chart should convey our message, not contradict it.
Alberto pointed to the wide interval for the Independents in the top section of the graphic:
The chart assumes readers understand the concept of statistical significance (a form of uncertainty). The wider the interval, the less reliable is the result.
In the experiment that was run, Independents who read articles about the Democrats shifting leftward were 6 percent less likely to vote for a Democrat in 2020, compared to Independents who read articles of "unrelated" topics. However, the size of the experiment was small, and so the uncertainty of that 6 percent is high.
If a different random sample of participants were used, then we expect to see a different result: not necessarily 6 percent. The bar tells us that the percent could reasonably range from -12% to roughly zero %. In a sense, the story here is the effect is likely to be negative but we need more data (more participants) to know the extent of the harm.
When the right edge of the bar is so close to zero, this is not a strong result. This is what concerns Alberto.
Further, the other three bars cross the zero line. This means that the results for Democrats, Republicans and the average voter are not conclusive. The experiment does not show that reading the left-shift articles would affect how they would vote.
I've just made a video explaining statistical significance. Check back here in the next few days. Here is the video.
***
Looking at the entire set of results, every group crosses the zero line, and the other result worth noting - in the bottom chart - is that Democrats are marginally more likely to vote for their party's nominee in 2020 if they had read the articles about the left shift, compared to reading about "unrelated" topics.
These bars are relatively wide, and most results are not distinguishable from zero, which suggests that they need to expand the sample size.
***
There are other issues with the design of this experiment. It needs a "negative control". There should be a third test group that reads articles about the Republican party moving right. Surely, Independents also take into consideration what is going on in the Republican party.
Also, as stated in the first sentence of the New York Times piece, their real question is "the potential electoral penalty of repelling persuadable voters". The experiment however does not address persuadables in a direct way. This design flaw can be stated as such: not all persuadables are independents, and not all independents are persuadables. Beware of taking the results about Independents and substituting with persuadables.
I also have a small edit for the claim that "because of the random assignment [Ed: who reads which set of articles] ... the difference in responses between the groups can be attributed to the effect of reading about the leftward shift." This statement works only if we add "compared to reading about other topics labeled 'unrelated' in the experiment." We can't say anything about topics not used in the experiment, and as I pointed out above, we specifically don't know what the differential response would be against reading about the Republican party shifting rightward. Nor is it a given that all likely voters read the types of articles used in this experiment.
[PS. The video explaining "not statistically significant" is now up and running. It gives another example - using the recent story about Instagram running an experiment to hide the Likes counter as fodder. Here's the link. ]
Recent Comments