Chopped legs, and abridged analyses

Reader Glenn T. was not impressed by the graphical talent on display in the following column chart (and others) in a Monkey Cage post in the Washington Post:

Not starting column charts at zero is like having one's legs chopped off. Here's an animated gif to show what's taking place: (you may need to click on it to see the animation)

Since all four numbers show up on the chart itself, there is no need to consult the vertical axis.

I wish they used a structured color coding to help fast comprehension of the key points.

***

These authors focus their attention on the effect of the "black or white cue" but the other effect of Trump supporters vs. non-supporters is many times as big.

Notice that on average 56% of Trump supporters in this study oppose mortgage assistance while 25% of non Trump supporters oppose it - a gap of about 30%.

If we are to interpret the roughly +/- 5% swing attributed to black/white cues as "racist" behavior on the part of Trump supporters, then the +/- 3% swing on the part of non-Trump supporters in the other direction should be regarded as a kind of "reverse racist" behavior. No?

So from this experiment, one should not conclude that Trump voters are racist, which is what the authors are implying. Trump voters have many reasons to oppose mortgage assistance, and racist reaction to pictures of black and white people has only a small part of play in it.

***

The reporting of the experimental results irks me in other ways.

The headline claimed that "we showed Trump voters photos of black and white Americans." That is a less than accurate description of the experiment and subsequent analysis. The authors removed all non-white Trump voters from the analysis, so they are only talking about white Trump voters.

Also, I really, really dislike the following line:

When we control for age, income, sex, education, party identification, ideology, whether the respondent was unemployed, and perceptions of the national economy — other factors that might shape attitudes about mortgage relief — our results were the same.                        

Those are eight variables they looked into for which they provided zero details. If they investigated "interaction" effects, only of pairs of variables, that would add another 28 dimensions for which they provided zero information.

The claim that "our results were the same" tells me nothing! It is hard for me to imagine that the set of 8+28 variables described above yielded exactly zero insights.

Even if there were no additional insights, I would still like to see the more sophisticated analysis that controls for all those variables that, as they admitted, shape attitudes about mortgage relief. After all, the results are "the same" so the researcher should be indifferent between the simple and the sophisticated analyses.

In the old days of printed paper, I can understand why journal editors are reluctant to print all those analyses. In the Internet age, we should put those analyses online, providing a link to supplementary materials for those who want to dig deeper.

***

On average, 56 percent of white Trump voters oppose mortgage relief. Add another 3-5 percent (rounding error) if they were cued with an image of a black person. The trouble here is that 90% of the white Trump voting respondents could have been unaffected by the racial cue and the result still holds.

While the effect may be "statistically significant" (implied but not stated by the authors), it represents a small shift in the average attitude. The fact that the "average person" responded to the racial cue does not imply that most people responded to it.

The last two issues I raised here are not specific to this particular study. They are prevalent in the reporting of psychological experiments.

 

Comments

[Ken]

The had 750 total trump voters. If you assume then that 600 support Trump say then there are about 300 presented with black and 300 with white. Given 50% support we can calculate approximate 95% CI on 50% which will be about +/-6%. Given that we are only comparing 51 to 59 then we have no difference. Things would be worse for the oppose trump group. I'm certain that if the possible assumptions for proportion supporting Trump were tried they would all say the same thing, no effect.

I'm rather unimpressed that they haven't performed a statistical test. Whatever some people may say about p-values they can be rather useful in cases like this, although odds tail and 95% might work as well.

Even if this wasn't flawed, there is always the file drawer problem. There are lots of researchers doing these sorts of analyses, some must produce interesting but incorrect results.

[Fabio Machado]

There's no proper control group in this set-up either. Where is the group that wasn't prompted at all? I would also like to see what the result were for non-white Trump supporters.

[Kaiser]

Ken: Good point. I thought they might have just enough to claim p < 0.05 but that is not the case. They would have needed about 600 on each side to make it. Surprised that as researchers, they skimmed over statistical significance. (On a further check, they have 750 voters which were further split into Trump supporters versus non-supporters so the sample size on the non-Trump supporters side must be even smaller.)

Fabio: Excellent catch too. They argue that there were "almost no" non-white Trump supporters, which is not true. They would have to over-sample though to get that estimate. I also like to run tests including the null control group - but often it is tricky to design. In this case, I would either show a neutral image without a person or not show an image at all. Naysayers will then argue that I would have introduced another potential confounder in either case.