Non-elites

From Mikhail Simkin comes some intriguing analysis of "experts"; in this line of research, experts are compared to the "general public" and often "proved" to be shenanigans. Stock pickers don't do better than apes; economists don't do better than Big Macs; you get the idea.  In a new twist, Simkin puts twelve images of modern art on his website, and asks visitors to distinguish between those by grand masters and those "ridiculous fakes" produced by him apparently on a computer.

Since conventional wisdom says elite universities provide better education, Simkin attempted to find out if there is a difference between "elites" and "the crowd" in their ability to recognize modern art. (Elites, to him, meant the Ivy League and Oxbridge.)  The following pair of histograms clinched his point:

we see that there is not much difference between the elite and the crowd.

Since the shapes of the histograms are similar, one might be inclined to agree with the statement.  This is until one notes the wildly different scales used because only 143 of the 56,020 quiz-takers could be identified as "elites".

The shapes are clarified if we use a relative scale (percentages) rather than absolute scale.  Further, the difference is more easily seen when cumulative percentages are plotted.  In other words, we are interested in comparing the proportion of respondents who score at least X points out of 12.

Two features are worth noting:

• A gap opens up between 4 to 7: specifically, 40% of "non-elites" scored 7 points or below while only 25% of "elites" scored 7 points or below.
• The curves criss-cross around 11 to 12: this shows that "non-elites" were more likely to have perfect scores (although this difference is small).  Perhaps museum directors don't have .edu addresses.

Notice that I plotted Elite vs Non-Elite rather than Elite vs All Respondents.  While it seems innocuous to use "All Respondents", and in this case, there is no noticeable difference since Elites were a tiny proportion, when the test group accounts for a significant proportion of the total, the value for "All Respondents" will be influenced by that for the test group.  As a general rule, compare A to not A.

Simkin's exercise raises many statistical issues of design, which we won't discuss here.

Source: "Properly Prescribed" (via, RSS Significance)

Comments

[jim]

No doubt, this question is better answered using traditional statistical techniques such as Analysis of Variance. What the chart does not answer is the "degree" of the significance of the difference (p-value).

[John Johnson]

I think Kolmogorov-Smirnov is better suited to this particular problem, although a chi-square test (on elite vs. non-elite, not elite vs. all) can be used as well. I think I'd prefer K-S.

[Hadley Wickham]

The big problem with this graphic is that we are terrible at comparing the horizontal distance between two curves - our eyes compare the shortest distance between them. A plot of the difference between the curves could be revealing.

[derek]

I'd be interested to see what that looks like when the y-axis is transformed to make a probability chart; will that opening up between 4 and 7 be as apparent, or more apparent?

[Zuil Serip]

I agree that the histogram has to be normalized in terms of %, but the cummulative % doesn't work for me in terms of building an immediate visual intuition of what is going on.

For comparing two simple distributions such as these, I'd use two superimposed histograms. Here is a variation on this theme - a superimposed violin plot.

http://img508.imageshack.us/img508/4882/eliteviolin2ba7.png

[Howard]

The labeling of the cumulative probability graph is wrong. It says "100% of subjects (in both groups) got AT LEAST 12 questions CORRECT."

[derek]

I don't have a problem with that label; instead I'd invert the cumulative curve to start at 100% ("100% got at least 0 correct") and slope down to one or two percent ("1.6% got at least 12 questions correct"). Then the "elite" line would be to the right of, and above, the general line, properly conveying the information that they did a little better overall.

I tried a probability scale, but it was disappointing, because the cumulative curve inevitably lost either the 100% or 0% cumulative values off to infinity. It did nicely display the differences at all other points though.

[Dan]

The superimposed histograms make it clear that the distributions have very different shapes, and that the Elite are not normally distributed. Since the distributions are so different here, I'm not sure if any conclusion can be made based on the data.

I'd say that realistically, there's a good chance that the Elites are a compound of two distributions: liberal arts majors who'd seen many of the pieces before and the rest, the rest being distributed identically to the plebeians. The liberal arts majors are added to a small sample of the rest and skew things to the right. It's impossible to know without more detailed information about the quiz-takers.