From Mikhail Simkin comes some intriguing analysis of "experts"; in this line of research, experts are compared to the "general public" and often "proved" to be shenanigans. Stock pickers don't do better than apes; economists don't do better than Big Macs; you get the idea. In a new twist, Simkin puts twelve images of modern art on his website, and asks visitors to distinguish between those by grand masters and those "ridiculous fakes" produced by him apparently on a computer.
Since conventional wisdom says elite universities provide better education, Simkin attempted to find out if there is a difference between "elites" and "the crowd" in their ability to recognize modern art. (Elites, to him, meant the Ivy League and Oxbridge.) The following pair of histograms clinched his point:
we see that there is not much difference between the elite and the crowd.
Since the shapes of the histograms are similar, one might be inclined to agree with the statement. This is until one notes the wildly different scales used because only 143 of the 56,020 quiz-takers could be identified as "elites".
The shapes are clarified if we use a relative scale (percentages) rather than absolute scale. Further, the difference is more easily seen when cumulative percentages are plotted. In other words, we are interested in comparing the proportion of respondents who score at least X points out of 12.
Two features are worth noting:
- A gap opens up between 4 to 7: specifically, 40% of "non-elites" scored 7 points or below while only 25% of "elites" scored 7 points or below.
- The curves criss-cross around 11 to 12: this shows that "non-elites" were more likely to have perfect scores (although this difference is small). Perhaps museum directors don't have .edu addresses.
Notice that I plotted Elite vs Non-Elite rather than Elite vs All Respondents. While it seems innocuous to use "All Respondents", and in this case, there is no noticeable difference since Elites were a tiny proportion, when the test group accounts for a significant proportion of the total, the value for "All Respondents" will be influenced by that for the test group. As a general rule, compare A to not A.
Simkin's exercise raises many statistical issues of design, which we won't discuss here.
Source: "Properly Prescribed" (via, RSS Significance)