Junk Charts

Serving donuts

David Leonhardt's article on the graduation rates of public universities caught my attention for both graphical and statistical reasons.

David gave a partial review of a new book "Crossing The Finish Line", focusing on their conclusion that public universities must improve their 4-year graduation rates in order for education in the U.S. to achieve progress.  This conclusion was arrived at through statistical analysis of detailed longitudinal data (collected since 1999).

This chart is used to illustrate this conclusion.  We will come to the graphical offering later but first I want to fill in some details omitted from David's article by walking through how a statistician would look at this matter, what it means by "controlling for" something.

The question at hand is whether public universities, especially less selective ones, have "caused" students to lag behind in graduation rate.  A first-order analysis would immediately find that the overall graduation rate at less selective public universities to be lower, about 20% lower, than at more selective public universities.  

A doubter appears, and suggests that less selective schools are saddled with lower-ability students, and that would be the "cause" of lower graduation rates, as opposed to anything the schools actually do to students.  Not so fast, the statistician now disaggregates the data and look at the graduation rates within subgroups of students with comparable ability (in this instance, the researchers used GPA and SAT scores as indicators of ability).  This is known as "controlling for the ability level".  The data now shows that at every ability level, the same gap of about 20% exists: about 20% fewer students graduate at the less selective colleges than at the more selective ones.  This eliminates the mix of abilities as a viable "cause" of lower graduation rates.

The researchers now conclude that conditions of the schools (I think they blame the administrators) "caused" the lower graduation rates.  Note, however, that this does not preclude factors other than mix of abilities and school conditions from being the real "cause" of lower graduation rates.  But as far as this analysis goes, it sounds pretty convincing to me.

That is, if I ignore the fact that graduation rates are really artifacts of how much the administrators want to graduate students.  As the book review article pointed out, at the less selective colleges, they may want to reduce graduation rates in order to save money since juniors and seniors are more expensive to support due to smaller class sizes and so on.  On the other hand, the most selective colleges have an incentive to maintain a near-perfect graduation rates since the US News and other organizations typically use this metric in their rankings -- if you were the administrator, what would you do?  (You didn't hear it from here.)

Back to the chart, or shall we say the delivery of 16 donuts?

First, it fails the self-sufficiency principle.  If we remove the graphical bits, nothing much is lost from the chart.  Both are equally impenetrable.

A far better alternative is shown below, using a type of profile chart.

Finally, I must mention that in this particular case, there is no need to draw all four lines.  Since the finding of a 20% gap essentially holds for all subgroups, no information is lost by collapsing the subgroups and reporting the average line instead (with a note explaining that the same effect affected every subgroup).  

By the way, that is the difference between the statistical grapher - who is always looking to simplify the data - and the information grapher - who is aiming for fidelity. 

Reference: "Colleges are lagging in graduation rates", New York Times, Sept 9, 2009; "Book review: (Not) Crossing the Finish Line", Inside Higher Education, Sept 9 2009.

A shocking failure to communicate

So said a reader, Stephen B., of the following graphic (note: pdf) in the London Times concerning Andy Murray's recent tennis triumphs.

How can we disagree?  Shocking?  Yes.  Failure?  Definitely.  Failing to communicate?  No doubt.

Let's first start with the five tennis balls at the bottom.  It fails the self-sufficiency test.  It makes no difference whether the balls (bubbles) are the same size, or different sizes.  Readers will look at the data and ignore the bubbles.

Amazingly, the caption said that "Murray has one of the best returns of serve in the game."  And yet, the graphic showed the five players who were better than Murray, and nobody worse!  For those unfamiliar with tennis statistics, it does not provide any helpful statistics like averages, medians, etc. to help us understand the data.

But that is only the beginning.

Take a look at these two donuts.

(The color scheme from light to dark: first, second, third, fourth round of tournament)

So we're told: the 75% of first-serve points won in the fourth round was 25.6% of the sum of the percentages of first-serve points won from first to fourth rounds (75%+70%+71%+76%).  What does this mean?  Why should we care?

The challenge with these two statistics is that they are correlated and have to be interpreted together.  If a first-serve is won, then there would be no second serve, etc.  Here's one attempt at it, using statistics from the Soderling-Federer match.  It's clear that Federer was better on both serves.

Reference: "Murray's march to the last eight", London Times.

 

Mini donuts

As a reader noted, this chart is essentially unreadable.  It contains data for the composition of diets in four countries during two time periods.

What might we want to learn from this data?

Are there major differences in diet between countries?

Within each country, are there changes in diet composition over the thirty years?

If there were changes in diet inside a country over time, did those reflect a worldwide trend or a trend specific to that country?

Unfortunately, the use of donut charts, albeit in small multiples, does not help the cause.  The added dimension of the size of the pies, used to display the total calories per person per day, serves little purpose.  Seriously, who out there is comparing the pie sizes rather than reading off the numbers in the donut holes if she wants to compare total calories?

This data set has much potential, and allows me to show, yet again, why I love "bumps charts".

Here is one take on it.  (Note that the closest data I found was for six different countries - China, Egypt, Mexico, South Africa, Philippines, India - and for different periods.)

The set of small multiples recognizes that the comparison between 1970 and 2000 is paramount to the exercise.  There is a wealth of trends that can be pulled out of these charts.  For example, the Chinese and Egyptians take in much more vegetables than the people of the other countries; in particular, the Chinese increased the consumption of vegetables drastically in those 30 years. (top row, second from left)

Or perhaps, for sugars and sweetners, consumption has increased everywhere except for South Africa.  In addition, the Chinese eat a lot less sugars than the other peoples. (top row, right)

Egg consumption also shows an interesting pattern.  In 1970, the countries had similar levels but by 2000, Mexicans and the Chinese have outpaced the other countries. (bottom row, right)

These charts are very versatile.  The example shown above is not yet ready for publication.  The designer must now decide what are the key messages, and then can use color judiciously to draw the reader's attention to the relevant parts.

Also, some may not like the default scaling of the vertical axes. That can be easily fixed.

Finally, here is another take which focuses on countries rather than food groups.  We note that too many categories of foods make it hard to separate them.

References: "Who's Eating What?", Wired, Oct 2008; "The Double burden of malnutrition", FAO, 2006.

Read fast, pay the price

At first, this looks like a decent chart despite the donut construct, which I cannot stand (but the Economist loves).

The accompanying text proclaimed: "Rock stars are famous for excess, and some pay the price".  The rest of the paragraph points out drug- and alcohol-related deaths, plus deaths due to "unhealthy lifestyles", which apparently include cancer and cardiovascular disease.

There is a gaping hole between what's on the chart and what's in the text.  They just talk past each other.

• The chart invites us to compare the European experience to the American experience. Each donut presents the proportion of total deaths by causes of death. The top donut presents American rock-star deaths, the bottom European ones. But this comparison has zilch to do with the key point, which is how rock stars are different from the rest of us.  The chart tells us nothing about the rest of us.  The 20% death by cancer would be entirely unremarkable if 20% of non-rock-star deaths also were attributed to cancer!

• We must also bear in mind that the base populations are rock stars who died young. This is a very specific demographic segment, and so the only valid point of reference are people who died young.  If we think along those lines, then among unmusical people, if they died young, what might have been the causes of death?  Drugs? Alcohol?  Accidents?  Suicide?  You bet.  I am not sure who is the authoritative source of such data but the CDC reported that among Americans aged 15-34 who died, the leading causes were "unintentional injury", suicides, homicides, cancer and heart disease.  Not much different from the above list...

• The deaths depicted in the two donuts totaled fewer than 100, and yet percentages are given to one decimal place.  This creates a false sense of precision not justified by the sample size.
• The deaths occurred over about 50 years.  It is very likely that the causes of premature death have shifted during this time span, making an aggregate analysis questionable.

Charting is much more than just aesthetics.  Some basic statistical common sense goes a long way.  This was observed long ago by Huff.

Source: "Rock stars: live fast, die young", Economist, Sept 4 2007.

Which is the bigger mess?

The Economist spoke of a "perfect mess".   Was that the Bundestag or the small-half-donut-inside-a-big-half-donut mess?
Presumably, it was something about the distribution of seats that so disturbed our esteemed editors but the reader is unlikely to empathize based on this chart.  One will be mistaken to think the size of the Bundestag doubled from 2002 to 2005; in fact, it only grew by 10 to 613 seats in 2005.

In the chart on the right, the gains and losses in seats by each party are made front and center.  This is a good chart if seat changes are the key message to be conveyed.  Lying hidden in this presentation is the fact that the two largest parties has continued to dominate despite losing some seats to the small parties.

Again, I turn to the Bumps chart.  Here, both the relative sizes of each party and the gains/losses are clearly depicted.
 
It is also easy to read off the rankings by number of seats in 2002 and 2005.  For instance, the Christian Democrats overtook the Social Democrats in 2005.  The Left Party enjoyed a spectacular rise while the Greens suffered, becoming the smallest party, even though it lost only four seats.  The total number of seats is also included without clutter.

At long last, the so-called mess is evident in the criss-crossing lines, indicating that the ranking of the parties was turned topsy-turvy in the recent election.

 
Reference: "A system in crisis, a country adrift", Economist, Sept 22 2005.
Thanks to Annette for help with this post.