Junk Charts

A shocking failure to communicate

So said a reader, Stephen B., of the following graphic (note: pdf) in the London Times concerning Andy Murray's recent tennis triumphs.

How can we disagree?  Shocking?  Yes.  Failure?  Definitely.  Failing to communicate?  No doubt.

Let's first start with the five tennis balls at the bottom.  It fails the self-sufficiency test.  It makes no difference whether the balls (bubbles) are the same size, or different sizes.  Readers will look at the data and ignore the bubbles.

Amazingly, the caption said that "Murray has one of the best returns of serve in the game."  And yet, the graphic showed the five players who were better than Murray, and nobody worse!  For those unfamiliar with tennis statistics, it does not provide any helpful statistics like averages, medians, etc. to help us understand the data.

But that is only the beginning.

Take a look at these two donuts.

(The color scheme from light to dark: first, second, third, fourth round of tournament)

So we're told: the 75% of first-serve points won in the fourth round was 25.6% of the sum of the percentages of first-serve points won from first to fourth rounds (75%+70%+71%+76%).  What does this mean?  Why should we care?

The challenge with these two statistics is that they are correlated and have to be interpreted together.  If a first-serve is won, then there would be no second serve, etc.  Here's one attempt at it, using statistics from the Soderling-Federer match.  It's clear that Federer was better on both serves.

Reference: "Murray's march to the last eight", London Times.

 

Round up

Here are some interesting reading from other places:

Tag clouds have caught on since we approved them a while ago.  One interesting use was at the Life Vicarious blog.  They use it to compare the inclinations of three New York-based restaurant reviewers.  What they should have done is to remove irrelevant words like  "one", "also", "many", "make"/"made", etc.  In statistics, this is called removing "noise" which helps bring out the "signal".

Andrew Gelman discussed the NYT article that reported the finding of unexpected male bias in the children of Asian American families.  He can be counted on to make useful comments on any accompanying graphics.  He rightly pointed out that this is one example of not starting at zero: the relevant baseline is 100 since the metric is essentially the over-age of males relative to females.  I also agree that a line chart with a longer time series plotting percentages rather than over-age would work better.

The racetrack chart made an appearance at Flowing Data.  This one is even more busy and just as impossible to decipher.

Recovery inside a recovery

While reconstructing the Dow price chart (here), I noticed that there was some dubious statistics going on behind the scenes.  The chart made the point that the 1929 bear market took over 20 years to recover to its peak value.  The mystery wrapped in the enigma is the existence of the time series for a 1937 bear market and a 1939 bear market.  This could not happen unless there were bears within bears and recoveries within recoveries.

The uncomplicated time-series view brings this situation out more clearly:



This is a sobering picture in the face of all the talk about "green shoots" and "bear market rallies".

From a statistical perspective, the 1937 and 1939 bear markets cannot be interpreted without noting that they happened inside of a larger bear market.

Don't mess with the scale

My friend Patrick pointed out the single biggest issue with the chart below -- that the designer chose a scale that precisely undermines the message of the chart.  Undermine may be too mild a word to use here; annihilate may be more apt.

The lines in this chart are anchored at the zero point on the time line (horizontal axis) used to indicate the bottoms of various bear markets in the Dow from 1929 to 2007.  From that anchor, time runs to the left showing the amount of time for the Dow to go from peak to bottom (the decline); time runs to the right showing the amount of time for the Dow to climb back to the prior peak (the recovery).  As the caption said, the point of the chart is "if the decline was fast, the recovery took a considerable time".

Funny thing then that the distances from the zero point are roughly comparable on the left as on the right.

This illusion resulted from some very convoluted and perplexing messing around with the horizontal scale.  First, the left-of-center scale is in months while the right-of-center is in years.  Second, the left-of-center scale has normal spacing while the right-of-center seemingly was suffering from spasms.  Take a closer look:

The first five years (0-5) took up about half the scale while the next five (5-10) took maybe one-eighth.  The first year (0-1) took about as much space as the next two years (1-3).

I am not quite sure what is the logic behind this but since the message of the chart has everything to do with the time duration, it is most unfortunate to introduce such distortions.

There is yet another "innovation" in this chart.  Notice that on the right side, the axis labels are irregular (more spasms)... 0,1,3,4,5,10,15,20, 25...  This is as if the designer is posing one of those IQ questions requiring readers to figure out the next number in the sequence.  The specific time intervals selected may have meaning: note that all the lines are straightened out in between these tick marks.  Given that each line represents a different historical sequence, it is difficult to comprehend the regularity of these intervals across history.  Perhaps this will prove to be the key to unlocking the secret of this chart.  Please comment below if you are able to unravel this mystery.

Besides, the same type of "innovation" was not applied to the left side of the chart.  Here, the designer opted to throw out all the data between the peak and the bottom and straightened out all the intermediate fluctuations.

Below are two different versions of this chart, basically restoring the time scale to the normal, equally spaced, symmetric appearance.  The top one used monthly Dow returns where the volatility obstructed our understanding of the trends, requiring the use of color to differentiate the lines.  In the next version, I used R to generate the loess estimates (a type of smoothing) and the trends became clearer.  (There was a prior discussion of loess on Junk Charts here.)

 

Now, these pictures are very different from the original graph!

I'd be very cautious about reading into these charts anyway.  This question is not one suitable for statistical analysis.  The sample size of six is far too small.  Each recession is different in terms of causes, remedies and context.  The fact that we call them recessions do not make them comparable.  Further, it is also impossible to know at this stage if the 2007 decline has reached bottom.  The chart designer essentially assumed this to be the case but who knows?

PS. Nick Rapp, one of the designers of the chart, responds in the comments.  He has started a blog to feature the work of his graphics team at AP.  His colleague has created an interactive version.  More than anything, this post highlights an aspect of the chart that Nick and his team clearly spent a lot of time doodling over.  The concept of the chart itself is wonderful actually, if I didn't say so already; it is essentially the same chart as the oft-printed chart where the anchor point is the start of each recession, only here the anchor is the bottom of each recession.

Disseminating junk

Tuesday was an up day in the stock markets.  What did the headline writers think?

Wall Street resumes rally following housing report  (Associated Press; link)

What did this great housing report say?  Here was a version from Marketwatch, with my comments.

Housing starts surge 22% on apartment building

It was the largest percentage gain in 19 years and was the first increase in eight months in the sector that was at ground zero in the global economic recession.

At the minimum, they claimed to see a local maximum, perhaps even a global maximum

The housing data in winter months are especially volatile because of the weather.

Was someone hedging?  It was unclear at this point how important this piece of information was.  Some professionals expressed skepticism that the news was really positive:

"We're inclined to write this off as a weather-related fluke for now, " wrote economists for Wrightson ICAP.

Notice they hedged too, using the words "inclined to".  Perhaps they had a reason to doubt the news.  The article now cited less rosy numbers:

But despite February's gain, housing starts are down 47% from a year ago, and are down 74% from the peak in early 2006. Permits are down 44% in the past year.

And even more bad news:

The National Association of Home Builders reported Monday that its sentiment index was stuck at 9 on a scale of 1 to 100 in March.

Did that really say 9 out of 100?  Sounded really bad for a nation of optimists and grade inflators.  One had to start wondering about the headline.  Was that 22% growth a fluke?  Did it mean anything?

Now, two-thirds of the way down, the article revealed its essence:

The government cautions that its monthly housing data are volatile and subject to large sampling and other statistical errors. In most months, the government can't be sure whether starts increased or decreased. In February for instance, the standard error for starts was plus or minus 13.8%. Large revisions are common.

The key news about the sampling variability was a throw-away line ("for instance").

Here, the author was confused about standard error and margin of error.  Standard error, being a standard deviation, cannot be a negative number.  The range of 22% plus or minus 13.8% is a margin of error.  This is a gigantic range, between minus 8.2% to 35.8%.  While we conclude that the growth was statistically larger than zero, we have to wonder whether this number has historical significance (best in 19 years?)

Worse than that, in any month when the growth is less than 13.8%, this level of sampling error means the growth rate is not statistically different from zero.  Thus, the most newsworthy sentence of the entire piece was:

In most months, the government can't be sure whether starts increased or decreased.

In other words, this statistic of growth in housing starts is junk.  If we really want to examine this statistic, the survey needs to use a much larger sample.

MarketWatch is to be commended for noticing the volatility issue.  Of the dozens of articles that came up in a Google search about the housing starts data, none others mentioned this problem.

The original government press release is here (pdf), with all the fine print.

Eternal optimist

Chris P pointed us to the "Financial Comeback" calculator, surely a well-meaning joke from the folks at the Times.  Here is how one gets to make a 40% loss back in just six years!


Surely, someone has to tell them about simulation.  They have to assume a probability distribution on the annual returns, and show us some sample paths.  Using the average annualized historical return in essence wipes out all variability and no wonder it's smooth sailing upwards.  Eternal optimist.  

Here is Chris' comment:

The bad news is that the range of values it offers does not include the return on the market from last year (-31% to -36%).  I guess they are optimistic.

The interactive features of this chart, however, impressed me.  The smooth adjustment to the chart as one slides the control, including the automatic choice of appropriate axis labels, is very nice indeed.

Okay, they want to trademark the name of the calculator so perhaps this is serious.

Reference: "Calculate your financial comeback", Jan 6 2009.

A Harvard mess 2

Jon's comment on the previous post pretty much anticipated this post.   The prior post concentrated on graphical matters.  However, the biggest issue with that chart is the choice of metrics.  If the idea is to explore the potential adverse effect of a sharp decline in endowment investment performance, then it is not clear why one should be comparing the proportion of endowment funds and the proportion of operating revenues paid for by endowment funds.  A missing element from these two series is the relative size of the budgets of these different departments.

The next chart shows the proportion of each school's operating costs accounted for by endowment funds together with the total size of its operating costs.


We can turn the ratio around and directly compute how much of the total amount of endowment funds distributed for operating costs is accounted for by each school.  This is really the simplest metric that gets to the question.

 

There are really two possible worries: for the School of Arts and Sciences, who pays for just about $1 billion costs with endowment funds, any significant reduction in distribution will leave a gaping hole; for a department like Radcliffe that pays for over 80% of its operating budget out of endowment funds, obviously a reduction in distribution can cause problems but we are talking about a base of $18 million rather than $1 billion.

Reference: Harvard University Financial Report, 2008; Harvard Fact Book 2007

PS. The first source did not contain any data on operating budgets so the first set of graphs (now replaced) did not show what I intended to show.  The new ones used the right data and had the right order of magnitude in terms of budgets ranging from millions to 1 billion.  The 2008 data are not available as of yet.

R power, math stats power

Amusingly, the New York Times finally got wind of the R software.   See the article here.

We in the statistics community owe these folks a lot of gratitude for developing such a flexible, powerful software.  It is unfortunate that they didn't mention graphing as one of the great strengths of the software.

Equally amusingly, the Wall Street Journal told us what we already know, that we have the best jobs in the world.  Their discovery here.


 

Opinion graph

Sharon P. pointed us to this graph, which can make one speechless.

Reference: "Bulls, Bears, Donkeys, Elephants", New York Times, Oct 14 2008.

A joke

For light entertainment, see this chart sent in by Chris P.  Original here.

Chris said: "his graph breaks a few rules, but it has a clear message".

The shocking, out of the box column certainly grabs attention, and it is probably true that football coaches earn too much money.  But the chart really falls down on this one issue:

What's the median salary of these football coaches?

Reference: "Academic salaries", PHD Comics.