Junk Charts

Tiger tiger

Picked up the Metro paper the other day and found them ventilating about the possibility that Tiger Woods used steroids; the news was that a Canadian doctor he (and other professional athletes) hired has been caught with HGH and drug equipment. In the section on why Tiger couldn't be doping, the following chart appeared:

 

According to this line of argument, since steroids should improve driving distances, and since driving distance determines overall performance, the fact that his average driving distance "remained almost constant throughout the years" proved that he did not dope.

Now, I have no idea if he dopes or not.  But this particular argument is full of holes.  In the modern era, steroids are used not just for enhancing brute strength but also shortening recovery times, prolonging training, etc.  Also, it holds only if overall performance is heavily affected by driving distance.

  The bar chart has multiple problems:

• The choice of starting the vertical scale at 250 is completely arbitrary, and as been shown before, cutting off the bottoms of bars is a bad idea -- the lengths of the remaining parts are no longer proportional to the stated data.
• The choice of the three years is also unexplained, especially when 2001 is not in the middle of 1997 and 2009. 
• The horizontal gridlines are totally redundant since all three numbers sit in the very last section (290-300).  

Why were those three years chosen?  The following line chart that plots all the data may give us a clue:

The choice of 2001 and 2009 means we missed the peak of his driving distance performance.  Looking at the standardized units, we see that at its peak, the driving distance was about 2.6 times the standard deviation above his career average (the zero line using the scale on the right). 

The difference between 1997 and the peak was about 20, which looked large compared to the standard deviation of 6 over this entire period. Establishing a reference point is very important to interpreting any observed difference.

This is one of the few occasions where double axes can be recommended.  The two axes in fact plot the same data, only reflecting a difference in scale.

Reference: "Three reasons to believe he's totally clean", Metro USA, Dec 16 2009.

Worthy of the Times

Andrew Gelman has put up a great post, discussing how he collaborated with the New York Times editors to transform his chart to a publishable form.  Some time ago, there was a discussion here about the publish-worthiness of my charts (actually, the lack thereof), and I explained that to take my charts to that level would require quite a bit of extra work.  Gelman speaks here from first-hand experience.

Here are the two versions side-by-side.  Readers on the Gelman blog are debating which version is "better" with Gelman himself voting for the revised NYT version:

Let's compile a list of the changes:

1. Removed 2000 data and relied on only 2004 data
2. Reduced number of groups of senators from 7 to 5 (with a special calling out of Senator Lieberman)
3. Re-ordered senators to facilitate classifying into Democrats, Independent, Republicans
4. Added annotation explaining the grouping of senators; in particular, qualified why the 49 Democrats and 32 Republicans were set aside
   
5. Removed scale labels on the top of the chart, retaining only the bottom labels
6. Vastly increased the area devoted to text labels, which now covered half of the chart
7. Added white vertical gridlines
8. Instead of coloring the cross marks, colored the background of the chart; introduced the yellow color for Lieberman (which Gelman originally colored blue)
9. Removed reference to the national average but explained clearly in the legend that the percentages are relative to the national average.
   

Of these, 2,3,4 are definite improvements. 1 is acceptable since 2000 and 2004 data are quite highly correlated (note, however, Iowa, Virginia and Ohio.) 7 looks brilliantly here although we typically don't recommend adding gridlines and such. 5 is also acceptable because of 7 -- with the gridlines, one doesn't need to have two sets of labels. 8 can be debated: I'd like to see a version with large colored squares instead of cross marks in lieu of the background; the current look is not bad. 9 can also be mulled over: would it be so bad as to insert in parentheses 73% after the words "national average" in the legend?  I suspect some feel that this may cause confusion because readers now need to understand relative versus absolute values. 6 is the most debatable because the data-ink ratio has been vastly reduced. 

There are several good features of the original chart that they left unaltered, which should also be duly noted:

• Allowed the positive side of the scale to extend to 10% even though the largest data only reached about 6%. While this creates empty space, it helps readers judge the magnitude of negative numbers relative to positive ones.
• Subdued horizontal gridlines
• Retained the title of the chart (I actually prefer that they use the title to summarize the finding as opposed to the current one which takes a neutral stance.)
• Retained the horizontal scale labels at 5% apart

One final comment on something omitted from both charts: I'd prefer that they plot all the dots for the 49 + 32 senators, on the top and bottom lines respectively, instead of just plotting the group averages.  It would be really interesting to see what the spread is around the 0% point.

Following one's nose 2

This is the second post on the immigration paradox study, first discussed on the Gelman blog.  My prior post on the graphing aspect is here; this post focuses on the statistical aspects. I am working backwards on Andrew's discussion points.

Which difference is most interesting?

5. Agree with Andrew; they should publish similar analyses on other minority groups as soon as possible.  One thing that strikes me when looking at the interaction plot is that the U.S. born non-Latino whites have a much higher incidence of mental illness.  The difference between different subgroups of Latinos paled in comparison to the difference between non-Latinos and the Latinos.  This latter difference is particularly acute among the U.S. born than the immigrants. The importance of the Latino analysis hinges upon whether the "paradox" is also found among other minority groups.

(Chris P also pointed this out in his comment on the previous post.)

Disaggregation, Practical Significance, and the Meaning of Not Significant

2. Andrew is also right in expressing moderate skepticism about this sort of disaggregation exercise.  He connects this to the subtle statistical point that "the difference between significant and not significant is not significant."  A related but less obtruse issue is that as one disaggregates any data, the chance of seeing variations that stray from the average gets higher and higher.  This is because the sample size is decreasing, and so the statistical estimates are less reliable.

(To give a flavor of the scale, there were a total of 2500 Latinos in the sample, with 500 Puerto Rican Latinos. The analysis drilled down to the level of different types of mental disorders, subgroups of Latinos, and also adjusted for demographics.  The details of the demographic adjustment are not available but in any case, one should be concerned about whether there were sufficient numbers of say, male immigrant Puerto Rican Latinos age 18-25 with income < $10,000 living in a rental apartment, for such an elaborate exercise.)

Expanding on this point further, one observes that the measured gap between U.S. born and immigrant Puerto Rican Latinos was about 5%.  But this 5% is probably of considerable practical significance since the base rate of incidence is about 30% (I say probably since I am not an expert in mental illness).  The current statistical analysis judged this to be insignificant -- if the sample size were larger, this difference could conceivably be statistically significant, and also practically significant.

But, doesn't the significance test deal with the small sample size problem?  Yes, if the authors merely described the Puerto Rico result as inconclusive.  Here, as is done very commonly, insignificance is equated to "no difference": they said

No differences were found in lifetime prevalence rates between migrant and U.S.-born Puerto Rican subjects.

In reality, a difference of 5% was found in the sample that was analyzed.  The statistical procedure found that this difference could have been a result of chance -- notice "could", not "must".  If the measured difference was 0.5% on 30%, then I might be willing to accept a finding of "no difference"; when it was 5% on 30%, I would like to see a larger sample analyzed.

The Meaning of Paradox

1. Andrew was perplexed by why the phenomenon is known as a "paradox". I had the same issue until I read the paper. The authors were a bit sloppy in the abstract. In the paper itself, they explained that the conventional wisdom has it that immigrants should be more likely to have mental illness because of the stress from the immigration process, and yet the statistics showed the exact opposite. That is the paradox.

Publication Bias

I was a little shocked to see the data tables that gave all the estimates of the various effects at the various subgroup levels: shocked because the authors were allowed (or asked) to include only the p-values that were below some unspecified level (which I surmised is 10% although a 5% significance level is used to judge significance as per convention). This is publication bias within publication bias. P-values that are not significant still provide valuable information and should not be omitted. They did provide confidence intervals but for each subgroup separately, rather than for the difference -- and as they noted, such intervals by themselves are inconclusive when they overlap moderately.

Life-enabling charts

In response to my call for positive examples, reader Merle H. sent in an example of how good charts can make our lives simpler and easier.

All of us have seen the following presentation of air travel data.

Not trying to pick on Travelocity - it's the same format whether you use Expedia or any of the airline sites.  For those customers who are looking to decide what dates to travel so as to minimize their air fare, this format is very cumbersome to use.

What about this fare chart at FuncTravel.com?

As you mouse along the line chart, the average fare for each day is visible.  Clicking on a particular day will fix the departure or return dates.

So much easier, isn't it?

A few caveats, though:

• Instead of just providing the historical averages, they should consider including information on variability, such as bars that indicate the middle 50% or 75% of prices.  Also, what about a sliding control for customers to decide which period of past history the averages should use?  More recent data may be more representative.
• This particular feature appeals to the price-sensitive, date-flexible customer segment.  Not everyone will pick itineraries based on those criteria.  There is an easy fix. If some controls are available for customers to indicate other preferences, e.g. exclude all British Airways flights, include only evening flights, etc., and the chart can update itself based on such selections, then the chart becomes a lot more flexible, and useful to many more customers.
• As with many automatically generated charts, the chosen labels on the vertical axis are laughable.  That should be relatively easy to fix, you'd think.
  

A great start.  I happen to notice that Travelocity has a beta feature that shows a similar chart.  A revolution in how travel sites present data to us is long overdue. 

The hard work of entertaining

Stefan pointed us to his work for the UN GEO (United Nations Global Environment Outlook) data portal.  This set of information posters highlights a vexing issue that crops up on Junk Charts from time to time, that is, the proper balance between information and entertainment value of data displays.  While this blog concerns itself primarily with the former, it does not mean that we are blind to the flashier side of the enterprise.

Let's take Stefan's recycling spiral chart as an example.  One must admit that visually this presentation is more appealing than either a data table or a set of bar charts.  The reader can obtain the primary piece of information, which is the ranking of different countries in terms of the proportion of collected waste that is recycled. 

And if the reader is curious enough, the chart also provides the data on the per-capita amount of waste collected in each of these countries.  (Like the table and bar chart, this display also has the problem that it is one-dimensional, thus the countries can be sorted by proportion of recycling but then the waste collected data will be out of order.)

For those readers who would like to understand the data better, they would want to know some of the following:

• Is there a relationship between amount of waste collected and amount of waste recycled?
• Are there differences in culture resulting in different recycling rates?
• Is the level of development of a country predictive of its recycling rate?
• Why are some countries recycling more of its waste, and others less?

To address these types of questions, one can start with the following scatter plot.

 

With the exception of South Korea, there is a general pattern of positive correlation: the more waste collected per capita, the larger proportion of such waste recycled.  Any dots that are not in the bottom left or top right quadrant are exceptions to the rule.  These countries are labeled in red or blue, the former indicating that the amount of collection is above average while the rate of recycling is below average. 

Because there is sampling error, dots that are close to the average dot (the center of this scatter plot) are probably just average.  Roughly speaking, dots in the gray circle are close enough to the center that I would not consider them exceptional cases.  That leaves Spain and Iceland in the red corner, and South Korea in the blue corner.  If both data series are considered together, these three countries should merit attention; if only the proportion of recycling is considered, then one would pay attention to Italy, Turkey and Slovak Republic on the lower end and South Korea on the high end.

Scatter plots are very versatile.  The following one explores the issue of development level.  Surprisingly, the level of recycling seems to have little to do with development; the countries are quite widely scattered.

 

Technical note: The data on both axes are expressed in "standardized" units.  So the zeroes represent the average per-capita waste collected, and the average proportion of waste recycled (only of those countries depicted in the original chart).  +1 indicates an amount that is one standard deviation above the average.  Think of "standardized units" as measuring how extreme is a particular country with respect to the average. 

A second lease on life

I hinted at it in the last post, and some readers also made similar suggestions.  What happens if we plot the U.S. life expectancy data in relative terms (indiced) rather than in absolute terms?

The result is highly revealing, and that is why we should always look at the data many ways.  While in the original chart, the differences in the race/gender segments were essentially obscured by the overall slowly-growing trend, in our new chart, we took out the trend, isolating the growth rates.



The reconstructed chart showed that:

• Between 1970 and roughly 1990, blacks of both genders gained in life expectancy at a rate higher than the national average, while white females lagged behind white males
• However, almost all of the gain by blacks were attained between 1970 and 1984, and in the 10-15 years following, this excess gain was wiped out so that by 1992 or so, the black male, black female and white male lines again converged.
• Starting in 1995, black males again achieved significant improvement in life expectancy.  This time, black females did not follow their male counterparts.  Meanwhile, white females continue to lag behind.

Not being a health care specialist, I can't say what happened to the cohorts of the 1970s, the 1980s and 1995.  One thing is for sure: these insights are hard to glean from the original.

 

Reference: "CDC says life expectancy in the US is up, deaths not", Miami Herald, Aug 19 2009.  CDC Life expectancy data.

Disseminating junk

Tuesday was an up day in the stock markets.  What did the headline writers think?

Wall Street resumes rally following housing report  (Associated Press; link)

What did this great housing report say?  Here was a version from Marketwatch, with my comments.

Housing starts surge 22% on apartment building

It was the largest percentage gain in 19 years and was the first increase in eight months in the sector that was at ground zero in the global economic recession.

At the minimum, they claimed to see a local maximum, perhaps even a global maximum

The housing data in winter months are especially volatile because of the weather.

Was someone hedging?  It was unclear at this point how important this piece of information was.  Some professionals expressed skepticism that the news was really positive:

"We're inclined to write this off as a weather-related fluke for now, " wrote economists for Wrightson ICAP.

Notice they hedged too, using the words "inclined to".  Perhaps they had a reason to doubt the news.  The article now cited less rosy numbers:

But despite February's gain, housing starts are down 47% from a year ago, and are down 74% from the peak in early 2006. Permits are down 44% in the past year.

And even more bad news:

The National Association of Home Builders reported Monday that its sentiment index was stuck at 9 on a scale of 1 to 100 in March.

Did that really say 9 out of 100?  Sounded really bad for a nation of optimists and grade inflators.  One had to start wondering about the headline.  Was that 22% growth a fluke?  Did it mean anything?

Now, two-thirds of the way down, the article revealed its essence:

The government cautions that its monthly housing data are volatile and subject to large sampling and other statistical errors. In most months, the government can't be sure whether starts increased or decreased. In February for instance, the standard error for starts was plus or minus 13.8%. Large revisions are common.

The key news about the sampling variability was a throw-away line ("for instance").

Here, the author was confused about standard error and margin of error.  Standard error, being a standard deviation, cannot be a negative number.  The range of 22% plus or minus 13.8% is a margin of error.  This is a gigantic range, between minus 8.2% to 35.8%.  While we conclude that the growth was statistically larger than zero, we have to wonder whether this number has historical significance (best in 19 years?)

Worse than that, in any month when the growth is less than 13.8%, this level of sampling error means the growth rate is not statistically different from zero.  Thus, the most newsworthy sentence of the entire piece was:

In most months, the government can't be sure whether starts increased or decreased.

In other words, this statistic of growth in housing starts is junk.  If we really want to examine this statistic, the survey needs to use a much larger sample.

MarketWatch is to be commended for noticing the volatility issue.  Of the dozens of articles that came up in a Google search about the housing starts data, none others mentioned this problem.

The original government press release is here (pdf), with all the fine print.

Eternal optimist

Chris P pointed us to the "Financial Comeback" calculator, surely a well-meaning joke from the folks at the Times.  Here is how one gets to make a 40% loss back in just six years!


Surely, someone has to tell them about simulation.  They have to assume a probability distribution on the annual returns, and show us some sample paths.  Using the average annualized historical return in essence wipes out all variability and no wonder it's smooth sailing upwards.  Eternal optimist.  

Here is Chris' comment:

The bad news is that the range of values it offers does not include the return on the market from last year (-31% to -36%).  I guess they are optimistic.

The interactive features of this chart, however, impressed me.  The smooth adjustment to the chart as one slides the control, including the automatic choice of appropriate axis labels, is very nice indeed.

Okay, they want to trademark the name of the calculator so perhaps this is serious.

Reference: "Calculate your financial comeback", Jan 6 2009.

The shackle of time 2

In the last post, we removed the time dimension in order to clarify certain aspects of the S&P 500 returns.  We found that with an investment horizon of five years, there was historically about a 25% chance of losing money and a 25% chance of more than doubling.

Even though we looked at cumulative returns, it was still the case that the data was serially correlated; in other words, it could be that the eventual return was not independent of the starting year of the five-year period.  To gauge this, we must return to the time dimension that was previously removed.

The chart on the right plots the five-year returns for all five-year periods starting from 1910.  What it shows is that even with longer time frames, timing or luck still plays a key role. 

For example, any such investment in the S&P 500 between late 1950s and 1980 did not double in five years no matter which year the investment was made.  Then again, if the investment was made in the 40s and 50s, no one lost money in a five-year period, similarly in the 1980s.

So the fact that we saw a 25% chance of doubling (or losing money) over history says much less about what might happen in the next 5 years than the simple number suggests.

In response to a reader's comment - the data series was described as "real total return" so these are inflation-adjusted.

A joke

For light entertainment, see this chart sent in by Chris P.  Original here.

Chris said: "his graph breaks a few rules, but it has a clear message".

The shocking, out of the box column certainly grabs attention, and it is probably true that football coaches earn too much money.  But the chart really falls down on this one issue:

What's the median salary of these football coaches?

Reference: "Academic salaries", PHD Comics.