Answering an open call

Dan Goldstein, who writes the Decision Science News blog, relays an internal debate occurring at Yahoo! about the relative merits of some simple charts. From what I can tell, they used three types of methods (known as "Search", "Baseline", and "Combined") on four sample data sets with different subjects ("flu", "movies", "music", "games") and compared the performance of the methods. I imagine the underlying practical question to be: does having search data improve the performance of some kind of predictive model that can be applied to the different data sets? There exists an existing baseline model that does not use search data.

(I noticed that Dan has since put up the final version of the chart they decided to use for publication. I will ignore that for the moment, and put up my response. Their final version is similar to my revised version.)

I'd like to use this data to reiterate a couple of principles that I have championed here over the years.

First, we must start a bar chart at zero. There were some back and forth on Dan's blog about whether this should be an iron-clad rule, and some comments about it being not a big deal. It is a big deal; just take a look:

The left chart has the full scale while the right chart chopped off the parts below 0.4. The result of the chopping off is that the relative lengths of the bars become distorted. For the music data, the search method appears (on the right) to be half as effective as the baseline, which is far from reality as shown on the left chart.

I used R to generate these charts, and was pleasantly surprised that the barplot function automatically assumes that bars start at zero. If you try to start the vertical axis above zero, the bars would literally walk off the chart, making it extremely ugly! (I had to pull some tricks to create the version shown above.)

***

Andrew Gelman suggested using a line chart. He also recently wrote that he has become a fan of line charts. Long-time readers know I am a fan of line charts, too... and I have tifosi who come here to complain about my over-use of line charts, especially when we have categorical data (as here!).

In particular, I have written about grouped bar charts before, and most of the time, they can be made into line charts and made clearer and better. (See here, or here.)

Some of the readers of Dan's blog complained that the dot plot makes it difficult to compare the performance of say the search method across different subjects (data sets). They think the bar charts do this better.

If comparing across subjects is the key activity for the reader of this chart, then a line chart is even better. Imagine you are reading the bar chart and comparing across subjects. Follow your eyes. You are essentially tracing lines across the top of the bars. The line chart makes this explicit. That, to me, is the key argument for using line charts in place of grouped bar charts.

I have made this argument before. Here is an illustration of the argument. The broken red lines are the same as the lines in the line chart.

***

On the line chart shown above, it's easy to see that the Combined method has the best average performance, and is never worse than the other two methods for any of the subjects. It also shows that the music subject differentiates the three methods most, primarily because the search data was not adding much to the effort. There is also no need to add colors, which can quickly make the bar charts unwieldy and disorienting.

In the final chart, shown on the right below, I flipped the two axes, changed the plot characters, used colors, shifted emphasis slightly to dots rather than lines, and started the chart at 0.5 (!)

 

Line charts are more flexible in that they can make sense even when the axis does not start at zero. In particular, when the point of the chart is to make comparisons, that is, to look at the gaps between dots or lines, rather than the absolute values, then it is fine to start the axis at some place other than zero.

Take again the example of the performance on music by the three methods (red line). The drop in performance between combined and baseline and that between baseline and music are indeed roughly equal. The vertical distances to the bottom of the chart are still distorted as in the bar chart but in a line chart, readers are less likely to get distracted by those distances because the bars are not there.

Comments

[Alex]

A great post made even better by the use of the term tifosi! It must be Calcio season again!

[Andrew Gelman]

Yes, line charts rule. I particularly like your illustration showing how a dot chart actually gets implicitly read as a line chart.

[Kaiser]

Alex: Thanks for noticing. Are there any Calcio watching groups in NYC?

[Basil]

I find this argument a little ironic as I teach this to 7th grade students. The scale of your chart is also a very important factor that many don't emphasize. Differences can be blown up or seemed to be oblivous.

[Stubborn Mule]

You make an excellent case here for the use of line charts with categorical data. With both dot plots and line charts, the zero baseline is less important (I believe) as the eye tends to judge position as opposed to the length judgement made with bar charts. Your chart labelled "Another Chart" seems to support that view, but is there any reason you based the other line chart at zero? Just consistency with the bar charts?

[Rick Wicklin]

I think you can improve interpretation by rearranging the order of the horizontal axis. For example, if you order by mean reponse, the order is Music, Games, Movies, Flu. This enables the reader to deduce additional information about the categories being studied and allows a comparison of similar categories because similar categories will be near one another.