Junk Charts

Reading

Here are some things I have been reading while I'm traveling (the posting schedule will be erratic):

Does the vaccine matter?  Shannon Brownlee and Jeanne Lenzer investigates for The Atlantic.  About 100 million Americans get the flu shot each year; what benefit does it confer?  This is an excellent article.

Some provocative quotes:

Flu comes and goes with the seasons, and often it does not kill people directly, but rather contributes to death by making the body more susceptible to secondary infections like pneumonia or bronchitis. For this reason, researchers studying the impact of flu vaccination typically look at deaths from all causes during flu season, and compare the vaccinated and unvaccinated populations.

The estimate of 50 percent mortality reduction is based on “cohort studies,” which compare death rates in large groups, or cohorts, of people who choose to be vaccinated, against death rates in groups who don’t. But people who choose to be vaccinated may differ in many important respects from people who go unvaccinated—and those differences can influence the chance of death during flu season. [Ed: people who can afford the flu shot vs. those who can't; people who are more health-conscious vs. those who aren't, etc.]

“For a vaccine to reduce mortality by 50 percent and up to 90 percent in some studies means it has to prevent deaths not just from influenza, but also from falls, fires, heart disease, strokes, and car accidents. That’s not a vaccine, that’s a miracle.”

In the flu-vaccine world, Jefferson’s call for placebo-controlled studies is considered so radical that even some of his fellow skeptics oppose it. ... “It is considered unethical to do trials in populations that are recommended to have vaccine,” a stance that is shared by everybody from the CDC’s Nancy Cox to Anthony Fauci at the NIH. They feel strongly that vaccine has been shown to be effective and that a sham vaccine would put test subjects at unnecessary risk of getting a serious case of the flu.

Another pie chart, Fox News (via FlowingData, thanks to reader Katherine M for the pointer; also via Wonkette and reader omegatron).

Yet another pie chart, Business Insider.  Not on the same scale as the one above but still why?

Clean Water Act Violations, New York Times. Can we trust tap water?  As usual, a set of small bars would work better than concentric circles.

How does your state compare to California? (via Pew and Mother Jones) This is a nice illustration that often it is better to plot data derived from the raw data, as opposed to the raw data itself.  Since the designer decided to hide the information, let's figure out what were the cut-off points for the color categories.  If the size of each category is not the same, the designer needs to explain the scale.  Also, the two shades of light blue are hard to tell apart.  But all in all, a good effort here.

The real climategate

Climategate is all the rage at the moment. What interests me about this episode is not the integrity of certain scientists, or science in general, nor the culture of academia, and certainly not the evidence of climate change. For me, the real climategate is the woeful state of statistical education.  Let me explain.

Here is the infamous email: (via Nathan Silver, with my highlights)

From: Phil Jones
To: ray bradley ,mann@[snipped], mhughes@
[snipped]
Subject: Diagram for WMO Statement
Date: Tue, 16 Nov 1999 13:31:15 +0000
Cc: k.briffa@[snipped],t.osborn@[snipped]
Dear Ray, Mike and Malcolm,

Once Tim’s got a diagram here we’ll send that either later
today or first thing tomorrow. I’ve just completed Mike’s Nature
trick of adding in the real temps to each series for the last 20
years (ie from 1981 onwards) amd [sic] from1961 for Keith’s to
hide the decline. Mike’s series got the annual land and marine
values while the other two got April-Sept for NH land N of 20N.
The latter two are real for 1999, while the estimate for 1999
for NH combined is +0.44C wrt 61-90. The Global estimate for
1999 with data through Oct is +0.35C cf. 0.57 for 1998.

Thanks for the comments, Ray.

Cheers, Phil

What concerns me is Phil Jones' describing what he did as a "trick" to "hide the decline".  He apparently thought that he was doing something shameful. But when is it shameful to extend the plot of a time series so as to display the long-term trend, and not be misled for short-term fluctuations? This is providing statistical context to the data being examined. Lots of people are condemning this as a willful act to mislead the public but if they have some statistical literacy, they will understand that finding the appropriate time scale to look at the data is one of the most important tasks of analyzing time series data. It's a problem when even prominent scientists do not comprehend why they should be doing this.

I have always wondered why in climatology as well as in economics, we rarely see decomposed time-series plots (at least not in the public's eye). 

On the right, I found on-line a plot of a decomposition of beer sales that separates out seasonality, trend and other parts of a time series. The original data is shown up top. In practice, newspapers and blogs give us such plots all the time when they should show us the third plot down (the trend with the seasonal factor removed), unless the story is about seasonality.

Note to self: should include basic time-series decomposition in the intro stats syllabus; much too important a topic to leave to a second course.

 

Worthy of the Times

Andrew Gelman has put up a great post, discussing how he collaborated with the New York Times editors to transform his chart to a publishable form.  Some time ago, there was a discussion here about the publish-worthiness of my charts (actually, the lack thereof), and I explained that to take my charts to that level would require quite a bit of extra work.  Gelman speaks here from first-hand experience.

Here are the two versions side-by-side.  Readers on the Gelman blog are debating which version is "better" with Gelman himself voting for the revised NYT version:

Let's compile a list of the changes:

1. Removed 2000 data and relied on only 2004 data
2. Reduced number of groups of senators from 7 to 5 (with a special calling out of Senator Lieberman)
3. Re-ordered senators to facilitate classifying into Democrats, Independent, Republicans
4. Added annotation explaining the grouping of senators; in particular, qualified why the 49 Democrats and 32 Republicans were set aside
   
5. Removed scale labels on the top of the chart, retaining only the bottom labels
6. Vastly increased the area devoted to text labels, which now covered half of the chart
7. Added white vertical gridlines
8. Instead of coloring the cross marks, colored the background of the chart; introduced the yellow color for Lieberman (which Gelman originally colored blue)
9. Removed reference to the national average but explained clearly in the legend that the percentages are relative to the national average.
   

Of these, 2,3,4 are definite improvements. 1 is acceptable since 2000 and 2004 data are quite highly correlated (note, however, Iowa, Virginia and Ohio.) 7 looks brilliantly here although we typically don't recommend adding gridlines and such. 5 is also acceptable because of 7 -- with the gridlines, one doesn't need to have two sets of labels. 8 can be debated: I'd like to see a version with large colored squares instead of cross marks in lieu of the background; the current look is not bad. 9 can also be mulled over: would it be so bad as to insert in parentheses 73% after the words "national average" in the legend?  I suspect some feel that this may cause confusion because readers now need to understand relative versus absolute values. 6 is the most debatable because the data-ink ratio has been vastly reduced. 

There are several good features of the original chart that they left unaltered, which should also be duly noted:

• Allowed the positive side of the scale to extend to 10% even though the largest data only reached about 6%. While this creates empty space, it helps readers judge the magnitude of negative numbers relative to positive ones.
• Subdued horizontal gridlines
• Retained the title of the chart (I actually prefer that they use the title to summarize the finding as opposed to the current one which takes a neutral stance.)
• Retained the horizontal scale labels at 5% apart

One final comment on something omitted from both charts: I'd prefer that they plot all the dots for the 49 + 32 senators, on the top and bottom lines respectively, instead of just plotting the group averages.  It would be really interesting to see what the spread is around the 0% point.

Reading

Here are some of my favorite links from other places:

A spatial journey illustrating a very long scale, created by the Genetic Science Learning Center (here)

Long scales are very difficult to deal with in charts; I have never been satisfied with log scales since it addresses the designer's challenge of trying to fit everything onto one page, bu does not deal with the reader's need to compare the elements accurately

Not sure how this helps but perhaps some of you will figure it out

Tommi left a comment about this conceptual chart by xkcd, which has been making the rounds.  Fits into our Light Entertainment category.

Says there is no optimal chart type.  A type that works very well for one data set may get hopelessly cluttered for another, similar data set.

From fellow bloggers (especially Jorge), a whole series of views of the U.S. unemployment figures by state over time.  Alternatives that are much more interesting to look at than the typically line chart. Jorge even found something in Excel that looks good.

Following one's nose 2

This is the second post on the immigration paradox study, first discussed on the Gelman blog.  My prior post on the graphing aspect is here; this post focuses on the statistical aspects. I am working backwards on Andrew's discussion points.

Which difference is most interesting?

5. Agree with Andrew; they should publish similar analyses on other minority groups as soon as possible.  One thing that strikes me when looking at the interaction plot is that the U.S. born non-Latino whites have a much higher incidence of mental illness.  The difference between different subgroups of Latinos paled in comparison to the difference between non-Latinos and the Latinos.  This latter difference is particularly acute among the U.S. born than the immigrants. The importance of the Latino analysis hinges upon whether the "paradox" is also found among other minority groups.

(Chris P also pointed this out in his comment on the previous post.)

Disaggregation, Practical Significance, and the Meaning of Not Significant

2. Andrew is also right in expressing moderate skepticism about this sort of disaggregation exercise.  He connects this to the subtle statistical point that "the difference between significant and not significant is not significant."  A related but less obtruse issue is that as one disaggregates any data, the chance of seeing variations that stray from the average gets higher and higher.  This is because the sample size is decreasing, and so the statistical estimates are less reliable.

(To give a flavor of the scale, there were a total of 2500 Latinos in the sample, with 500 Puerto Rican Latinos. The analysis drilled down to the level of different types of mental disorders, subgroups of Latinos, and also adjusted for demographics.  The details of the demographic adjustment are not available but in any case, one should be concerned about whether there were sufficient numbers of say, male immigrant Puerto Rican Latinos age 18-25 with income < $10,000 living in a rental apartment, for such an elaborate exercise.)

Expanding on this point further, one observes that the measured gap between U.S. born and immigrant Puerto Rican Latinos was about 5%.  But this 5% is probably of considerable practical significance since the base rate of incidence is about 30% (I say probably since I am not an expert in mental illness).  The current statistical analysis judged this to be insignificant -- if the sample size were larger, this difference could conceivably be statistically significant, and also practically significant.

But, doesn't the significance test deal with the small sample size problem?  Yes, if the authors merely described the Puerto Rico result as inconclusive.  Here, as is done very commonly, insignificance is equated to "no difference": they said

No differences were found in lifetime prevalence rates between migrant and U.S.-born Puerto Rican subjects.

In reality, a difference of 5% was found in the sample that was analyzed.  The statistical procedure found that this difference could have been a result of chance -- notice "could", not "must".  If the measured difference was 0.5% on 30%, then I might be willing to accept a finding of "no difference"; when it was 5% on 30%, I would like to see a larger sample analyzed.

The Meaning of Paradox

1. Andrew was perplexed by why the phenomenon is known as a "paradox". I had the same issue until I read the paper. The authors were a bit sloppy in the abstract. In the paper itself, they explained that the conventional wisdom has it that immigrants should be more likely to have mental illness because of the stress from the immigration process, and yet the statistics showed the exact opposite. That is the paradox.

Publication Bias

I was a little shocked to see the data tables that gave all the estimates of the various effects at the various subgroup levels: shocked because the authors were allowed (or asked) to include only the p-values that were below some unspecified level (which I surmised is 10% although a 5% significance level is used to judge significance as per convention). This is publication bias within publication bias. P-values that are not significant still provide valuable information and should not be omitted. They did provide confidence intervals but for each subgroup separately, rather than for the difference -- and as they noted, such intervals by themselves are inconclusive when they overlap moderately.

Getting the Nobel wrong

Why do we pay so much attention to seemingly inconsequential details of a chart's scale, colors, labels, etc.?

Here's why. (Reader Omegatron pointed us to the FAIL blog that captured this beauty.)

Notice the messed-up horizontal scale, in particular, the failure to start the axis at 0.  The result: the tiniest difference presented as a wide gulf.

The graph, published by the Washington Post, has since been fixed.  See here.  Nevertheless, the comments left by readers lent witness to the confusion.  I copied the first bunch that mentioned the graphical display - there were plenty of for-and-against-the-prize comments, many assuming that the poll result was as lop-sided as the chart seemed to indicate.

By starting at a base of 45% (as of this reading), your graphic grossly misrepresents the results of your poll. The "no" bar is four times as big as the "yes" bar, giving the visual impression that the vote must have gone 80-20 against Obama's Nobel. On the contrary, as of now the vote is 53-47 against. Whoever produced this graphic should re-read Edward Tufte's "The Visual Display of Quantitative Information."

Posted by: threefab | October 9, 2009 8:33 AM

Someone made comment that the graph is misleading, but it really isn't if you know how to read a graph.

This type of graph highlights the difference, not the complete number of votes and is appropriate when viewing percentages.

Posted by: gconrads | October 9, 2009 8:39 AM

why is graph not drawn to scale? The vote is 51 - 49 no and the graph looks like an overwhelming number of "voters' said no

Posted by: spitts1 | October 9, 2009 8:41 AM

The graph is purposely designed to make it appear that there are a huge number of no votes and demonstrates obvious bias.

Posted by: fingersfly | October 9, 2009 8:41 AM

I'm looking at the graphic here and cannot figure out what you guys are trying to show?
The tiny slice of blue is supposed to be 50% and the huge slice of red is supposed to be 50% and the scale at the bottom is......
WHAT?

Posted by: Tomcat3 | October 9, 2009 8:53 AM

It is my understanding that the President did not "win", but was "awarded" the prize...much like a bonus given to you from your employer...deserving or not. In addition, I could not help but notice that after casting my vote, the graphic scale indicated 49% yes and 51% no; however, the graph lines seemed much more disproportionate for only a 2% spread with emphasis on “no”…

Posted by: jonbwnfd | October 9, 2009 9:40 AM

Why does the graph make the vote look so lopsided?

Posted by: subwayguy | October 9, 2009 9:48 AM

The graph is not lopsided .. the numbers are. Read the posts and you will see the posts are against the idea not for.

Posted by: T-Tom | October 9, 2009 9:53 AM 

Who designed the bar graph? The difference between the "yes" votes and the "no" votes is two percentage points, yet the "no" bar is three times as long as the "yes" bar.

In fact, "yes" is 49% and "no" is 51%: the graph is supposed to illustrate reality, not. . . well, just what DOES it illustrate?

Posted by: cmcintyr | October 9, 2009 10:02 AM

Inconsequential detail?  Big fail?  You decide.

A tale of four charts

Speaking of rules for making charts, I think the most important is "if at first you don't succeed, try, try and try again."  It's absolutely essential to produce multiple looks before settling on the one that helps tell the story.

While researching U.S. consumer credit this week, I came across these four views of presumably the same data set.

The chart on the top left (via Business Insider) shows a downward sloping line, with a steep decline on the right edge of the chart.  The authors clearly wanted to show us that consumer credit in the U.S. was collapsing.  The bottom was falling out.  To aid in this effort, they chose to:

• start the plot at the peak of the time series (2000);
• set the minimum value of the vertical scale to coincide with the lowest value attained thus far (2009) -- we have reached rock bottom, ladies and gentlemen; and
• pay no particular attention to the 0% line, which is placed in the lower half of the chart, rather than the middle, thus obscuring the fact that credit grew at faster rates during the height of the boom than it has been declining during the recession.

(Thanks to Excel's default setting, we are treated to a centipede effect on the horizontal axis.  Also thick lines and line shadows.)

Readers who previously complained about my willingness to draw a line through things that shouldn't be connected will be none too amused by the use of a line chart to plot year-on-year changes.  Thus, the zig-zagging of the line represented the change of change, which has been dubbed the "second derivative" during a period when optimists used technical wizardry to show us "green shoots".  To read this chart properly, one should focus on the actual annual decline (the dots) rather than the line.

If one takes a longer-term view, as in the chart on the top right (via Rolfe Winkler), the recent drop in consumer credit can be put in perspective.  Since the 1960s, consumer credit has been positively exploding with few years of decline.  Note, again, that the falling line between 2000 and 2008 represented a slowing rate of growth, not a decline.  The question to ask is: after so many years of almost continuous growth, is the current correction such a big cause for alarm?

This situation of exploding growth followed by a slight correction is better visualized in the third chart (lower left, via chartingtheeconomy.com)  The author also answered a question asked by Rolfe, which is to compare the total consumer credit to the population, as he plotted the per-capita consumer credit.  An eyeball estimate tells us that consumer credit jumped by more than 800 percent since the 1970s, and the current retrenchment is a blip if we take a long view.  The explosion in consumer credit has no doubt enhanced wellbeing in the U.S. for decades; even though credit might well have been over-extended in the recent past, it is far from something evil.

The final chart (bottom right, via The Big Picture) should carry a health warning.  It really should not be shown by itself, or without comment.  Both charts on the right, in fact, came from the same presentation, by David Rosenberg.  This chart, not surprisingly, is the most dramatic -- the chart designer is going for the jugular.  The decline on the right side is much more exaggerated than in any of the other three.  The trick here is:

• to plot the dollar value of credit change, rather than percentage value, and thus ensuring that the further back the time, the more insignificant the change;
• to connect dots which bring out the steep decline when in fact, the steep drop reflected the second derivative; and
  
• to put the 0% line below the middle of the chart, which causes the bottom "half" of the chart to be smaller than the top "half", which plays with our perception so that we may not realize that there were multiple years of growth above the absolute level of decline recently experienced.
  

If this chart were to be believed, our focus should not be on the cliff-diving at the far right -- instead, the chart has the hallmark of a system getting completely out of control, and oscillating to oblivion.  One hopes that is not the message intended by its creator.

For anyone creating charts, it would be a great idea to have attempted all of these versions, and more.

Lining them up

Consider this simple and effective display accompanying the NYT article titled "Recession Drives Women Back to the Work Force".  This is somewhat surprising because jobs are harder to come by during a recession. 

The journalist also told us that the interpretation of this data is controversial among economists.  The trend seemed to have been a decline in participation till 2004 and then a rise since then for 25-to-34-year-old women.  If there were an up-trend, it appeared before the start of this recession.  And participation rate dropped after the 2001 recession.  So much is still under study.

The use of small multiples is a good idea, and the color coordination between the two charts is well thought out.  A minor miss is the vertical scale: for any small multiples chart, one should always use the same scale.  In this case, using the same scale will allow readers to compare the levels of participation between men and women.

The following chart plots a related data set, focusing on the difference between men and women, among those married.  As before, the participation rates are of very different levels.  The male rate has steadily declined in recent decades while the female rate rose from the 30s in the 60s to about 70% in the 90s. 

Reference: "Recession Drives Women Back to the Work Force", New York Times, September 19, 2009; Bureau of Labor Statistics.

A world full of bubbles

As promised, we stick to bubbles.  Like the street artist blowing soap bubbles at passers-by, this map -- published in the Guardian (UK) -- is a gift of bubbles.

And our reader Frederic M. is not amused.  "A tremendous failure", he said.

In terms of conveying the data, a simple bar chart would do a better job in exposing the biggest polluters, as well as the relative magnitude between the biggies and the small fish.

The chart reveals more problems if one clicks on, say, Europe, and sees the following:

For starters, compare the bubbles labeled 858, 468, 586, 418 with those labeled 23, 20, 18.  And look at the little ones in the periphery labeled 133, 174, 128.  Baffling, isn't it?

What they did was to print the ranks for every country, except the top four in Europe for which the ranks are placed next to the country name (in small font), and the actual amounts are placed in the middle of the bubbles.  The ranks, of course, are pretty useless, and they obliterate the scale of the differences between countries.

Besides, the bigger the polluter, the smaller the rank but the larger the bubble.  This built-in disconnect can also be disorienting.

Every bubble chart typically contains lots of data labels, and the reason is that the bubble form lacks self-sufficiency.  Without the data labels, the reader has trouble comparing the areas.

Reference: "The Carbon Atlas", Guardian, Dec 9 2008.

Pie Cubed

Omegatron also did away with a set of cascading pie charts on Wikipedia, a particularly ineffective use of this chart type.  Whenever there are more than two or three categories, the necessary use of many colors can really make one's head spin.

Here, the cascade is being used like a log scale, to artificially elevate the small pieces, which unfortunately are also the least significant pieces of the energy pie.  There is no reason for nuclear, bio-mass, hydro and "others" to add to 100% except that the author decides to group them together.  The 41% nuclear or the 41% solar heating in the second and third charts, respectively, have no meaning in the larger context.

In deference to the original author, Omegatron's new version preserves the arbitrary three-level cascade.  He converts to stacked bar charts, which brings out the differences better.


He also sensibly exposes the original proportions rather than the arbitrary relative proportions.  For example, nuclear energy accounts for 6% of the total, not 41% of the arbitrary "others" bucket which in turn contains 14% of the total.

I'd prefer an even cleaner presentation with unstacked bar charts.  This can be done in either one chart with all eleven categories, or in two charts, as shown below.  The two-chart version assumes that the reader have two key questions: alternative energy sources as a proportion of the total, and the mix of different sources within the alternative category.

With the ordinary bar chart, many fewer colors are needed, and there is no need to print out each data point, nor a need to use guides to point to labels and data.  The trouble of the latter is its tendency to draw attention to the least important aspects of the data.

With this further example, I continue to find the Wikipedia rule to discourage text annotations on graphs bewildering.   Such a rule apparently does not apply to data labels, as can be seen here.  Of course, a graph without any labeling of categories is robbed of meaning but if labels can be saved, so should annotations!