Junk Charts

Statistical adjustment in charts

On the book blog, I often talk about the reasons why statisticians adjust data, and why it is necessary in order to paint a proper picture of what the data is saying. (See here or here.)

On this blog, I have frequently complained about how the "prior information" on maps is too strong - large regions dominate our perception regardless of the data. In the U.S., large but sparsely populated states attain disproportionate attention.

So, why not bring "statistical adjustment" to maps?

***

That's exactly what cartograms do. For example, look at the following pair of maps created by the people at Leicestershire County Council. (PDF link here)

The map on the left and the cartogram on the right plot identical data. The only difference is that each hexagon on the cartogram represents an equal number of people. The two views give very different impressions: the big dark green patch on the middle-right of the map -- representing a relatively sparse neighborhood -- is shrunk to a single dark green hexagon on the cartogram. Meanwhile, the most deprived areas (dark purple) which look relatively small on the map are expanded to quite a few hexagons.

According to the map, most of the county live in areas ranked in the half considered less deprived (green), and that is good news. But wait... there is a lot of purple in the cartogram!

The real piece of news is that the majority of people live in the half of the neighborhoods considered more deprived (purple) but this uncomfortable fact is well-hidden in the mostly green map on the left.

Given that the measures of "deprivation" are about people, not geographical neighborhoods, the cartogram is much closer to the real world experience... notwithstanding the obvious geographical distortion introduced by the statistical adjustment.

According to Alex L., who is part of the team producing these graphics:

LSOAs were created for the 2001 [UK] Census to disseminate the data and are generally considered to represent 'neighbourhoods'. They are created to have a broadly consistent population (approx 1500 people in 2001) and socio-economic traits.

***

Question: Is there any reason to show the map at all?

 

The wall of blinking lights

Reader Alex L. submitted this chart showing the evolution of quality of life in Warwickshire in the U.K.

 This wall of lights is drawing way too much power. Let's make a list of fixes:

• Stretch out the hemisphere, turning those arcs into horizontal lines
• Allow readers to read horizontally, rather than centrifugally (?)
• Align horizontally all of the labels for the quality of life indicators
• Allow readers to read indicator labels in one direction, rather than inside-out on the right hemisphere and outside-in on the left hemisphere
• Assume readers understand that the first year for which there is data is the "baseline year"
• Remove the distance between one data point and another, which makes unnecessary the white gridlines
• Use rectangles (rather than circles) as they can be packed more tightly
• Order the indicators in a meaningful way

Eventually this chart reveals itself as a heatmap:

The heatmap is much better. But the heatmap doesn't expose the trends clearly, especially the differences between indicators. The heatmap function (in R) has a built-in clustering method which automatically groups the indicators by similarity of trends. The color scheme should really be reversed since on this chart, red is good, and blue is bad; the default orientation of the column labels is also annoying. The bad indicators are clustered to the top, the good ones in the middle and the neutral ones at the bottom.

***

The next version uses the line chart, in a small-multiples setting. Now we have something to chew on.

Although not done here, we can order this set of charts using the clustering results from the heatmap.

The lesson is that the pretty colors in the heatmap really tell us much less than the plain levels in a line chart.

 

 

Showing off the world in charts

Stefan S. who works for the UN data project and is a regular contributor to this blog, points us to a new report they have issued that contain a host of charts. The report is an update on what has happened to our Earth since 1992 (The Earth Summit). Link to the PDF file here.

***

This life expectancy chart (shown on left) uses a Bumps-type chart, and is very nicely done, clean and informative. 

***

This age distribution chart shown on the right is unusual. It's a case of the data defeating the chart type. The magnitude of the 5-year changes is just not large enough as a percentage of the total to register. On a different data set, I can see this chart type being more effective.

***

Now, this criss-cross chart (bottom left) reminds me of Friedman's foolish attempt some time ago. It has various issues, like dual axes, excessive labels and inattentive titles (not indicating that the base population was only of developing countries).

  Instead, I attempted an area chart, using population size as the primary metric. Perhaps a more direct way to illustrate this point is to plot the growth rate of the slum population versus that of the total population.

***

This map is excellent, showing the spatial distribution of the countries with above-average and below-average GDP per capita. It would be even better if smaller geographic units can be used so that the distribution within each country can also be seen.

***

I'd like to salute all the people around the world who work at statistical agencies and who collect and make sense of all of this data, without which any of these charts would not have been possible.

Worst statistical graphic nominated

Phil, over at the Gelman blog, nominates this jaw-dropping graphic as the worst of the year. I have to agree:

Should we complain about the "pie chart"/4 quadrants representation with no reference to the underlying data? Or the "pie within a pie within a pie" invention, again defiantly not to scale? Or the creative liense to exaggerate the smallest numbers in the chart ($2 billion, $0.3 billion) making it disproportionate with the other pieces? Or the complete usurping of proportions (e.g. the $0.2 billion green strip on the top right quadrant compared to the $0.3 billion tiny blue arc on the top left quadrant)?

Or the random sprinkling of labels and numbers around the circle even though if one takes the time, one notices that the entire chart contains only 8 numbers, as follows:

***

Instead, we can display the data with a small multiples layout showing readers how the data is structured along two dimensions.

Or a profile chart may also work:

 

 

The hazard of casual analysis of hazards

Martin's quick analysis of the Japanese earthquake relative to other earthquakes produced a number of intriguing charts, including the following map which illustrates the "ring of fire" (Wiki link):

Imagine that one knows nothing about tectonic theory and geography. Such a map would be quite illuminating.

***

The comments section contains an exchange between two Martins about the hazard of non-experts plotting and then commenting on data they know little about.  In the past, this conversation would take place behind closed doors as the statistician works with the geologist, each learning from the other. In our modern age, it is common for both sides to take up arms in public with colorful language.

The lesson for us spectators is the importance of understanding how data is collected, how data is defined, how data is processed, etc.

***

The plots Martin made do give me a chance to talk about a few interesting statistical issues relating to plotting data.

First, this bar chart of the number of earthquakes over time seems to indicate that earthquake activity has been increasing over time:

The chart doesn't have a vertical scale so it is hard to judge whether the growth is meaningful or not. The US Geological Service doesn't seem to think this is a remarkable event, as the other Martin pointed out. One explanation is the increasing sensitivity of measuring equipment, which leads to more small-magnitude earthquakes being recorded over time.

Whether or not this hypothesis is true in the context of earthquakes, this is a very important phenomenon that occurs often. One of the mysteries in the epidemiology of autism is the recent rise in the number of cases but this is made complicated by the much higher probability of diagnosis in recent times. Similarly, as the stigma against reporting rapes, harassment and other crimes dissipate, more such criminal reports will be filed, and how does one distinguish between higher reporting and higher incidence, both of which lead to higher counts?

This same phenomenon happened during the Toyota brake scare last year. It appeared as if the problem was getting worse while the scandal brewed. But as the awareness of the potential risk increases, so too will the probability of someone reporting an issue.

One other feature of the bar chart worth noting is the faux plunge in the number of earthquakes in 2011. My recommendation is either to omit the incomplete year, or to forecast the end-of-year count (labelling it clearly as a forecast).

Finally, for data relating to rare events like earthquakes, one should try to take as long a view as possible. Twenty years is good but a hundred years would be better.

***

Next, Martin created this histogram that plots the number of earthquakes of different magnitudes over the last two decades or so.

As Martin noted, Richter scale is a log scale, meaning a 1-unit increase on the horizontal direction is really a 10 fold increase.

Martin's point is to show how strong the earthquake in Japan was compared to historical earthquakes. This is a useful and interesting question to ask, and his choice of graphics cannot be faulted.

What the other Martin was complaining about is the third corner of our Trifecta checkup: whether the right data has been used. Martin used what he was able to find. The histogram in fact looked quite nice, was rather symmetric, and suggested a tractable mathematical model. But if we proceed along that path, we would have chosen the wrong model.

The trap is a missing data problem that will not be evident to non-experts. As discussed above, small-magnitude quakes are not being tracked. Thus, the left side of the histogram severely under-represents the true frequency of such small tremors.

There is a Guthenberg-Richter law relating earthquake magnitude and frequency. It's a power law. Roughly speaking, for each 1 unit decrease in Richter magnitude, earthquakes of that magnitude occur about one-tenth as  frequently. This law fits the actual data really well when Richter magnitude is high but at lower magnitudes, geologists believe that our records are severely under-counted.

In the chart shown on the right, which comes from this paper (PDF link) about predicting earthquakes, the dots sit on the line at higher magnitudes but the observed frequencies fall markedly below the line (projection) for magnitude lower than 2. It suggests that the bars on the left side of the histogram above should have been much taller than we are seeing. For a proper power law, the bars on the left should be the tallest of all!

 

 

 

 

Charts should not be used as map lessons

Like Australia-based reader Ken B., I don't understand why many chart designers insist on using charts to deliver lessons to the public on map geography. Here is a recent example from Down Under, on earthquakes: (click on this link for the interactive version)

Was there a quake that shook the middle of the Pacific? Did a new geological formation give New Zealand a Pinocchio nose? No and no. The ugly presentation of the 2010 and 2011 Christchurch earthquakes -- as two ends of a dumbbell -- makes clear the straitjacket that maps are when it comes to delivering quantitative information.

Besides, the bubbles represent the relative magnitude of the quakes when one would hope that their sizes represent the geographical extent of the damage; at least, that would be information that has a spatial dimension.

The location of the quake is the only data with a spatial dimension surfaced on this plot. The only purpose of the map background is to tell us where Christchurch, Sichuan, etc. are on a map. In order to deliver this map lesson, the designer has to hide all of the more interesting data, like the relative magnitudes, the time-lines, the extent of the damage, the mortality rates, etc. In my mind, that is a very poor tradeoff.

Handling multi-level data in multiple charts

Stefan S., whose team created the inkblot charts featured here, has updated his page of charts with a bunch of new ones, including some bubble charts. I had fun looking at a few of his experiments, and learnt a bit from them.

This chart deals with the geographical distribution of CO2 emissions and of wealth, and their correlation. The "standard" format would be to put pairs of columns onto a world map. That format has various weaknesses, and it is great to see Stefan try to reimagine the chart.

I especially love how he broke up the map into continents and subregions and arranged these pieces in a clean manner. He also recognized that stacked columns are not great for comparisons as the two pieces being compared don't usually sit at the same level. So in this chart, most pieces are level at their base. The solution is not perfect though, as for instance in the European section, it was very hard to put everything level without introducing white space.

Stefan also realized that you can't make it easy to compare distribution within a continent and across the globe in one chart, so he created the right-side column to solve this problem. Again, it's a good effort, not entirely successful but a very good start.

***

Another chart I like is the inkblot chart that deals with all levels of the data simultaneously. These, I think, manage to both engage our brains and entertain us.

***
Stefan specifically asked for feedback on this bubble chart:

 I must say the map legend is a lovely touch.

As a whole, the scatter plot is effective at showing the inverse positive [oops, see comment below] correlation between development and per-capita emissions.

The effort is enormously ambitious in terms of stuffing as many dimensions as possible onto the same chart. I feel like the data can benefit from being shown in a set of charts, rather than one.

For example, having the continental averages plotted with all the individual countries doesn't work for me. I'd rather see individual plots for each continent, with the continental average plotted against a background of all the individual countries. This proposed view is reminiscent of the Gapminder presentation from what seems like eons ago.

Also, many metrics are found here: population, per-capita emissions, total emissions, human development index. Anyone familiar with this business knows that there are controversies around different pairs of metrics, e.g. per-capita versus total emissions, total emissions v. population, per-capita emissions v. population. Thus, a panel of scatter plots that focuses on different pairs of metrics would encourage readers to ponder these practical questions. Otherwise, we may feel like being dumped in the deep end.

I'm sure Stefan would appreciate other comments on this or other charts.

 

Loss aversion and faux accuracy

Reader Bernie M. is not a fan of this Economist chart.

The chart was prepared by Aurora Flight Sciences, an aircraft manufacturer, commissioned by a professor who supports the concept of maintaining a fleet to pump sulphuric acid into the stratosphere as a way to induce artificial cooling to counteract human-induced global warming.

The chart appears to compare many different ways of shooting the acid into the skies along two dimensions: cost and altitude.

Bernie wrote:

I find the choice of axes extremely counterintuitive. Altitude one would expect on the y-axis. And mixing up the scatter chart elements with the connected line chart doesn' really help either.

The convention regarding axes is to put the outcome variable on the vertical, and the explanatory variable on the horizontal. Thus, in this case, if the cost of a particular solution is primarily determined by the "altitude" (presumably of where the acid would be released), then the designer has followed convention. It is unfortunate that "altitude" is more intuitively put on the vertical axis, but I suspect that defying convention might cause more confusion.

On the other hand, if altitude and cost are not related to each other but two different metrics to evaluate geoengineering concepts, then Bernie's point is right on - swap the axes!

***

The use of connected lines for two of the solutions but not the rest is a symptom of what I have called "loss aversion". The horror of leaving some of the data on the cutting floor.

The only mention of altitude in the article refers to Aurora's assertion that it is sufficient to use newly designed aircraft flying at 20-25 kilometers. If that is Aurora's preferred solution, there is little reason to show all the other altitude configurations that are suboptimal.

Perhaps the designer wants to make the point that the Boeing 747 solution is inferior to the Aurora solution because Aurora could design aircraft to fly at 10-15 km at a lower cost?  If so, then the chart is very misleading in not providing a comparable cost for Boeing's solution if required to fly at 20-25 km.

When comparing different entities, it is always a bad idea to treat the entities differently. Comparison is only possible on equal footing.

In fact, I think the chart would be a lot clearer if they dropped the altitude dimension on the floor. For each solution, plot the yearly cost at the optimal altitude selected by the respective engineers. Use a bar chart. With a single dimension, it is much easier to accommodate the very long data labels.

(Now, I'd defer to the geoengineers as to whether the altitude dimension is dispensable. I don't have any expertise in this science. Judging from the Aurora red line, I'm assuming that there can be feasible solutions at all altitudes, which leads me to conclude that altitude isn't all that.)

***

So what is the biggest problem with this chart? It is the faux accuracy.

Given the tremendous amount of uncertainty surrounding these projected costs, one would expect very big error bars around the cost estimates. Using single dots with no error bars is hard to stomach.

 

 

Weekend reading

Lots of ideas from readers have been gathering dust in my mailbox. Here are a bunch of links, with a few comments of mine.

***

This first link I'm not sure what to make of. I think the architects and graphic designers amongst you may be able to make sense of it. Not me. It came with this description: "dr. dr. crash and dr. trash of m-a-u-s-e-r
analyzed worlds most junk magazines and visualized their data." For the intrepid (and I claim no liability):

    "Jetistics: The Analysis of Junk. The Junk of Analysis?"

***

This is yet another instance of the trend of infographics infiltrating PR releases.

This is yet another example of a map adding little or no value to the data. The presence of geographic data is not an excuse to give a lesson on maps.

It would be one thing if the geographic location helps the readers understand the data but in most such charts, the map merely says "Reader, I presume you are map illiterate, so let me tell you  South Africa is at the southern tip of the African continent..."

Also notice that the bar charts are sorted by average size of invoices, which is definitely less meaningful than total amount invoiced. This, I suspect, is the failure to ask the pertinent question, which is at the top of the Trifecta checkup.

  

  ***

"19 Must-See Biotech Infographics", according to Kelly Davis of the BioBlogging Project.

  #2 on this list is a chart (rather old data) on GM food, an issue of concern to me. In the Trifecta checkup, this addresses an important question, and displays very relevant data but uses a poor chart... too many colors, colors not carrying any meaing, hard-to-read labels.

Of the other links, these are more interesting: #10, #12, #17, #19, #8, #9.

 

 

***

Cleanup job for dirty oil

Note: Posting will be slow in the next few weeks due to holidays.

Vado in vacanza a Roma presto. Se abita alla citta', mi manda una email per favore. Forse ci possiamo incontrare.

***

Reader Daniel L. submitted this chart a while back, and it's an instructive one.

This Chicago Tribune chart accompanies an analysis by Greentopia of the "green-ness" of 10 oil companies. The company concocted a ranking based on the composite of 6 factors (e.g. emissions, efficiency).

The bar charts have a couple of unusual, but unconvincing, features:

1. Typically, in a bar chart, longer is better but when bars are used to depict ranks, shorter is better. In charting, it is usually safer to satisfy expectations, or risk being misinterpreted.

2. For rank n, the length of the bar is (n+2). The chart designer just decided that the piece including the actual rank should be thrice the size of the other highlighted pieces. There are a total of 12 strips in each bar.

Why is it a bad thing to add 2 units to every bar? Consider rank 1 vs rank 3. If n = n, then the ratio of bar lengths is 3 to 1. If n = n+2, then ratio is 5 to 3, and not the same! Thus, once the extra unit is added to each bar, the comparative lengths mean something different.

***

One thing is left unexplained: how is the overall ranking derived from the factor ranking? Hess, with three #1 ranks, would seem to be in contention for overall #1.

***

Daniel thinks a bumps chart would work nicely, and it certainly would for any kind of ranking data. A slight variation of the Tribune chart would also work nicely... think of the bar as consisting of ten little lights. For each rank, the appropriate light is switched on.  (This is in essence a dot plot.)

In both these variations, the charts are self-sufficient -- there is no need to print all the ranks on the chart as shown above.

***

Daniel also commented that it is difficult to incorporate the stock price data onto a bumps chart. Why bother? What is the point of including the stock price data anyway? If it is included, readers have to be given tools to interpret such data, in particular, some explanation ought to be provided for large jumps or slips. In addition, the scales should be tailored to allow comparisons of relative value, rather than sticking to equal scales for each stock (Dona Wong covers this topic well in her book.)