Junk Charts

Loss aversion and faux accuracy

Reader Bernie M. is not a fan of this Economist chart.

The chart was prepared by Aurora Flight Sciences, an aircraft manufacturer, commissioned by a professor who supports the concept of maintaining a fleet to pump sulphuric acid into the stratosphere as a way to induce artificial cooling to counteract human-induced global warming.

The chart appears to compare many different ways of shooting the acid into the skies along two dimensions: cost and altitude.

Bernie wrote:

I find the choice of axes extremely counterintuitive. Altitude one would expect on the y-axis. And mixing up the scatter chart elements with the connected line chart doesn' really help either.

The convention regarding axes is to put the outcome variable on the vertical, and the explanatory variable on the horizontal. Thus, in this case, if the cost of a particular solution is primarily determined by the "altitude" (presumably of where the acid would be released), then the designer has followed convention. It is unfortunate that "altitude" is more intuitively put on the vertical axis, but I suspect that defying convention might cause more confusion.

On the other hand, if altitude and cost are not related to each other but two different metrics to evaluate geoengineering concepts, then Bernie's point is right on - swap the axes!

***

The use of connected lines for two of the solutions but not the rest is a symptom of what I have called "loss aversion". The horror of leaving some of the data on the cutting floor.

The only mention of altitude in the article refers to Aurora's assertion that it is sufficient to use newly designed aircraft flying at 20-25 kilometers. If that is Aurora's preferred solution, there is little reason to show all the other altitude configurations that are suboptimal.

Perhaps the designer wants to make the point that the Boeing 747 solution is inferior to the Aurora solution because Aurora could design aircraft to fly at 10-15 km at a lower cost?  If so, then the chart is very misleading in not providing a comparable cost for Boeing's solution if required to fly at 20-25 km.

When comparing different entities, it is always a bad idea to treat the entities differently. Comparison is only possible on equal footing.

In fact, I think the chart would be a lot clearer if they dropped the altitude dimension on the floor. For each solution, plot the yearly cost at the optimal altitude selected by the respective engineers. Use a bar chart. With a single dimension, it is much easier to accommodate the very long data labels.

(Now, I'd defer to the geoengineers as to whether the altitude dimension is dispensable. I don't have any expertise in this science. Judging from the Aurora red line, I'm assuming that there can be feasible solutions at all altitudes, which leads me to conclude that altitude isn't all that.)

***

So what is the biggest problem with this chart? It is the faux accuracy.

Given the tremendous amount of uncertainty surrounding these projected costs, one would expect very big error bars around the cost estimates. Using single dots with no error bars is hard to stomach.

 

 

Showing dynamics on a business chart

Dave S. achieved a rare feat, which is to send in a great-looking set of charts. This post at Asymco is worth reading in its entirety; the author Horace discusses the process by which he worked through several charts, arriving at the one he's most happy with.

***

The secret to the success here is the careful framing of the question, and the collection of the appropriate data to address that question. The question is the competition between wireless phone vendors in the last three years. It was established that the right way to view this competition is in two dimensions: share of revenues, and share of profits. Note the word "share". Share of profits is not a metric that is often discussed but it is the right metric to compare to the share of revenues -- getting both numbers onto the same comparable scale is what makes this work.

Needless to say, the raw data one would collect come from the financial statements of the eight individual vendors. Plotting these numbers directly would be a mistake. So you take the numbers, making sure that you're really counting wireless revenues and wireless profits, and then compute the shares. (I am not actually sure that they have wireless profit data because large companies like Apple and Nokia typically don't break out their profit shares, even if they provide the revenue shares by line of business.)

Horace also avoided the plague of plotting all time-series data as line charts (similar to the plague of plotting all geographic data on maps). By plotting revenues and profits simultaneously, he no longer can plot time (years) on one of the two axes, and that is a good thing.

***

This is the final graph Horace landed on. It puts all the vendors at the origin in 2007 and then tells us where they landed in 2010 in terms of revenue and profit share growth/decline.

It would be even better if he makes the scales work harder: e.g. have equal lengths for the 10% change along both the vertical and horizontal axes. Alternatively, you can scale it such as each unit on either axis represent equal dollars.

This is a very focused chart that answers the question about the relative change in positioning of each vendor. What it doesn't answer is the starting position or ending position of each. Note, while Nokia is depicted as losing share on both revenues and profits, Nokia still has twice the revenue share of the other vendors, and out-earns everyone except Apple!

I am not saying this is a bad chart. It is designed to answer the relative question, not the absolute question. That's all.

***

There is one way to have the cake and eat it too.  Horace almost created that chart. He showed two scatter plots, one for 2007 and one for 2010.

If he just overlays one on the other, and use lines to connect the dots for each phone vendor, he will have a chart that shows absolute and relative values all at once. Here's a crude illustration of this: (missing the labels to show that the arrow end of the line represents 2010 positions)

I like this kind of chart a lot. It is great for showing dynamics in a set of variables, without actually making the chart dynamic.

(Even on this chart, it is better to harmonize the two scales.)

 

Unscientific American 1: misreadings

Chris P. sent me to this set of charts / infographics with the subject line "all sorts of colors and graphs."  I let the email languish in my inbox, and I now regret it. For two reasons: one, the topic of how scientists can communicate better with, and thus exert stronger influence on, the public is very close to my heart (as you can tell from my blogs and book), and this article presents results from a poll on this topic done on on-line readers of Scientific American, and Nature magazines; two, some of the charts are frankly quite embarrassing, to have appeared in venerable publications of a scientific nature (sigh); three, these charts provide a convenient platform to review some of the main themes on Junk Charts over the years.

Since the post is so long, I have split it into two parts. In part 1, I explore one chart in detail. In part 2, I use several other charts to illustrate some concepts that have been frequently deployed on Junk Charts.

***

Exhibit A is this chart:

First, take a look at the top left corner. At first glance, I took the inset to mean: among scientists, how much do they trust scientists (i.e., their peers) on various topics?  That seemed curious, as that wouldn't be a question I'd have thought to ask, certainly not as the second question in the poll.

 On further inspection, that is a misreading of this chart. The "scientists" represented above are objects, not subjects, in the first question. As the caption tells us, the respondents rated scientists at 3.98 overall, which is an average rating across many topics. The bar chart below tells us how the respondents rated scientists on individual topics, thus providing us information on the spread of ratings.

 Unfortunately, this chart raises more questions than it answers. For one, you're working out how the average could be 3.98 (at that 4.0 white line) when all but three of the topic ratings were below 3.98. Did they use a weighted average but did not let on?

Oops, I misread the chart, again. I think, what I stumbled on here is the design of the poll itself. The overall rating is probably a separate question, and not at all related to the individual topic ratings. In theory, each person can assign a subjective importance as well as a rating to each topic; the average of the ratings weighted by their respective importance would form his or her overall rating of scientists. That would impose consistency to the two levels of ratings. In practice, that makes an assumption that the topics span the space of what topics each person considers when rating the scientists overall.

***

The bar chart has a major problem... it does not start at zero.  Since the bars are half as long as the longest, you might think the level of trust associated with nuclear power or climate change would be around 2 (negative). But it's not; it's in the 3.6 range. This is a lack of self-sufficiency. The reader cannot understand the chart without fishing out the data.

Now, ask this question: in a poll in which respondents are asked to rate things on a scale of 1, 2, 3, 4, 5, do you care about the average rating to 2 decimal places?  The designer of the graphic seems to think not, as the rating was rounded up to the nearest 0.5, and presented using the iconic 5-star motive. I think this is a great decision!

But then, the designer fell for loss aversion: having converted the decimals to half-stars, he should have dropped the decimals; instead, he tucked them at the bottom of each picture. This is no mere trivia. Now, the reader is forced to process two different scales showing the same information. Instead of achieving simplification by adopting the star system, now the reader is examining the cracks: is the trust given citizens groups the same as journalists (both 2.5 stars) or do "people" trust citizens groups more (higher decimal rating)?

***

The biggest issues with this chart concern the identification of the key questions and how to collect data to address those questions. This is the top corner of the Trifecta checkup.

1) The writer keeps telling us "people" trust this and that but the poll only covered on-line readers of Scientific American and Nature magazines. One simply cannot generalize that segment of the population to the common "people".

2) Insufficient attention has been paid to selecting the right wording in the questions. For example, in Exhibit A, while the overall trust question was phrased as trusting the "accuracy" of the information provided by scientists vs. other groups, the trust questions on individual topics mentioned only a generic "trust".  Unless one thinks "trust" is a synonym of "accuracy", the differential choice of words makes these two set of responses hard to compare. And compairing them is precisely what they chose to do.

***

In part 2, I examine several other charts, taking stops at several concepts we use on Junk Charts a lot.

 

Buy one get one free costs you one way or another

This chart by Nielsen's mobile unit explaining the variability of cell phone usage by age group in the U.S. looks innocently standard. It's a grouped bar chart.

I have long argued that this type of chart should be used sparingly. Notice that the key point of the chart requires comparing voice (or text) usage across age groups; this means readers are asked to compare the relative lengths of the blue (or green) bars. With grouped bars, readers are asked to literally jump over bars to perform the comparisons. The more groups, the more bars to jump. I don't like to exercise my readers in this way.

Something else more damaging is lurking beneath. We can see this by splitting up the charts. Here, I'm reproducing the two charts as is, but on a small multiples format.

Recall the key practical question here is to compare usage across age groups. Now look at the massive amounts of empty space on the Voice usage chart: the space crowds out the data, meaning that the lengths of the voice bars are compressed, affecting our perception of differences.

What is causing this problem? It's the use of one axis for disparate data. Voice is measured in minutes and texts in units, and in trying to avoid double axes, the designer manages to make things worse by lumping them together.

In the Trifecta checkup, they outlined an interesting question, and collected the right data to address it but the execution of the graphic was wanting.

It's a case of buy one get one free, except that when you go home, you realize you would never buy that second item on its own.

 

PS. I have more to say about the statistical thinking behind this Nielsen press release at the sister blog.

Radar charts, Ripleyesque replies, and straight-ticket votes

As you may have noticed, posting has been sluggish lately, and this means I have a backlog of submissions. So please be patient if you sent something to me.

Alex C., at NUS, wasn't so patient so he wrote up his own post on this gaping chart from Accenture reporting on a survey of people's attitudes towards health care. In particular, Alex does a great job taking down the questions in the survey: for appetizers, he noted "So, the interviewees were asked 'Do you think it’s important to focus on delivering real improvements in the overall health of the nation?' I wonder who answered no?"

(Amusing aside: I fibbed. Alex wasn't impatient. He just couldn't believe how rude I was. After spending some unsavory time manually eyeballing the data from these charts, I accidentally emailed the spreadsheet to Alex, thinking that I was emailing myself. So Alex received a reply from me, with the spreadsheet attached but with no comments, and he figured I couldn't be bothered. Perhaps I "gave minimalist, Ripleyesque replies", he wondered.)

Well, I digress. There are a few other problems with this "radar" chart that Alex was too kind to overlook:

• All of the information is in the radius from the center of the circle to individual dots and yet lines were drawn to connect dots into a ragged circle, drawing attention away from the information
• Only on one quadrant was the scale of the radii provided, which frustrated me to no end when I tried to "eyelift" the data off the charts
• There are slips in craftsmanship as some of the dots seem to fall out of place, e.g. on this particular chart, the two dots for "issue 16" does not seem to be aligned with the label 16, similarly something seems off with the dots for "issue 13". (These minor slips become very obvious when you are lifting data off the charts.)
• Each category contains four questions and occupy one quadrant but the way the information is arranged on this chart type, one cannot visually aggregate the four individual scores to arrive at a category score.

***

Even more fundamentally, what data are being plotted? Turns out this is convoluted:

• For each of the 16 "issues", respondents are asked to rate the importance and the government's performance on separate 5-point scales.
• The top 2 points are considered "favorable", and the data depicted are the top-2-point proportions.

So the "gap" that Accenture consultants have stuck their fingers into is the proportion of respondents rating the government's role in an issue as "very important" or "essential" minus the proportion of respondents rating the government performance on that same issue as "fairly well" or "very well".

I just don't share the enthusiasm about this metric. Not merely because we are bound to think an issue is more important if the government has done a poor job at it.

***

Just to surface another problem: from these charts, it's clear that people in different countries approach 5-point scales differently. Perhaps in some countries (see right), they just fill out a "straight-ticket" vote for all issues?!

Thanks Alex for bringing this to our attention.

The PDF of the Accenture report is here.

Reading this before the long weekend may save your life

Here's a chart that can save your life.

Here's my version (pretty much any such grouped column charts can be replaced by line charts):

(Chart purists: I like profile charts which means I like to connect categorical data with lines.)

Anyhow, this data supposedly came from an FDA study, which the FDA has apparently now disowned, according to this AOL News report. Rats were used in this study, and the rate at which they developed significant tumor or lesion was measured. The graph illustrated a clear trend that the higher the doses of Vitamin A, the faster the rats developed cancer; this correlation was intact whether they were exposed to high or low levels of UV rays.

Notice that I switched the primary categorical axis to Vitamin A doses rather than high/low UV because the study concerned Vitamin A primarily, and levels of UV secondarily.

Using the Trifecta Checkup, we can see that they have the right question, and the right data but a suboptimal chart. Also, the original chart fails the self-sufficiency test: no point in printing the data on top of the columns when there is a vertical scale.

***

How will this save your life?

Vitamin A is widely added to sunblocks -- not because they have any screening value -- but because they may slow aging of the skin. But the study found that Vitamin A actually partially nullifies the screening ability of sunblocks.

About half of the 500 most popular sunblocks sold in the U.S. contain Vitamin A and only 39 out of the 500 are deemed safe by the Environmental Working Group, which has compiled a database of these products. (There are several other potentially harmful ingredients.)

The FDA denied that such a study existed although the reporter as well as EWG have copies of it. If this study is authentic, the FDA knew about this perhaps ten years ago.

Reference: "Study: Many Sunscreens May Be Accelerating Cancer", Andrew Schneider, AOL News, May 24 2010.

PS. I should explain to my non-U.S. readers that the U.S. is celebrating Memorial Day, the beginning of summer, on Monday so lots of people are going to beaches and other vacations.

Hoisted from the archives: a revolution

In October 2007, I wrote about the "canvass" metaphor for graphing software. This was what I said:

With the advent of AJAX and other interactive technologies, one can only hope that new graphing software will use the "canvass" metaphor.  If we want to reduce the spacing between bars, we should be able to grab the bars and move them together.  If we want to change the ordering, we should be able to mouse over some menu and select a pre-defined ordering scheme, or to drag and move bars around as we please. etc. etc.

To push this metaphor further, this kind of software should facilitate the "exploratory" stage of graph-making. I blogged about this stage of making sketches before. One longs for software that allows one to flip through many different chart types quickly, to settle on the desired type, and then to make the nitty-gritty changes to the axes, colors, dots, etc.

The revolution has arrived in the form of JMP's Graph Builder function. It is not perfect yet, as even the example I use will show, but I'm excited because we are getting closer to that "canvass" metaphor.

***

I'm going to re-make this inedible pair of donuts from an otherwise quite nice infographics on the growth and nature of spam in the last 10 years. (New Scientist)

I have pointed out the biggest shortcoming of donut charts often: the fact that the most important clue to the size of each sector of the underlying pie chart, that is, the angle at the center of the pie, has been cut off from the chart, and often, as in here, obscured by a number.

There are dramatic shifts in proportions of spam types during the last decade but the effect is underwhelming as depicted.

In the Graph Builder, I can push around the data and create different chart types.  First, I made a small-multiples bar chart.

By clicking on the word "Year" and dragging it to a box called "Overlay", I made a paired bar chart:

What about a dot plot instead? This change requires a right click but easy enough:

Here's where I encountered a little inconvenience. It's probably ignorance on my part since I didn't read the manual. I couldn't figure out how to increase the dot size for all dots at once, only one at a time.

In any case, I'm still searching.  I want to do a small-multiples line chart. For this, I drag the word "Year" into the bottom of the chart labelled "X", and then right-click to add a line to the dot chart.

This is close to a desired chart type for this data.  The change from year to year is highly apparent, and the increased and decreased spam types are also obvious. I would color the increases differently from the decreases if I have the time.

I had a very difficult time (and failed in) getting the year labels to say 1999 and 2009 which are the logical points for this data. JMP seems to have a mind of its own.

Since it takes no time, I experimented some more.  By moving "Category" to "Wrap", I reproduced the above chart but in a matrix form:

Finally, I made the "Category" an "overlay" which resulted in this chart.  This is kind of like the Bumps chart but obviously a bad idea for this data: (I'm not even showing the really ugly legend).

So, my dream toy -- the "canvass" style graph maker -- is here! It only takes a few minutes to move the data around this canvass, and see these different chart types.

***
I indicated that this goes a long way but isn't perfect. Right now, sketching and exploring is easy but refining and detailing is not as easy.

What I would like to see: once the general form of the chart is chosen, maybe a second canvass is needed, with Photoshop as a metaphor, in which we can chisel out the nitty-gritty details, like the axis labels, dot sizes, line widths and so on.

Also, the number of chart types can, and I presume will, be increased over time. For instance, I don't think the current version allows a profile chart; it seems to adhere to the overly-rigid rule that a categorical data series should not be connected by a line.

(I should say that in the current release, one way to accomplish this is to save the resulting graph-sketch as a "JMP script" and then go into the code and change things around. But since we are doing point and click, and visual interaction, why not go all the way?)

Most existing graphing software fall into two extremes: the Excel style which is super-rigid, or the R style which allows minute control over every little thing. This, I think, is the third way.

 

Reading: WSJ Guide to Information Graphics

Dona Wong, who had stints on the graphics teams at both the Wall Street Journal and the New York Times, has contributed a how-to book on statistical graphics. It is called "The Wall Street Journal. Guide to Information Graphics".

The biggest strength of this book is the material on data collection and selection, which is an overlooked aspect of statistical graphics. The content of p.103, for example, is not typically found in similar books: on this page, Wong works through how to determine the scales for two stock-price charts in such a way that the distances represent relative changes in stock prices (rather than absolute changes). Chapter 3 ("Ready Reference"), which covers this type of material, is almost as big as Chapter 2, which runs through basic rules of making graphs that should be familiar to our readers. Her philosophy, then, leans toward Tukey's as espoused in his seminal book EDA, although Wong keeps to the most basic elements (percentages, indices, log scales, etc.), obviously aiming for a different audience than Tukey.

The guidelines relating to making charts are prescriptive and concise. The following snippet (pp.72-73) is typical of the style:

 

Wong focuses on saying what to do, but (usually) not why. Perhaps for this reason, the book has no references or notes, except for mentioning Ed Tufte as Wong's thesis adviser. Almost all the best practices described in the book would meet with our approval. One that has not been featured much on this blog is the preference for shades of the same color to many different colors of the same shade.

Despite the title, the book actually discusses statistical graphics (same as Junk Charts), not "infographics" (as covered by Information Aesthetics, for example). Almost all the graphical examples are conceptual, and not based on real-life examples. This editorial decision has the advantage of sharpening the educational message but the disadvantage of being less engaging.

A unique feature of Wong's book is Chapter 5 ("Charting Your Course"), which covers business charts used to organize operational data, rather than present insights -- things like Gantt charts (which she calls work plans), org charts, flow charts, 2-by-2 matrices, and so on. Things that are in the toolkit of management consultants. This is an under-studied area, and deserves more attention. I am reminded of Tufte's re-design of bus schedules. This type of charts is different in the need to print all pieces of data onto the chart, the prevalence of text data (and the difficulty of incorporating them into charts), and efficient search as a primary goal. And it is in this chapter that the decision to stay conceptual diminishes the impact: it would be very valuable for readers to see a complete Gantt chart based on a real project, and how it evolves over the course of the project. I have always found these types of charts to start out nicely but gradually sink as details and detours pile up.

***

There is one chart on p.59 I would like to discuss.

 Here, Wong allows the use of double axes in certain cases, basically when the two data series have linearly-related scales. She appends the advice: "Adhere to the correct chart type for each series -- lines for continuous data and bars for discrete quantities... The only exception is when both data series call for a chart with vertical bars. In such instances, convert one to a line." (Regular readers know I don't think much of this rule.)

Based on the chart above, Wong either considers both revenue and market share to be discrete quantities, or considers revenue to be discrete and market share to be continuous. In my mind, both series are continuous data and a chart with two lines is appropriate here.

Aftershocks

The graphs in this BBC article comparing several recent earthquakes hit us like aftershocks.

This chart tries to inform us the size of the quake in China was by far the largest. (The Richter scale is a power scale.)

The spirals feel like the Austin Powers time machine, disorienting, and also distracting because the bubble chart uses the entire area of the circles to represent magnitude.  Try to guess what the relative amplitudes are before I disclose them below. (The red spiral for Italy was arbitrarily chosen as the index, with relative amplitude 1.) Bubbles are just horrible constructs, and for such a simple chart, they are worse than printing the data.

Amazingly, this is a double-axes bubble chart! The spirals hide the fact that the three gray circles are of different sizes, presumably color-coded to fit the "Strength" of the quakes.  The other axis is "Relative Amplitude" represented by the red circles. Even though the two metrics are on hugely different scales, both the gray circles and the red spirals were anchored off the Italy red spiral (area = 1).

The following junkart version, which places the three quakes relative to the underlying relationship between strength and amplitude, is more informative with less fuss.

In the next chart, the Italians are shown to have no math skills (when in fact they have a strong tradition in math). How is it that 295 and 2000 have equal-sized bars? That's because the selected scale does not fit the data.

It's a mystery why Deaths and Injuries make friends while they ostracize the Homeless.  The three series (deaths, injuries and homeless) can be displayed separately.

A simple data table, with appropriate highlighting, gets this information across without the confusion.

This next chart is decent.  It is more effective if they make the Italy and Haiti blocks 20 across (same as the China blocks), stacking them one over the other. By doing so, the chart reduces to one dimension and we do not need to judge areas.

I think there is a calculation error with the Italy numbers. If 1 in every 190 affected died, then the number of affected is 190 x deaths, which from the above bar chart, equals 56,000. If only 56,000 were affected, how could 1.5 million be left homeless? (Wikipedia said 65,000 were made homeless.)

Overlapping non-concentric bubbles are also in need of rescue. Bubbles encode data in areas, areas are a square function of radii, the distance from the center to the circumference. When circles are not concentric, the centers do not coincide. This makes judging radii harder, which makes judging areas harder.

Look at Haiti vs. Italy. According to the printed data, the light gray area is about 60, which would be 40% of the dark gray circle. Who would have guessed? (I checked the areas, and indeed the Haiti area was 40% larger than the Italy area.)

By the way, in the first chart, the relative amplitudes were 40, 1 and 5.  Who would have guessed?

Andrew Gelman recently talked about good graphics being hard. Graphs are easy to make but hard to perfect. These examples show the need for care.

Reference: "Why did so many people die in Haiti's quake?", BBC News, 14 February, 2010.

Leaving ink traces

Stefan S. at the UNEP GEO Data Portal sent me some intriguing charts, made from data about the environment.  The following shows the amount of CO2 emissions by country, both in aggregate and per capita.  We looked at some of their other charts before.

These "inkblot" charts are visually appealing, and have some similarities with word clouds.  It's pretty easy to find the important pieces of data; and while in general we should not sort things alphabetically, here, as in word clouds, the alphabetical order is actually superior as it spaces out the important bits.  If these were sorted by size, we'll end up with all the big blots on top, and a bunch of narrow lines at the bottom - and it will look very ugly.

The chart also breaks another rule. Each inkblot is a mirror image about a horizontal line. This arrangement is akin to arranging a bar chart with the bars centered (this has been done before, here).  It works here because there is no meaningful zero point (put differently, many zero points) on the vertical scale, and the data is encoded in the height of each inkblot at any given time.

Breaking such a rule has an unintended negative.  The change over time within each country is obscured: the slope of the upper envelope now only contains half of the change, the other half exists in the lower envelope's slope.  Given that the more important goal is cross-country comparison, I think the tradeoff is reasonable.

Colors are chosen to help readers shift left and right between the per capita data and the aggregate data.  Gridlines and labels are judicious.

As with other infographics, this chart does well to organize and expose interesting bits of data but doesn't address the next level of questions, such as why some countries contribute more pollution than others.

One suggestion: restrict the countries depicted to satisfy both rules (per capita emissions > 1000 kg AND total emissions > 10 million tonnes).  In this version, a country like Albania is found only on one chart but not the other.  This disrupts the shifting back and forth between the two charts.