Junk Charts

Unscientific American 2: a review of key concepts

This is part 2 of a two-part piece on a set of charts published in Scientific American, describing the results of a poll asking their readers about their trust of various types of experts. In part 1, I looked in detail at one of the charts: this chart has many problems-- problems with the execution of the bar chart, problems with the choice of comparisons, problems with the design of poll questions.

***

Long time ago, I wrote about racetrack graphs. An example shows up here. Note the length of the yellow track relative to the length of the pink track. It should be about 2:1 but this type of chart makes a many-fold illusion.

Bar chart, please.

 

 

 

 

 

 ***
For this chart illustrating fears of technology, the designer chose to emphasize the data (percentages), with a subtle embedding of a column chart in which the columns were over-stretched to form squares.

Love these pictures, very telling.

The data jump at us, rendering the graphical element--the embedded column chart--secondary, even redundant.

But columns are very useful things. They are self-sufficient. We can judge the lengths of columns (or bars) very easily, so we do not need to print data next to columns.

However, thin columns work better than thick columns, so stretching columns to squares is not a good idea, even if the relative areas are not distorted.

Also, even the choice of colors here can be quibbled with. For me, the dark pink draws more attention than the medium orange, which means that my eyes and my brain (reading the printed data) start having an argument.

So, here is a sketch of what a bar chart would look like. See, you don't need to print the data on a bar chart--it's not hard at all to see that about 50% fear nuclear power.  I have lightened the pink.  Just add in the pictures, and the captions on the right side of the chart, and you have an improved chart.

 

 

 

 

 

 

 

 

***

Now, start the stopwatch, and measure how much time it takes to figure out what this chart is telling you:

This is truly a head-turning chart if you know what I mean...

In the article, this chart has the title "Climate Denial on the Decline". Maybe, maybe not--but oh dear, how are we to tell?

Instead of fixing the chart, it is perhaps more important to fix the poll question. It's not a good idea to describe a trend when one has data from this year and last year only. If they are isolating the effect of some major event (say, a Kyoto protocol) that happened during that year, it would be fine but that's not what is attempted here.

***
For part 1, see here.

 

 

 

Unscientific American 1: misreadings

Chris P. sent me to this set of charts / infographics with the subject line "all sorts of colors and graphs."  I let the email languish in my inbox, and I now regret it. For two reasons: one, the topic of how scientists can communicate better with, and thus exert stronger influence on, the public is very close to my heart (as you can tell from my blogs and book), and this article presents results from a poll on this topic done on on-line readers of Scientific American, and Nature magazines; two, some of the charts are frankly quite embarrassing, to have appeared in venerable publications of a scientific nature (sigh); three, these charts provide a convenient platform to review some of the main themes on Junk Charts over the years.

Since the post is so long, I have split it into two parts. In part 1, I explore one chart in detail. In part 2, I use several other charts to illustrate some concepts that have been frequently deployed on Junk Charts.

***

Exhibit A is this chart:

First, take a look at the top left corner. At first glance, I took the inset to mean: among scientists, how much do they trust scientists (i.e., their peers) on various topics?  That seemed curious, as that wouldn't be a question I'd have thought to ask, certainly not as the second question in the poll.

 On further inspection, that is a misreading of this chart. The "scientists" represented above are objects, not subjects, in the first question. As the caption tells us, the respondents rated scientists at 3.98 overall, which is an average rating across many topics. The bar chart below tells us how the respondents rated scientists on individual topics, thus providing us information on the spread of ratings.

 Unfortunately, this chart raises more questions than it answers. For one, you're working out how the average could be 3.98 (at that 4.0 white line) when all but three of the topic ratings were below 3.98. Did they use a weighted average but did not let on?

Oops, I misread the chart, again. I think, what I stumbled on here is the design of the poll itself. The overall rating is probably a separate question, and not at all related to the individual topic ratings. In theory, each person can assign a subjective importance as well as a rating to each topic; the average of the ratings weighted by their respective importance would form his or her overall rating of scientists. That would impose consistency to the two levels of ratings. In practice, that makes an assumption that the topics span the space of what topics each person considers when rating the scientists overall.

***

The bar chart has a major problem... it does not start at zero.  Since the bars are half as long as the longest, you might think the level of trust associated with nuclear power or climate change would be around 2 (negative). But it's not; it's in the 3.6 range. This is a lack of self-sufficiency. The reader cannot understand the chart without fishing out the data.

Now, ask this question: in a poll in which respondents are asked to rate things on a scale of 1, 2, 3, 4, 5, do you care about the average rating to 2 decimal places?  The designer of the graphic seems to think not, as the rating was rounded up to the nearest 0.5, and presented using the iconic 5-star motive. I think this is a great decision!

But then, the designer fell for loss aversion: having converted the decimals to half-stars, he should have dropped the decimals; instead, he tucked them at the bottom of each picture. This is no mere trivia. Now, the reader is forced to process two different scales showing the same information. Instead of achieving simplification by adopting the star system, now the reader is examining the cracks: is the trust given citizens groups the same as journalists (both 2.5 stars) or do "people" trust citizens groups more (higher decimal rating)?

***

The biggest issues with this chart concern the identification of the key questions and how to collect data to address those questions. This is the top corner of the Trifecta checkup.

1) The writer keeps telling us "people" trust this and that but the poll only covered on-line readers of Scientific American and Nature magazines. One simply cannot generalize that segment of the population to the common "people".

2) Insufficient attention has been paid to selecting the right wording in the questions. For example, in Exhibit A, while the overall trust question was phrased as trusting the "accuracy" of the information provided by scientists vs. other groups, the trust questions on individual topics mentioned only a generic "trust".  Unless one thinks "trust" is a synonym of "accuracy", the differential choice of words makes these two set of responses hard to compare. And compairing them is precisely what they chose to do.

***

In part 2, I examine several other charts, taking stops at several concepts we use on Junk Charts a lot.

 

The cross-hairs of religions

Long-time reader Nick B. found this attractive flow chart.

The chart was produced by the Internet Monk blog. The data was culled from this report (PDF) by the Pew Forum.

The cross-hairs trumpet excitement but the reader is left without much. One could tell that the unaffiliated proportion (red) has more than doubled, mostly at the expense of Catholics (green); that most religions retain the vast majority of their faithful (at least by internal proportions); and that people of one or another faith  move to one or another faith.

Yet, any of these high-level insights do not require a chart that contains data on movement between each pair of religions.

One smart thing about this chart is the inclusion of "unaffiliated / no religion", which completes the picture; otherwise, some previously faithful people would drop off the chart (literally).

The other smart thing is its self-sufficiency: none of the data is printed on the chart, and I doubt readers miss them.

***

Here, I attempt an alternative, which is a variant of the Web of Debt chart discussed here.

Note the economy of colors, lines, etc. I have chosen to use the number of people with a particular childhood faith as the base for all the percentages; other bases can be selected. For example, the unaffiliated has grown by 144% of the childhood base, with about half of that growth coming from previous Protestants; meanwhile, an exodus of Catholics has occurred. (PS. the data for other faiths being incomplete in the aforementioned report, I made up some of the data so as to finish the chart.)

If the line thickness is made proportional to the percentages, that would eliminate the need to have all those numbers on the chart.

Cleanup job for dirty oil

Note: Posting will be slow in the next few weeks due to holidays.

Vado in vacanza a Roma presto. Se abita alla citta', mi manda una email per favore. Forse ci possiamo incontrare.

***

Reader Daniel L. submitted this chart a while back, and it's an instructive one.

This Chicago Tribune chart accompanies an analysis by Greentopia of the "green-ness" of 10 oil companies. The company concocted a ranking based on the composite of 6 factors (e.g. emissions, efficiency).

The bar charts have a couple of unusual, but unconvincing, features:

1. Typically, in a bar chart, longer is better but when bars are used to depict ranks, shorter is better. In charting, it is usually safer to satisfy expectations, or risk being misinterpreted.

2. For rank n, the length of the bar is (n+2). The chart designer just decided that the piece including the actual rank should be thrice the size of the other highlighted pieces. There are a total of 12 strips in each bar.

Why is it a bad thing to add 2 units to every bar? Consider rank 1 vs rank 3. If n = n, then the ratio of bar lengths is 3 to 1. If n = n+2, then ratio is 5 to 3, and not the same! Thus, once the extra unit is added to each bar, the comparative lengths mean something different.

***

One thing is left unexplained: how is the overall ranking derived from the factor ranking? Hess, with three #1 ranks, would seem to be in contention for overall #1.

***

Daniel thinks a bumps chart would work nicely, and it certainly would for any kind of ranking data. A slight variation of the Tribune chart would also work nicely... think of the bar as consisting of ten little lights. For each rank, the appropriate light is switched on.  (This is in essence a dot plot.)

In both these variations, the charts are self-sufficient -- there is no need to print all the ranks on the chart as shown above.

***

Daniel also commented that it is difficult to incorporate the stock price data onto a bumps chart. Why bother? What is the point of including the stock price data anyway? If it is included, readers have to be given tools to interpret such data, in particular, some explanation ought to be provided for large jumps or slips. In addition, the scales should be tailored to allow comparisons of relative value, rather than sticking to equal scales for each stock (Dona Wong covers this topic well in her book.)

Oil spills bring out the worst

Infographics types are having a field day telling us how "big" the big BP oil spill truly is. Unfortunately, whenever size is concerned, the specter of pie charts beckons.

Via reader Fia, we are alerted to the discussion at Discover's Cosmic Variance blog about the following chart (the entire poster hosted here at Iglucruise):

Sean's original post focuses on the fundamental issue in making pie charts of making the square of the radius proportional to the data, rather than the radius. The current version (shown left below) at Iglucruise has been corrected and the sizes of the pies are much closer together.

A more pressing issue with pie charts is that we do poorly in assessing relative sizes of circular areas. Try figuring out how much smaller is the BP spill relative to the largest circle in the chart...

***

Applying the Trifecta Checkup, we find problems with the choice of chart type, and also the choice of the practical question. The question of comparing this spill to past spills is premature given that the oil is still gushing. In addition, the nature of the 1991 spill (Iraq's deliberate dumping) is in a different class from accidental spillage.

***

Reader Curtis R. pointed us to another infographics about the spill, this one made by Fast Company, a business magazine. Their concept is reminiscent of the racetrack graph.

In other racetrack graphs, the tracks are close together, and terminate at different angles to the center; the data is encoded in the angles.

In this chart, the angle is everywhere 270 degrees, but the distance between the tracks are varied, and the data is encoded in the circumferences.

Alternatively, this can be thought of as a set of concentric circles with a right angle cut off. Since the circumference is proportional to the radius, the data can be found either in the radii or in the track lengths.

Judging the relative lengths of the tracks is not any easier in judging the relative areas of circles. One can stare at the orange and green arcs for however long, but can one tell the orange is shorter than the green by 3.3 million?

In other words, this chart is not self-sufficient. If the data is not printed directly onto the chart, it is impossible for readers to interpret it.

***

I would like to see a chart that looks like the following, which emphasizes the open-ended, progressive nature of the spill, using past spills as benchmarks:

Reading this before the long weekend may save your life

Here's a chart that can save your life.

Here's my version (pretty much any such grouped column charts can be replaced by line charts):

(Chart purists: I like profile charts which means I like to connect categorical data with lines.)

Anyhow, this data supposedly came from an FDA study, which the FDA has apparently now disowned, according to this AOL News report. Rats were used in this study, and the rate at which they developed significant tumor or lesion was measured. The graph illustrated a clear trend that the higher the doses of Vitamin A, the faster the rats developed cancer; this correlation was intact whether they were exposed to high or low levels of UV rays.

Notice that I switched the primary categorical axis to Vitamin A doses rather than high/low UV because the study concerned Vitamin A primarily, and levels of UV secondarily.

Using the Trifecta Checkup, we can see that they have the right question, and the right data but a suboptimal chart. Also, the original chart fails the self-sufficiency test: no point in printing the data on top of the columns when there is a vertical scale.

***

How will this save your life?

Vitamin A is widely added to sunblocks -- not because they have any screening value -- but because they may slow aging of the skin. But the study found that Vitamin A actually partially nullifies the screening ability of sunblocks.

About half of the 500 most popular sunblocks sold in the U.S. contain Vitamin A and only 39 out of the 500 are deemed safe by the Environmental Working Group, which has compiled a database of these products. (There are several other potentially harmful ingredients.)

The FDA denied that such a study existed although the reporter as well as EWG have copies of it. If this study is authentic, the FDA knew about this perhaps ten years ago.

Reference: "Study: Many Sunscreens May Be Accelerating Cancer", Andrew Schneider, AOL News, May 24 2010.

PS. I should explain to my non-U.S. readers that the U.S. is celebrating Memorial Day, the beginning of summer, on Monday so lots of people are going to beaches and other vacations.

Junk Charts talk

The NYU talk was quite successful, and it was great to meet some of our readers in person. The slides can be downloaded from here (800 KB pdf). There were many thoughtful questions from the audience.

The talk was called "Five Years of Chart Reading". Probably should have subtitled it, "condensed to 20 minutes". While preparing the talk, I realized that I have written over 500 posts since the blog came into being. I know, that works out to be only once every 3 or 4 days, what a lazy bum. But since most of my posts are substantive, the collection really did cover a lot of ground.

For many readers, especially those who have followed the blog from its early days, the contents of the talk should be familiar. I had time to develop only one general theme--not surprisingly, I picked "self-sufficiency". 

I did introduce one new idea: the "Trifecta Checkup" framework for chart critique. I believe this is the constant behind pretty much all the posts on Junk Charts. All outstanding charts have all three elements in harmony. Typically, a problematic chart gets only two of the three pieces right. I will be elaborating on this framework in subsequent posts.

In the first half of the talk, Dona Wong gave examples of her work at the Wall Street Journal, and Siegel+Gale, which is a brand consulting company that designed the 1040EZ form. Like myself, she believes the key to good graphics lies with the upfront "data strategy", and not tools or software. I particularly liked her examples of portraying relative energy consumption of various household appliances, displaying daily spending patterns in a way that answers the question directly, and revamping the weather section to be more "business-friendly".

Check out Wong's new book on graphics here.

Reader's indigestion

Reader Chris B. pointed us to this unfortunate chart, based on a one-question on-line poll conducted by Reader's Digest. 

The data is highly structured: for each country, respondents, identified as male or female, are asked about their favorite methods to discipline their kids. (At first, I thought the "male" and "female" meant what methods they would apply to sons versus daughters but based on the summary paragraph, I now feel they refer to the genders of the respondents.)

The textual summary is extremely well-written, and successfully points to the most salient information (my italics and bolding):

Spare the rod, period. That's what parents across the globe told us when we asked how they discipline their children. Respondents in all 16 countries in this month's global survey picked a good talking-to as the best tactic for teaching a lesson, by a wide margin. Taking away a privilege placed second. Two other traditional forms of discipline-sending kids to their rooms and spanking-were the least favored choices in all but two countries. Among respondents who did favor physical punishment, men outnumbered women in every country except Canada, France, and India. Not a single woman in the United States expressed a preference for spanking.

***

Unfortunately, the graphical summary is a complete failure.

One feature plotting against the designer is that the general profiles of the responses are very similar between countries, and so the differences are well hidden inside this small-multiples display.

It also takes on an elongated form, making it almost impossible to compare the top two countries with the bottom two countries.

When data has such strong structure, it is a blessing to the chart designer. In the first chart, I made a set of profile charts, in small multiples. On average, parents everywhere act very similarly. There are some subtle differences: one common pattern, occurring in the Philippines, Malaysia, India, France, Brazil, etc., is the preference for a talking-to over all other methods; another pattern, applying to Netherlands, Spain, Australia, Canada, etc. is a talking-to, followed by taking away privileges with sparing use of the other two methods.

In some countries, like Australia, Brazil, Canada, Spain, Italy, etc., the gender of respondents mattered little but in the United States for instance, female respondents are more likely to prefer a talking-to while men liked using sticks. 

Is it really the case that parents punish sons and daughters using the same methods? This poll seems to think so.

***

If we want to expose the minute differences at the level of country-gender, then something like this would do:

The purpose is to surface any outliers. I really can't say there are any here. The supposed reversion of responses by gender in India, France, and Canada is hardly worth noting since the physical punishment category is hardly used. (Reflection of reality, or response bias due to sensitive subject?)

Notice that these new charts do not have the data printed on them - the graphical elements are sufficient to show what the data is; readers are not auditors.

Whither infographics

Reader Aleks found this infographics poster (by Phil Gyford) which is sure to excite (or exercise) some people.

I admire the work of infographic artists in processing and structuring huge amounts of data.  I think many such presentations, especially the interactive ones, are terrific in empowering readers (users?) to slice and dice data.

Some infographics are produced by people who probably see themselves as artists first, and the charts as objects of art. (Diagrams of network structure come to mind.) That's fine, too. And I can appreciate them as if I am in MoMA.

Some infographics are daunting works of blood and sweat. They make our jaws drop, we wonder how they did that. They remind us of the Great Wall of China, the pyramids, etc. The layers upon layers of details are there to  dazzle us, to prove the point.

***

But I like my charts to tell me something important about the data. I want charts to be "self-sufficient". If readers must consult the raw data (printed on the chart) in order to get the message, then all the graphical constructs (bars, dots, axes, etc.) are redundant!

In addition, I don't like to make readers do a lot of work. The task of extracting insights from the data should fall to the designer, not the readers.

***

That said, judging from the circulation of infographics on the Web, these displays clearly are popular so one can't argue with that.

What's your view of the state of infographics? Love it or hate it?

 PS. Robert at EagerEyes made comments on this topic, likening visualization to a "cargo cult". He calls for drawing a line in the sand. My feeling, as outlined above, is that there could be different classes of infographics -- my personal interest is in those that contribute to effective communication but others may be interested in artistic rendering, story-telling and exploration, technical wizardry, etc.

Definitely not working

Catherine Rampell ponticificated on the nature of working families in the NYT Economix blog recently.  Are both members of the family working?  Or not working?

One thing is for sure.  Her chart just doesn't work.  Too many colors, unnecessary gridlines, should label the series directly in lieu of the legend.  Fails our self-sufficiency criterion: once every piece of data is printed on the chart via data labels, everything else is rendered redundant!

And the chart conveys the wrong message!  Looking at this chart, the reader would think the message is: the distribution of work status among married couple families has not changed materially between 2001 and 2009.  That is pretty much Rampell's conclusion.  She said that the gray bar moving from 17.5% to 18.8% in 2009 was "not a huge jump from year to year", and the the pink bar going from 8.2% to 9.9% in 2009 was a "slightly bigger increase".

This unfortunately paints the whole picture.  For a start, most stacked column charts can be replaced with line charts like this:

Because the vertical scale is no longer constrained at the upper end, the changes from year to year will appear more magnified in this version.  We may think some of the 2009 data to be different from the other 8 years, and we would be right.  Sometimes a small change is a big change - because the thing being measured just doesn't vary much.  The next chart highlights this:

I went back to the raw data, expressed in the number of married couple families, and then created an index, with the 2003 count as 100 for each category.  The black line shows the underlying demographic growth in the total number of such families over the 9 years.  It's now obvious that there has been a pretty remarkable jump in families with only the wives working.

The difference between this chart and the other two charts is the choice of comparisons.  The first two purports to look for year-on-year changes; such comparisons are hard because how big a shift is to be considered noticeable?  In the last version, we regard as noticeable a growth curve over time that differs significantly from the underlying growth trend (black line).

More sophisticated statistical tests can also be used to establish whether the shift in distribution is "statistically significant" but I think the visualization is sufficient.

Reference: "Working Families, Not Working", New York Times, Jan 15 2009.