Junk Charts

Reading this before the long weekend may save your life

Here's a chart that can save your life.

Here's my version (pretty much any such grouped column charts can be replaced by line charts):

(Chart purists: I like profile charts which means I like to connect categorical data with lines.)

Anyhow, this data supposedly came from an FDA study, which the FDA has apparently now disowned, according to this AOL News report. Rats were used in this study, and the rate at which they developed significant tumor or lesion was measured. The graph illustrated a clear trend that the higher the doses of Vitamin A, the faster the rats developed cancer; this correlation was intact whether they were exposed to high or low levels of UV rays.

Notice that I switched the primary categorical axis to Vitamin A doses rather than high/low UV because the study concerned Vitamin A primarily, and levels of UV secondarily.

Using the Trifecta Checkup, we can see that they have the right question, and the right data but a suboptimal chart. Also, the original chart fails the self-sufficiency test: no point in printing the data on top of the columns when there is a vertical scale.

***

How will this save your life?

Vitamin A is widely added to sunblocks -- not because they have any screening value -- but because they may slow aging of the skin. But the study found that Vitamin A actually partially nullifies the screening ability of sunblocks.

About half of the 500 most popular sunblocks sold in the U.S. contain Vitamin A and only 39 out of the 500 are deemed safe by the Environmental Working Group, which has compiled a database of these products. (There are several other potentially harmful ingredients.)

The FDA denied that such a study existed although the reporter as well as EWG have copies of it. If this study is authentic, the FDA knew about this perhaps ten years ago.

Reference: "Study: Many Sunscreens May Be Accelerating Cancer", Andrew Schneider, AOL News, May 24 2010.

PS. I should explain to my non-U.S. readers that the U.S. is celebrating Memorial Day, the beginning of summer, on Monday so lots of people are going to beaches and other vacations.

Popularity contests and charts

Nick Cox commented on Aleks's post on Andrew's blog on whether chartjunk is better than Tuftian-style charts. Nick made the point that charts should be subjected to popularity contests (my paraphrase). He seemed to say all of statistical science, perhaps all of science, should be fair game for studies of popular opinion.

Interestingly, Aleks's post was about a popularity contest of a different kind: he alerted us to a research paper using Amazon's Mechanical Turk to assess different types of charts.  I haven't had time to review this paper yet but it sounds more promising than the 20-sample experiment. When I read that paper, I will hold my doubts about "crowdsourcing", "prediction markets", and such like.

Back to Nick's comment. But a lot of science have already been turned into popularity contests in recent years - evolution, climate change, causes of autism, bird flu vaccines, etc. have all become politicized, with scientists kicked to the curbside. Based on these results, I wouldn't recommend it.

Nick also pointed out that the "chartjunk is better" paper won the best paper award at the conference where the researchers presented it. I think it said a lot about the lack of statistical expertise on the judging panel more than anything. I stated in my original review that this sort of research is useful and necessary, but it ought to be done with higher standards.

My prior posts on the chartjunk paper:

8 Red Flags ...

More Questions than Participants

A Significant Mystery

Light entertainment: cougars alert

Chris P sent this one along. Rarely has a simple linear equation been rendered so hard to understand. Have some fun while scouting the cougars!  The write up is here.

 

8 Red Flags about the "Useful chartjunk" paper

Reader Darryl guessed correctly that I'd be interested in this paper in which the authors assert that chartjunk of the USA Today type is more "useful" than Tuftian "plain" graphics. (Via Information Aesthetics) I applaud the attempt to put Ed Tufte's theories to the statistical test, and I have written about Bill Cleveland's experiments. However, after reading their paper carefully, I must conclude that the design of the statistical experiment contains so many major flaws that it is hard to take their conclusion seriously.

***

Please see the companion post on the book blog for technical comments. This post focuses on conceptual issues.

***

RED FLAG 1

The sample size consisted of 20 students. Flipping open any elementary statistics textbook, you will find the standard advice to ignore experiments with fewer than 30 observations.

RED FLAG 2

No mention of how participants were "recruited". (Or for that matter, how experimenters were recruited. See RED FLAG 4)

RED FLAG 3

The charts used in the study were mind-numbingly simple. The five examples given in the paper all contained data series with exactly 5 numbers. Many of the charts had little of interest. For example, the chart shown right was titled "Diamonds were a girl's best friend" and showed a rise then fall of diamond prices, huh?

RED FLAG 4

The degree of subjectivity in this experiment is mind-boggling. Instead of a multiple-choice test, a "single experimenter" conducted interviews with participants, asking open-ended questions. These answers were later scored by a "single experimenter". Whether the interviewer and the scorer were the same person was not known. The identity of the experimenter, his/her affiliation, and how he/she was recruited were not mentioned.

RED FLAG 5

The interviewer was allowed, in fact instructed, to "prompt" participants until he/she was satisfied with the final answer. Multiple prompts were allowed. However, only whether any prompting was needed was used in the scoring; the number of prompts used was ignored.

RED FLAG 6

The response of participants were scored against a "checklist" but the checklist was not released with the paper. The guidelines for scoring were described in detail but they appeared to leave room for discretion (e.g. 2 points for "providing most of the relevant information" -- what does it mean by "most"?) The transcripts of the interviews were not published and therefore it's hard to understand the effect of (multiple) prompting.

RED FLAG 7

Some of the questions posed to the participants after they viewed the charts were very silly. Q2 (values) was "What are the displayed categories and values?" Is a good chart defined as one that leads readers to retain displayed values? Not in my book.

RED FLAG 8

The participants were asked to inspect a succession of 14 charts "for as long as they needed", and then answer questions. As a result, the effect being measured is hopefully confounded with (1) memory capacity and (2) how much time the participants chose to spend on reading the charts.

***

Just to underline RED FLAG 4 above, I cite the paragraph where the researchers described their subjective "scoring" system: (By "Holmes", they meant the USA Today style chartjunk.)

To a participant looking at the Holmes 'Monstrous Costs' chart, we would ask question Q3: 'What is the basic trend of the chart?' If the participant responded, 'I don't understand', we would elaborate: 'Tell me whether the chart shows any changes and describe these changes.' The participant might answer 'The teeth get bigger every year.' This answer would score 1 point, as it is not a complete answer (with incorrect information about the period of the data reported) but provides at least some information that the bars increase. The experimenter would then provide additional prompts starting with 'Can you be more specific?' A complete answer scoring four points might be 'The chart shows that campaign expenditures by the house increased by about 50 million dollar every two years, starting in 1972 and ending in 1982.'

Energy- and search-efficient

At our recent NYU talk, Dona Wong presented a graphic that her team produced at Wall Street Journal, which expressed in dollar terms the cost of running each appliance in the household. The graphic was a nice picture of a hypothetical house with each appliance labeled with a dollar amount.

Reader Chris B. complimented the following graphic, found at GE's website, for simplifying the exploratory process to help consumers with energy efficiency information. This is really a kin of what Dona presented.

The graphic is interactive. Mousing on the images leads to pop-ups showing the name of the appliance and some information on energy consumption. The appliances are sorted from using most energy to using least energy. The sorting variable can be energy consumed, or cost. The user can choose daily, monthly or annual cost. Also, the cost is tailored to a specific state.

What can be improved?

• With so many appliances being depicted, it would help to provide intelligent clustering. For example, the appliances could be put into natural groups first before sorting, say small kitchen appliances, large kitchen appliances, living room appliances, etc.
• The distribution of cost/energy consumption tends to be heavily skewed so a few lines can be drawn to indicate clusters of appliances with similar characteristics. For the above configuration, the top 5 (central air conditioner, water heater, electric furnace, freezer, refrigerator) have annual costs of more than $300, the next 4 cost between $100 and $200, etc. In particular, a lot of the smaller appliances at the bottom half of the chart cost only $10 annually or less.
• Get rid of the appliance total - or make it interactive such that the user can select a basket of appliances. It is implausible that one's home has exactly one of each of the depicted appliances.

Prof. Latner asked a great question during the NYU talk. What are the underlying assumptions of the data? Any such chart would have assumed some average level of usage of these appliances, average location within a state, etc. The question is really about whether the profile of an "average user" is useful. If there is a lot of variability around the "average user", then statistical avaerages can be meaningless. This is the topic of Chapter 1 of Numbers Rule Your World.

Light entertainment: Pie chart plated

Thanks Julien D. for the pointer.

(Wheel of Nutrition by Rui Pereira and Hafsteinn Juliusson for HAF, more here).

From fellow readers 2

The Facebook privacy chart that's been circulating widely (thanks to Eronarn): a FAQ on how to read the chart sorely needed.

BBC to beam general election results on to Big Ben (thanks to Julien D.): London readers, did this happen?

Another example of an infographics poster (thanks to Daniel L.), this one concerning the use of cell phones by teenagers. Daniel said:

Check out the pie graphs under the sexting category.  He's showing his percentage in color, but leaving the rest of the pie white.  Awesome!  Surefire way to get the data-ink ratio right where you want it to be.

Daniel also commented on this montrosity. I add:

This is a racetrack graph combined with a (redundant) pie chart. A double crown!

Since the tracks all start at the uppermost point of the circles, the racetrack chart plots cumulative data. This jars with the pie chart which plots each category separately.

Weekend cross-posting

On the book blog this past week, I discussed Steven Strogatz's column "Chances Are" in the New York Times. Professor Strogatz has been hired as a columnist to write about mathematical topics, and his most recent column concerns Bayes' Theorem, and its use in analyzing screening tests. In addition, a reader was curious about the (over-)reaction of  officials in Boston who advised residents to stop drinking tap water for fear of possible contamination which has not been verified.

Untangling Europe's debt web

A number of blogs have hailed this NYT diagram/chart/infographic as "nice". The accompanying article is here.

 

Whether this is nice or not depends on what message you want to convey with this graphic. If it is entanglement, then yes, the graphic reveals the complexity very well. If one wants to understand the debt situation in Europe, then no, this chart doesn't make it clear at all.

From the perspective of someone wanting to dissect the debt web, an enhanced data table is hard to beat.

The first section looks at the interdependency between the five troubled countries, collectively known as PIIGS on Wall Street. The additional debt owed to Britain, Germany and France are shown below. Notice that the original chart does not treat these three countries the same way as PIIGS: we do not know what the values are of the arrows pointing from these three into PIIGS.

Expressed on per-capita terms, Ireland stands out as the worst of the bunch while the citizens of the other four countries are bearing roughly equal amounts of debt per person.

***

I tried to come up with something more "fun", as below:

Here, I opted to use a small multiples chart to split the countries. In so doing, I accepted redundancy in search of clarity. Each amount is plotted twice once as a borrowing (red line) and once as a lending (black arrow).

It is immediately clear why Greece is the most urgent issue.

Perhaps the chart type is not as important as the transformations I did to the data:

1) All amounts shown are "net" amounts between any pair of countries. In the original data, there are two arrows between each pair. For example, Italy owes Ireland $46 million but Ireland owes Italy $18 million; this means Italy owes Ireland $28 million net.

2) All amounts are expressed per capita. Since the populations of these countries vary from 4.5 million (Ireland) to 60 million (Italy), the total debt cannot and should not be compared to each other.

3) Not shown here but I also expressed the net amount lent/borrowed per dollar of GDP. This is another metric that makes sense. The nominal GDP of these countries range from $0.2 - $2 trillion. The PPP GDP has a similar range.

4) One item I did not fix is the currency. Given the fluctuation in exchange rate between the Euro and US$, I think it may be better to express all the numbers in Euros.

A next step would be to include Britain, Germany and France.

Reference: "In and Out of Each Other's European Wallets", New York Times, April 30, 2010.

PS. Reader JF pointed out an inconsistency in the numbers on the chart. I revised the chart to fix this issue. In the current chart, one can read the information as: the average Portuguese owes Spain $5,453, owes Italy $141, while having lent $903 to Greece and $1,561 to Ireland. Each chart can be interpreted from the perspective of the average citizen in that particular country. (For details, see the comments below.)

Election spinner

Bernard L. sent in this chart a while ago, and with the looming British elections, it's a good time to show it, and ask readers how to spin this election. (Via Guardian)

In particular, could someone help me understand the tri-color spinner? Given that the change in seats for the three parties combined should be zero, I don't get how this can fit into a concentric-circles presentation. If you click on the link to the original chart, you can move the black dot around the circle.

In addition, I'm mystified why the constituencies can be depicted on a graph paper, each one the same size as the other. This is not the first time I have seen the U.K. mapped in this way so there must be some reason behind this choice. (For reference, I have never seen the 50 states mapped in this fashion.)

Reference: "Election Map and Swingometer 2010", Guardian (UK), April 5 2010.