Junk Charts

Reading

Here are some things I have been reading while I'm traveling (the posting schedule will be erratic):

Does the vaccine matter?  Shannon Brownlee and Jeanne Lenzer investigates for The Atlantic.  About 100 million Americans get the flu shot each year; what benefit does it confer?  This is an excellent article.

Some provocative quotes:

Flu comes and goes with the seasons, and often it does not kill people directly, but rather contributes to death by making the body more susceptible to secondary infections like pneumonia or bronchitis. For this reason, researchers studying the impact of flu vaccination typically look at deaths from all causes during flu season, and compare the vaccinated and unvaccinated populations.

The estimate of 50 percent mortality reduction is based on “cohort studies,” which compare death rates in large groups, or cohorts, of people who choose to be vaccinated, against death rates in groups who don’t. But people who choose to be vaccinated may differ in many important respects from people who go unvaccinated—and those differences can influence the chance of death during flu season. [Ed: people who can afford the flu shot vs. those who can't; people who are more health-conscious vs. those who aren't, etc.]

“For a vaccine to reduce mortality by 50 percent and up to 90 percent in some studies means it has to prevent deaths not just from influenza, but also from falls, fires, heart disease, strokes, and car accidents. That’s not a vaccine, that’s a miracle.”

In the flu-vaccine world, Jefferson’s call for placebo-controlled studies is considered so radical that even some of his fellow skeptics oppose it. ... “It is considered unethical to do trials in populations that are recommended to have vaccine,” a stance that is shared by everybody from the CDC’s Nancy Cox to Anthony Fauci at the NIH. They feel strongly that vaccine has been shown to be effective and that a sham vaccine would put test subjects at unnecessary risk of getting a serious case of the flu.

Another pie chart, Fox News (via FlowingData, thanks to reader Katherine M for the pointer; also via Wonkette and reader omegatron).

Yet another pie chart, Business Insider.  Not on the same scale as the one above but still why?

Clean Water Act Violations, New York Times. Can we trust tap water?  As usual, a set of small bars would work better than concentric circles.

How does your state compare to California? (via Pew and Mother Jones) This is a nice illustration that often it is better to plot data derived from the raw data, as opposed to the raw data itself.  Since the designer decided to hide the information, let's figure out what were the cut-off points for the color categories.  If the size of each category is not the same, the designer needs to explain the scale.  Also, the two shades of light blue are hard to tell apart.  But all in all, a good effort here.

The real climategate

Climategate is all the rage at the moment. What interests me about this episode is not the integrity of certain scientists, or science in general, nor the culture of academia, and certainly not the evidence of climate change. For me, the real climategate is the woeful state of statistical education.  Let me explain.

Here is the infamous email: (via Nathan Silver, with my highlights)

From: Phil Jones
To: ray bradley ,mann@[snipped], mhughes@
[snipped]
Subject: Diagram for WMO Statement
Date: Tue, 16 Nov 1999 13:31:15 +0000
Cc: k.briffa@[snipped],t.osborn@[snipped]
Dear Ray, Mike and Malcolm,

Once Tim’s got a diagram here we’ll send that either later
today or first thing tomorrow. I’ve just completed Mike’s Nature
trick of adding in the real temps to each series for the last 20
years (ie from 1981 onwards) amd [sic] from1961 for Keith’s to
hide the decline. Mike’s series got the annual land and marine
values while the other two got April-Sept for NH land N of 20N.
The latter two are real for 1999, while the estimate for 1999
for NH combined is +0.44C wrt 61-90. The Global estimate for
1999 with data through Oct is +0.35C cf. 0.57 for 1998.

Thanks for the comments, Ray.

Cheers, Phil

What concerns me is Phil Jones' describing what he did as a "trick" to "hide the decline".  He apparently thought that he was doing something shameful. But when is it shameful to extend the plot of a time series so as to display the long-term trend, and not be misled for short-term fluctuations? This is providing statistical context to the data being examined. Lots of people are condemning this as a willful act to mislead the public but if they have some statistical literacy, they will understand that finding the appropriate time scale to look at the data is one of the most important tasks of analyzing time series data. It's a problem when even prominent scientists do not comprehend why they should be doing this.

I have always wondered why in climatology as well as in economics, we rarely see decomposed time-series plots (at least not in the public's eye). 

On the right, I found on-line a plot of a decomposition of beer sales that separates out seasonality, trend and other parts of a time series. The original data is shown up top. In practice, newspapers and blogs give us such plots all the time when they should show us the third plot down (the trend with the seasonal factor removed), unless the story is about seasonality.

Note to self: should include basic time-series decomposition in the intro stats syllabus; much too important a topic to leave to a second course.

 

Following one's nose 2

This is the second post on the immigration paradox study, first discussed on the Gelman blog.  My prior post on the graphing aspect is here; this post focuses on the statistical aspects. I am working backwards on Andrew's discussion points.

Which difference is most interesting?

5. Agree with Andrew; they should publish similar analyses on other minority groups as soon as possible.  One thing that strikes me when looking at the interaction plot is that the U.S. born non-Latino whites have a much higher incidence of mental illness.  The difference between different subgroups of Latinos paled in comparison to the difference between non-Latinos and the Latinos.  This latter difference is particularly acute among the U.S. born than the immigrants. The importance of the Latino analysis hinges upon whether the "paradox" is also found among other minority groups.

(Chris P also pointed this out in his comment on the previous post.)

Disaggregation, Practical Significance, and the Meaning of Not Significant

2. Andrew is also right in expressing moderate skepticism about this sort of disaggregation exercise.  He connects this to the subtle statistical point that "the difference between significant and not significant is not significant."  A related but less obtruse issue is that as one disaggregates any data, the chance of seeing variations that stray from the average gets higher and higher.  This is because the sample size is decreasing, and so the statistical estimates are less reliable.

(To give a flavor of the scale, there were a total of 2500 Latinos in the sample, with 500 Puerto Rican Latinos. The analysis drilled down to the level of different types of mental disorders, subgroups of Latinos, and also adjusted for demographics.  The details of the demographic adjustment are not available but in any case, one should be concerned about whether there were sufficient numbers of say, male immigrant Puerto Rican Latinos age 18-25 with income < $10,000 living in a rental apartment, for such an elaborate exercise.)

Expanding on this point further, one observes that the measured gap between U.S. born and immigrant Puerto Rican Latinos was about 5%.  But this 5% is probably of considerable practical significance since the base rate of incidence is about 30% (I say probably since I am not an expert in mental illness).  The current statistical analysis judged this to be insignificant -- if the sample size were larger, this difference could conceivably be statistically significant, and also practically significant.

But, doesn't the significance test deal with the small sample size problem?  Yes, if the authors merely described the Puerto Rico result as inconclusive.  Here, as is done very commonly, insignificance is equated to "no difference": they said

No differences were found in lifetime prevalence rates between migrant and U.S.-born Puerto Rican subjects.

In reality, a difference of 5% was found in the sample that was analyzed.  The statistical procedure found that this difference could have been a result of chance -- notice "could", not "must".  If the measured difference was 0.5% on 30%, then I might be willing to accept a finding of "no difference"; when it was 5% on 30%, I would like to see a larger sample analyzed.

The Meaning of Paradox

1. Andrew was perplexed by why the phenomenon is known as a "paradox". I had the same issue until I read the paper. The authors were a bit sloppy in the abstract. In the paper itself, they explained that the conventional wisdom has it that immigrants should be more likely to have mental illness because of the stress from the immigration process, and yet the statistics showed the exact opposite. That is the paradox.

Publication Bias

I was a little shocked to see the data tables that gave all the estimates of the various effects at the various subgroup levels: shocked because the authors were allowed (or asked) to include only the p-values that were below some unspecified level (which I surmised is 10% although a 5% significance level is used to judge significance as per convention). This is publication bias within publication bias. P-values that are not significant still provide valuable information and should not be omitted. They did provide confidence intervals but for each subgroup separately, rather than for the difference -- and as they noted, such intervals by themselves are inconclusive when they overlap moderately.

Following one's nose 1

Andrew Gelman has a great post about a so-called Immigrant paradox here, which should be interesting to our readers too.

He posed a set of sharp questions.  My read, in reverse order:

6. The graph is pretty effective, I agree.  This is known as an "interaction plot".  The message the authors were trying to send was that the gap between immigrants and U.S. born in terms of prevalence of mental illness is not constant across sub-groups of Latinos.  For example, the gap for Mexicans (light blue) is larger than the gap for Puerto Ricans (pink).  Thus, the authors concluded that one should be careful about speaking of an aggregate (average) gap.

The graph lays this out clearly.  The steeper the line, the bigger the gap between the  immigrants and non-immigrants.

When Andrew showed this, I knew for sure someone will cry foul that a line is drawn between unrelated, discrete things.  Indeed, the very first commenter weighed in with this complaint.  In fact, whenever I show such charts to non-statisticians, a lot of people have this reaction.

So I'll take this as another chance to convince you to release interaction plots from jail.

Typically, a dissenter will offer up a dot plot as an alternative.  So let's look at the same chart without the lines.  Since the reader is supposed to figure out how the gap between U.S. born and immigrant groups across different subgroups of Latinos, the proverbial nose is tracing a line from a left dot to a right dot.  Thus, to follow one's nose is to mentally draw the lines I just removed.  The chart designer has done us a favor by making the lines explicit.

In addition, as Andrew pointed out, it is always better to try to get rid of the legend and put the line labels directly onto the chart.

One shortcoming of the interaction plot is that it does not disclose the relative importance of the different lines, which correspond to the relative proportions of people in these subgroups.  Without this information, the reader will likely assume the lines have equal weight.  This assumption, as I will explain in a future post, may be a problem.

This post dealt with the graphical aspect.  I will have more to say about Andrew's other points on the statistics in a future post.

Of placebos and straw men

Note: This post is purely on statistics, and is long as I try to discuss somewhat technical issues.

(Via Social Sciences Statistics blog.)

This article in Wired (Aug 24,2009) is a must-read.  It presents current research on the "placebo effect", that is, the observation that some patients show improvement if they believe they are being treated (say, with pills) even though they have received "straw men" (say, sugar pills) that have no therapeutic value.

The article is a great piece, and a terrible piece.  It fascinated and frustrated me in equal measure.  Steve Silberman did a good job bringing up an important topic in a very accessible way.  However, I find the core arguments confused.

Let's first review the setting: in order to prove that a drug can treat a disease, pharmas are required by law to conduct "double-blind placebo-controlled randomized clinical trials".  Steve did a great job defining these: "Volunteers would be assigned randomly to receive either medicine or a sugar pill, and neither doctor nor patient would know the difference until the trial was over."  Those receiving real medicine is known as the treatment group, and those receiving sugar pills is the placebo control group.  Comparing the two groups at the end of the trial allows us to establish the effect of the drug (net of the effect of believing that one is being treated).

(I have run a lot of randomized controlled tests in a business setting and so have experience interpreting such data.  I have not, however, worked in the pharma setting so if you see something awry, please comment.)

Two key themes run through the article:

1) An increasing number of promising drugs are failing to prove their effectiveness.  Pharmas suspect that this is because too many patients in the placebo control group are improving without getting the "real thing".  They have secretly combined forces to investigate this phenomenon.  The purpose of such research is "to determine which variables are responsible for the apparent rise in the placebo effect."

2) The placebo effect meant that patients could get better without getting expensive medicine.   Therefore, studying this may help improve health care while lowering cost.

Theme #1 is misguided and silly, and of little value to patients.  Theme #2 is worthwhile, even overdue, and of great value to patients.  What frustrated me was that by putting these two together, not sufficiently delineating them, Steve allowed Theme #1 to borrow legitimacy from Theme #2.

To understand the folly of Theme #1, consider the following stylized example:

Effect on treatment group = Effect of the drug + effect of belief in being treated

Effect on placebo group = Effect of belief in being treated

Thus, the difference between the two groups = effect of the drug, since the effect of belief in being treated affects both groups of patients.

Say, if the treatment group came in at 15, and the placebo at 13, we say the effect of the drug = 15 - 13 = 2.

A drug fails because the effect of the drug is not high enough above the placebo effect.  If you are the pharmas cited in this article, you describe this result as the placebo effect is "too high".  Every time we see "placebo effect is too high", substitute "the effect of the drug is too low". 

Consider a test of whether a fertilizer makes your plant grow taller.  If the fertilized plant is the same height as the unfertilized plant, you would say the fertilizer didn't work.  Who would conclude that the unfertilized plant is "unexpectedly tall"?  That is what the pharmas are saying, and that is what they are supposedly studying as Theme #1.  They want to know why the plant that grew on unfertilized soil was "so tall", as opposed to why the fertilizer was impotent.  (One should of course check that the soil was indeed unfertilized as advertised.)

Take the above example where the effect on the placebo group was 13.  Say, it "unexpectedly" increased by 10 units.  Since the effect of the treatment group = effect of drug + effect of believing that one is treated, the effect of the treatment group also would go up by 10.  Because both the treatment group and the control group believe they are being treated, any increase in the placebo effect would affect both groups equally, and leave the difference the same.  This is why in randomized controlled tests, we focus on the difference in the metrics and don't worry about the individual levels.  This is elementary stuff.

One of their signature findings is that some cultures may produce people who tend to show high placebo effects.  The unspoken conclusion that we are supposed to draw is that if these trials were conducted closer to home, the drug would have been passed rather than failed.  I have already explained why this is wrong as described... the higher placebo effect lifts the metrics on both the treatment and the control groups, leaving the difference the same.

There is one way in which cultural difference can affect trial results.  This is if the effect of the drug is not common to all cultures; in other words, the drug is effective for Americans (say) but not so for Koreans (say).  Technically, we say there is a significant interaction effect between the treatment and the cultural upbringing.  Then, it would be wrong to run the trial in Korea and then generalize the finding to the U.S.  Note that I am talking about the effect of the drug, not the effect of believing one is being treated (which is always netted out).  To investigate this, one just needs to repeat the same trial in America; one does not need to examine why the placebo effect is "too high".

I have sympathy for a different explanation, advanced for psychiatric drugs.  "Many experts are starting to wonder if what drug companies now call depression is even the same disease that the HAM-D [traditional criterion] was designed to diagnose".  The idea is that as more and more people are being diagnosed as needing treatment, the average effect of the drug relative to placebo group gets smaller and smaller.  This is absolutely possible: the marginal people who are getting diagnosed are those with lighter problems, and thus those who derive less value from the drug, in other words, could more easily get better via placebo.  This is also elementary: in the business world, it is well known that if you throw discounts at loyal customers who don't need the extra incentive, all you are doing is increasing your cost without changing your sales.

No matter how the pharmas try, the placebo effect affects both groups and will always cancel out.  Steve even recognizes this: "Beecher [who discovered the placebo effect] demonstrated that trial volunteers who got real medication were *also subject to placebo effects*."  It is too bad he didn't emphasize this point.

On the other hand, Theme #2 is great science.  We need to understand if we can harness the placebo effect.  This has the potential of improving health care while at the same time reducing its cost.  Of course, this is not so useful for pharmas, who need to sell more drugs.

I think it is not an accident that Theme #2 research, as cited by Steve, are done in academia while Theme #1 research is done by an impressive roster of pharmas, with the help of NIH.

The article also tells us some quite startling facts:

- if they tell us, they have to kill us: "in typically secretive industry fashion, the existence of the project [Theme #1] itself is being kept under wraps."  Why?
- "NIH staffers are willing to talk about it [Theme #1] only anonymously, concerned about offending the companies paying for it."
- Eli Lilly has a database of published and unpublished trials, "including those that the company had kept secret because of high placebo response".  Substitute: low effect of the drug.  This is the publication bias problem.
- Italian doctor Benedetti studies "the potential of using Pavlovian conditioning to give athletes a competitive edge undetectable by anti-doping authorities".  This means "a player would receive doses of a performance-enhancing drug for weeks and then a jolt of placebo just before competition."  I hope he is on the side of the catchers not the cheaters.
- Learnt the term "nocebo" effect, which is when patients develop negative side effects because they were anticipating them

Again, highly recommended reading even though I don't agree with some of the material.  Should have focused on Theme #2 and talk to people outside pharma about Theme #1.

Serving donuts

David Leonhardt's article on the graduation rates of public universities caught my attention for both graphical and statistical reasons.

David gave a partial review of a new book "Crossing The Finish Line", focusing on their conclusion that public universities must improve their 4-year graduation rates in order for education in the U.S. to achieve progress.  This conclusion was arrived at through statistical analysis of detailed longitudinal data (collected since 1999).

This chart is used to illustrate this conclusion.  We will come to the graphical offering later but first I want to fill in some details omitted from David's article by walking through how a statistician would look at this matter, what it means by "controlling for" something.

The question at hand is whether public universities, especially less selective ones, have "caused" students to lag behind in graduation rate.  A first-order analysis would immediately find that the overall graduation rate at less selective public universities to be lower, about 20% lower, than at more selective public universities.  

A doubter appears, and suggests that less selective schools are saddled with lower-ability students, and that would be the "cause" of lower graduation rates, as opposed to anything the schools actually do to students.  Not so fast, the statistician now disaggregates the data and look at the graduation rates within subgroups of students with comparable ability (in this instance, the researchers used GPA and SAT scores as indicators of ability).  This is known as "controlling for the ability level".  The data now shows that at every ability level, the same gap of about 20% exists: about 20% fewer students graduate at the less selective colleges than at the more selective ones.  This eliminates the mix of abilities as a viable "cause" of lower graduation rates.

The researchers now conclude that conditions of the schools (I think they blame the administrators) "caused" the lower graduation rates.  Note, however, that this does not preclude factors other than mix of abilities and school conditions from being the real "cause" of lower graduation rates.  But as far as this analysis goes, it sounds pretty convincing to me.

That is, if I ignore the fact that graduation rates are really artifacts of how much the administrators want to graduate students.  As the book review article pointed out, at the less selective colleges, they may want to reduce graduation rates in order to save money since juniors and seniors are more expensive to support due to smaller class sizes and so on.  On the other hand, the most selective colleges have an incentive to maintain a near-perfect graduation rates since the US News and other organizations typically use this metric in their rankings -- if you were the administrator, what would you do?  (You didn't hear it from here.)

Back to the chart, or shall we say the delivery of 16 donuts?

First, it fails the self-sufficiency principle.  If we remove the graphical bits, nothing much is lost from the chart.  Both are equally impenetrable.

A far better alternative is shown below, using a type of profile chart.

Finally, I must mention that in this particular case, there is no need to draw all four lines.  Since the finding of a 20% gap essentially holds for all subgroups, no information is lost by collapsing the subgroups and reporting the average line instead (with a note explaining that the same effect affected every subgroup).  

By the way, that is the difference between the statistical grapher - who is always looking to simplify the data - and the information grapher - who is aiming for fidelity. 

Reference: "Colleges are lagging in graduation rates", New York Times, Sept 9, 2009; "Book review: (Not) Crossing the Finish Line", Inside Higher Education, Sept 9 2009.

Degrees of likeness 2

We left off the other day with an interactive graphic with the ability to peer into subgroups.  This feature assumes implicitly that the overall average obscures differences within subgroups.  What statisticians do with this type of data is to compare the subgroups, and identify the factors that make someone different from the average.

For example, there is a clear distinction between the employed and the unemployed in how they spend the day (not surprising).

This happens to be what NYT printed in the paper edition that day.  (Note, though, that the graphic loses quite a bit without the interactivity.)

On the other hand, there appears to be little differentiation between men and women.

Nor is there much difference between blacks and whites.

One factor that matters is age.  Older people are not exactly like the young.  A lot of these factors (for example, age and employment status) are correlated, by the way.

I showed all these in order to talk about the statistical concept of "aggregation".  We noted that the distribution of time use of the employed is different from that of the unemployed.  Thus, we cannot use the "average" distribution to describe both groups, and so we show the data in disaggregated form.  Similarly for time use and age.  

But there is not much gain in disaggregating race and gender: the "average" is representative of the subgroups for these two factors.  This is one distinction I see between information graphics and statistical graphics: the former typically shows all possible subgroups while in the latter, the designer zooms in on the factors that matter.

Sociology of numbers

I picked up a copy of AM New York (free newspaper given out in the subway stations) yesterday morning.  So we are told: tanning beds are killers.  See here for example.

How bad?

The article began:

Using a tanning bed regularly is as deadly as taking arsenic, a shocking study to be published today says.
The report found that the risk of skin cancer jumped by 75 percent when tanners started regularly using the beds before the age of 30.

A few paragraphs later, an "occasional tanning bed user" made a confession:

"I always knew it wasn't good for your skin.  Definitely 75 percent risk of cancer is not that encouraging and not worth the risk."

What?

What started out as a relative increase in risk of 75 percent (those using tanning beds compared to those who don't) ended up as a 3 in 4 chance of getting cancer!

This reminds me of Joel Best's books in which he explored how data gets "adulterated" as it moves through society.  I find his perspective fascinating and his books well worth a read.  They are not your typical statistics book for sure.






 

A shocking failure to communicate

So said a reader, Stephen B., of the following graphic (note: pdf) in the London Times concerning Andy Murray's recent tennis triumphs.

How can we disagree?  Shocking?  Yes.  Failure?  Definitely.  Failing to communicate?  No doubt.

Let's first start with the five tennis balls at the bottom.  It fails the self-sufficiency test.  It makes no difference whether the balls (bubbles) are the same size, or different sizes.  Readers will look at the data and ignore the bubbles.

Amazingly, the caption said that "Murray has one of the best returns of serve in the game."  And yet, the graphic showed the five players who were better than Murray, and nobody worse!  For those unfamiliar with tennis statistics, it does not provide any helpful statistics like averages, medians, etc. to help us understand the data.

But that is only the beginning.

Take a look at these two donuts.

(The color scheme from light to dark: first, second, third, fourth round of tournament)

So we're told: the 75% of first-serve points won in the fourth round was 25.6% of the sum of the percentages of first-serve points won from first to fourth rounds (75%+70%+71%+76%).  What does this mean?  Why should we care?

The challenge with these two statistics is that they are correlated and have to be interpreted together.  If a first-serve is won, then there would be no second serve, etc.  Here's one attempt at it, using statistics from the Soderling-Federer match.  It's clear that Federer was better on both serves.

Reference: "Murray's march to the last eight", London Times.

 

Round up

Here are some interesting reading from other places:

Tag clouds have caught on since we approved them a while ago.  One interesting use was at the Life Vicarious blog.  They use it to compare the inclinations of three New York-based restaurant reviewers.  What they should have done is to remove irrelevant words like  "one", "also", "many", "make"/"made", etc.  In statistics, this is called removing "noise" which helps bring out the "signal".

Andrew Gelman discussed the NYT article that reported the finding of unexpected male bias in the children of Asian American families.  He can be counted on to make useful comments on any accompanying graphics.  He rightly pointed out that this is one example of not starting at zero: the relevant baseline is 100 since the metric is essentially the over-age of males relative to females.  I also agree that a line chart with a longer time series plotting percentages rather than over-age would work better.

The racetrack chart made an appearance at Flowing Data.  This one is even more busy and just as impossible to decipher.