Big Data, Plainly Spoken (aka Numbers Rule Your World)

Popularizing tiny effects

I found my way to Mark Liberman's post at Language Log by way of a comment by Kyle on Andrew Gelman's post about Dubner's response to our Freakonomics article. I've always enjoyed Mark's posts and this one is no exception. His first bullet point speaks to one of my chief worries about Freakonomics-style analyses.

For background, Mark raised some doubts about recent academic work that supposedly shows that the left-right asymmetry in the QWERTY keyboard design affects our perception of words. The researchers concluded: "Words with more right-side letters were rated to be more positive, on average, than words with more left-side letters. We call this relationship the QWERTY effect."

***

Mark did some quick analyses which failed to replicate the finding. But his first point has nothing to do with replication. It is valid even if the original research has been done impeccably. Here are the words you must read:

1. The QWERTY effect's size. As far as I'm concerned, and as far as the general public is concerned, the size (and therefore the practical importance) of the QWERTY effect (if it exists) is the key question. This is not an entirely subjective matter — we can ask, as I did, what proportion of the variance in human judgments of the emotional valence of words is explained by the "right side advantage". The answer is "very little", or more precisely, around a tenth of a percent at best (at least in the modeling that I've done).

I focused on the effect-size question because the press release said the following (and the popular press took the hint):

Should parents stick to the positive side of their keyboards when picking baby names – Molly instead of Sara? Jimmy instead of Fred? According to the authors, “People responsible for naming new products, brands, and companies might do well to consider the potential advantages of consulting their keyboards and choosing the 'right' name."

So C&J may not be interested in my subjective evaluation of the effect size, but they promoted their own subjective evaluation by suggesting that the effect is important enough to matter to people choosing names. I felt (and feel) that this represents a serious exaggeration of the strength of the effect; and it seemed (and seems) appropriate to me to say so publicly.

***

Mark's complaint is similar to my response to several results championed by the Freakonomics team, including the "surname effect" as it relates to winners of the Nobel Economics Prize, and the "birthday effect" as it relates to sports leagues.

The common ingredients of such analyses are: published, peer-reviewed scholarly work that identifies an interesting effect meeting the standard of statistical significance, followed by the media's amplification and popularization of results that (a) ignore practical significance; and (b) apply a causal interpretation, possibly unknowingly.

(a) Practical significance

Statistical significance is designed to measure one thing only: how likely would we observe the effect being investigated assuming the effect does not exist (i.e., what's the chance of a false positive)? We need this concept because many observed effects (especially small effects) can happen by chance and therefore should not be attributed to the factor being studied.

Statistical significance is necessary but not sufficient for practical value. In other words, a practically meaningful effect must be statistically significant but there are many statistically significant effects that have little to no practical value.

Statistical significance will get one published in a peer-reviewed journal but it's not the job of a journal editor to discern practical value. Aside from statistical significance, the editor's other standard is contribution to the scholarship (i.e. novelty), which tells us nothing about practical value either. Thus, the "peer review" standard cannot defuse this issue.

As Mark pointed out, the practical importance of an effect is given by its effect size. For the QWERTY effect, it is one tenth of one percent at best. If you list out all of the factors that may affect "human judgments of the emotional valence of words", you will have a long list. If you now rank the factors in terms of their effect sizes, where would the QWERTY effect fall? Mark is saying the effect is one tenth of a percent at best and he's implying there are other factors more important than QWERTY at play here. Why focus everyone's attention on QWERTY when other factors may be even more important?

This is clearer in the Freakonomics examples. In the "birthday effect" research, it is acknowledged that genetic factors like gender and whether one's dad is a professional athlete are much more powerful than one's birthday. While the birthday effect is statistically significant, it is relatively small so why focus everyone's attention on that rather than on more powerful factors? As for Nobel economics prizes, one can name a variety of factors, such as educational background, creativity, intelligence, and influence that are more powerful than the first letter of one's surname.

***

Are Mark and I fussing over little details? No.

Go back to the 5% statistical significance level. By accepting this convention, we accept that 1 in 20 results are wrong. And no one knows which one is the false positive. If you have to take a guess, you would be more suspicious of the results in which the effect size is small, or when the the significance is barely achieved. Because of the 5% convention, you wouldn't be surprised that lots of published results just make the 5% mark. In addition, Andrew Gelman (link) tells us why the conventional acceptance criterion ensures that if the true effect is small, the estimated effect size will be exaggerated (and thus inaccurate).

For all these reasons, statisticians are reluctant to tout small effects that just achieve 5% statistical significance, and especially in observational data.

In some of these examples, there is no way to arrange randomized experiments to verify the reported effects. We can't run experiments on birthdays and sports success, for example. But when such experiments are possible, the results have often been ugly. To see what I'm talking about, I highly recommend reading this blog post by Gary Taubes, a long-time science reporter (thanks to John for his comment on my post on the red meat finding).

Taubes's is a really long article. Here are the two key quotes:

this meat-eating association with disease is a tiny association. Tiny. It’s not the 20-fold increased risk of lung cancer that pack-a-day smokers have compared to non-smokers. It’s a 0.2-fold increased risk — 1/100th the size...very few epidemiologists would ever take seriously an association smaller than a 3- or 4-fold increase in risk. These Harvard people [i.e., the researchers behind the red meat study] are discussing, and getting an extraordinary amount of media attention, over a 0.2-fold increased risk.

... every time in the past that these researchers had claimed that an association observed in their observational trials was a causal relationship, and that causal relationship had then been tested in experiment, the experiment had failed to confirm the causal interpretation — i.e., the folks from Harvard got it wrong. Not most times, but every time. No exception. Their batting average circa 2007, at least, was .000.

All those peer-reviewed, statistically-significant results amounted to a huge pile of misinformation... but a lot of press, which by the way does not have the habit of retracting erroneous reporting of this type.

***

To summarize: a lot of published effects are tiny, which means they have no practical value even if they have entertainment value; moreover, when reported effects are tiny, there is a good chance that they are false positives so trumpeting them for titillation can set you up for later embarrassment.

I'll address the other aspect of this practice, that of unwarranted causal interpretations, in a later post.

 

We sometimes need a muzzle

Let the data speak. That's advice frequently given. But it is bad advice, especially in this Big Data era.

While doing research, reader Winnie H. found this webpage (link), apparently generated by some automated software that explores every variable found on a dataset about people living in the 32257 zip code. How else do we explain the data table shown on the right?

 

The research question seemed to be how your first name can kill you. The table extracts the most common first names among the residents who died during the year; for each first name, it computes the average age at death of all those who died. (There is another table that does the same thing for last names.)

***

I did a little exploration. First, the data is split by gender which helps us see the true variability. I then estimated the average age and the standard deviation of the age--but this can be done only for the 585 people who died and had one of those 10 first names. One can make the not-unreasonable assumption that the true average and standard deviation would not be far off.

The standard deviation is about 1.3-1.5 years. Using a conventional 95% interval estimate, we would expect the length of the interval to be about 5-6 years. The range of the average ages listed in the original table is 3.9 for females and 4.8 for males, so in both cases, the variability observed is basically noise.

***

It would be more scary if the observed variability is larger than noise. We'd likely see headlines screaming "Your first name reduces your life expectancy!!", or "Margaret, it's time to become Elizabeth!". And why not "James, if you want to live longer, become Elizabeth now!"

In situations like this, the statistician has to make sure the effect is not an artifact of her methodology. This data certainly do not speak for themselves. This data should probably be muzzled.

What can we learn from Freakonomics?

Stephen Dubner is very upset with the feature article by Andrew Gelman and myself about the Freakonomics franchise in American Scientist. (The original article is here, and I blogged about it here.) He wrote a long, angry response at the Freakonomics blog (link), in which he said

"their comprehension of democratic society is perhaps even worse than their comprehension of what we have written"; and

"this portion of the essay is so error-ridden and deprived of logic that it's hard to decipher"; and

"with rhetoric as contorted and sketchy as that, a reader has every right to distrust whatever logic you're about to unload".

Andrew and I have been branded enemies of Freakonomics even though we have both praised the franchise often. If Dubner were present at some of the talks I gave in the past year, he would have heard first-hand how I consider their work (together with Malcolm Gladwell's) to have fundamentally changed the landscape of pop-science publishing.

***

Andrew's measured response to Dubner's blog post is here. I highly encourage you to read it as it captures many of my thoughts as well.

I'd just add that we focused on the reading experience. Because the Freakonomics materials are regarded as "easy reading", the typical reader trusts the authors' take on various topics without examining endnotes or following up on research papers. If she is reading a popularization of science, she shouldn't need to.

Of course, one's reading experience is subjective, like many of the social-science questions addressed in Freakonomics. We voiced our opinions. We are aware that it's the voice of a silent segment of readers. I reported how I interacted with these materials as a practitioner of statistics; I don't expect everyone to agree but I don't see how that experience can be falsified.

As Andrew said, "We said our piece, Dubner's had his reply, and the reader can work it out from there."

 

Three complaints about medical journalism

It's very frustrating to read journalism about health and medicine. They repeat bad advice from papers that are published with narrowly-defined research objectives.

Here's a recent example about red meat (link; the study itself). The headline gives us a prescription: "Study gives more reasons for passing on red meat". The key paragraph tells us:

Each extra serving was also tied to a 16 percent higher chance of dying from cardiovascular disease in particular, and a 10 percent chance of dying from cancer -- even after taking into account other aspects of health and lifestyle that could influence the chance of dying, such as weight, smoking, the rest of their diet and socioeconomic factors.

To summarize: Food X is linked to disease Y. Therefore, eat less (or don't eat) X.

***

Complaint #1: there is no such thing as "good food" or "bad food"

Every food has harms and benefits. These studies are almost always looking for "causes" of disease. Rarely are studies reported to prove that an extra ounce of red meat, etc. will enhance our intelligence by 10% (I'm making this up.) Red meat surely has some benefits. So we must balance both the benefits and the harms in order to decide how much to eat.

 

Complaint #2: there are multiple causes for cancer/heart disease/etc.

There is an unspoken implication that if one eats less of food X, one will not get disease Y. This is wrong on several levels. First, a 10 or 12 or 16 percent reduction in mortality rate may sound impressive but what is the base rate? The average age of 120,000 people being analyzed was about 45-50. In following them for 22-28 years, about 24,000 died of any cause, that's 20 percent. A 10 percent improvement of this probability is 2 percent. Is this practically meaningful? Only you can tell.

Another problem is that cancer/heart disease etc. has multiple causes. Most people are exposed to multiple risks that could lead to these diseases. Reducing one type of risk most likely will not stop you from getting disease Y.

 

Complaint #3: you can only control for things you know about

One typical defense of such analysis is the use of control variables. In the above quote, we read something like "after taking into account other aspects of health and lifestyle that could influence the chance of dying, such as weight, smoking, the rest of their diet and socioeconomic factors."

Impressive, right? The problem is for most diseases like cancer, we don't yet, and may never, know the entire causal structure of how one gets it. We can't control for variables that we don't know are related to the disease, or ones we mistakenly thought to be unrelated, or ones that are not being measured.

Secondly, these regressions almost never take into account the fact that all those control variables are not independent; for example, poorer people are more likely to have less healthy diets, someone who eats more meat are more likely to eat less vegetables. Say, someone reduces meat intake. It is likely this person would be eating more vegetables and/or fruits. The single coefficient for meat in the regression would underestimate the impact.

***

I cannot end without pointing out that it is misleading for the reporter to say "Each extra serving was also tied to a 16 percent higher chance of dying from cardiovascular disease". If this was so, with 5 extra servings, we'd all be dead. It should really say "Have one extra serving per day every day you're alive", which was what the researchers were measuring.

Reading guides for instructors, book clubs, etc.

I want to point your attention to recent coverage of Numbers Rule Your World in Chance News. If you are using my book for class or a book club, their writeups act well as reading guides for several chapters.

Notes on Chapter 1 (ramp meters)

Notes on Chapter 3 (hurricanes, test bias) and Chapter 4 (drug testing)

***

Chance News has been an extremely valuable resource for statistics teachers for years. They pick up news stories and interesting items, ask intriguing questions but don't offer answer keys. Most of the questions are open-ended, which is as most statistics questions are. We owe much gratitude to the volunteers who have kept the site alive all these years.