Big Data, Plainly Spoken (aka Numbers Rule Your World)

Gelman speed read

For those who have found it tough to keep up with Andrew Gelman's prolificacy, here are some brief summaries of several recent posts:

On people obsessed with proving the statistical significance of tiny effects: "they are trying to use a bathroom scale to weigh a feather—and the feather is resting loosely in the pouch of a kangaroo that is vigorously jumping up and down." (link)

[I left a comment. In Big Data, we have thousands, no millions, of kangaroos jumping out of sync, but still one feather.]

On people testing a zillion things hoping to land on the one that "works": "I suggest you should fit a hierarchical model including all comparisons and then there will be no need for such a corrections." (link)

[This is something Andrew has been advocating for a while. The idea is that such models have in some sense a built-in correction for the multiple comparisons problem. Unfortunately, some researchers are wrongly interpreting Gelman. I recently read a report that cites Gelman's paper as evidence that "multiple comparisons" is not a real problem, and then proceed to fit dozens of regressions without any mechanism to control for multiple comparisons!]

On when to throw out all your data, the lot of it: "Sure, he could do all this without ever seeing data at all—indeed, the data are, in reality, so noisy as to have have no bearing on his theorizing—but the theories could still be valuable." (link)

On why some professors should be fired (link)

 

More on the AA paper

This is a supplement to the previous post about a new research paper on the effect of Alcoholics Anonymous, and an NY Times exposition that I commented on. A misreading of that article led me to complain about per-protocol analysis, which wasn't the methodology behind the Humphrey et. al. research. I will explain their methodology in this post (known as instrumental variables analysis).

***

In the last post, I showed this hypothetical situation, involving patients who "cross over" (disobey treatment assignment) in a randomized experiment.

In the paper, actual treatment is measured by the change in frequency of attending AA meetings (relative to baseline).

Because initial treatment assignment (rows) is random, one expects that equal proportions of people would have moved out of state, got married, got divorced, etc. Similarly, one expectas equal proportions of people would have increased AA attendance. But in the table above, 90% of people in the treatment arm upped attendance while only 60% of those assigned to no treatment increased attendance. (The researchers use a continuous scale of frequency rather than proportion but the concept is the same.)

Of course, the random assignment to treatment itself is a cause of higher relative attendance. People are told to go to AA meetings. But there are other reasons for increased attendance, such as self-motivation leading those in the no-treatment arm to cross over.

In ITT analysis, you ignore the actual attendance, and analyze how treatment assignment affects the amount of drinking.

Alternatively, one can run a regression of frequency of AA meetings on amount of drinking (relative to baseline). This will yield a result such as "the more meetings someone attends, the less they drink". The problem with this analysis is that while the initial assignment is random, the actual attendance is tainted by selection bias.

***

Instead of using the actual frequency of AA meetings as a regressor, the instrumental variables (IV) analysis uses a predicted frequency of AA meetings. The prediction is itself a regression of treatment assignment and demographic variables on the actual frequency of AA meetings. In other words, we only care about the proportion of the variability in AA attendance that can be explained by the random assignment (controlling for the demographic variables). The remaining variability (due to self-motivation, etc.) are left on the table.

This is the "correction" that Frakt inferred in the New York Times article. I think Frakt is correct that the conclusion can be applied only to those who obey the protocol but I don't think the researchers drop all non-compliers from the dataset.

Also, Humphrey, et. al. seem to be at odds with the author of The Atlantic article, as they say "The long-established positive association between AA involvement and better outcomes was therefore consistent with, but did not prove, causation."

NYT likes new AA study. Why I am not convinced.

[After communicating with Frakt, Humphrey and Dean Eckles, I realize that I was confused about Frakt's description of the Humphrey paper, which does not perform PP analysis. So when reading this post, consider it a discussion of ITT versus PP analysis. I will post about Humphrey's methodology separately.]

The New York Times plugged a study of the effectiveness of Alcoholics Anonymous (AA) (link). The author (Austin Frakt) used this occasion to advocate "per-protocol" (PP) analysis over "intent-to-treat" (ITT) analysis. He does a good job explaining the potential downside of ITT, but got into a mess explaining PP and never properly addressed the downside of PP. It's an opportunity missed because I fear the article confuses readers even more on an important topic.

The key issue at play is non-compliance in a randomized experiment. If some patients are assigned to AA treatment and others are assigned to some other treatment, typically some subset of patients will "cross-over," (or drop out altogether), and usually such cross-over is associated with the outcome being measured--for example, a patient assigned to AA treatment felt that AA was not working and aberrantly switched to the other treatment; or vice versa.

ITT and PP differ in how they deal with the subset of non-compliers. In ITT, you analyze everyone in the experiment based on their initial assignment, ignoring non-compliance. In PP, you drop all non-compliers from the study, and analyze the subset of compliers only. (Each analysis is "extreme" in its own way.)

Between these two, I usually preferred ITT. The PP analysis answers the question: "If everyone complied with the treatment, what would be its effect?" I don't find the assumption of zero non-compliance realistic. ITT answers a different question: "Of those who take are given the treatment, what would be the expected effect?" This effect is an average of those who complied and those who did not comply, weighted by the proportion of compliers.

***

Frakt lost me when he said:

In a hypothetical example, imagine that 50 percent of the sample receive treatment regardless of which group they've been assigned to. And likewise imagine that 25 percent are not treated no matter their assignment. In this imaginary experiment, only 25 percent would actually be affected by random assignment.

First of all, the arithmetic does not work. If we ignore assignment as he suggested in the first two sentences, then the patients can either have received treatment or not. But 50 percent plus 25 percent leaves 25 percent of the patients unaccounted for.

Here is an illustration of what I think Frakt wanted to get across:

Of the 50% assigned to the treatment, 90% (45 out of 50) complied and 10% crossed over. Of the other half initially assigned to no treatment, 60% (30 out of 50) crossed over to the treatment. All in all, 75% of the study population received treatment and 25% did not... regardless of their initial assignment.

In an ITT analysis, all patients in the table are analyzed. We compare the top row with the bottom row. By contrast, in a PP analysis, we only analyze the patients along the top-left, bottom-right diagonal, namely, the 65% of the patients who complied with the assigned treatment. So, we compare the top left corner with the bottom right corner.

The important question is whether this 65% subset constitutes a random sample. Frakt implies it is: "only 25 percent [i.e. 65 percent in my example] would actually be affected by random assignment." Maybe when he said "affected by", he didn't really mean random; because it should be obvious that treatment is no longer randomized within the 65% subset.

If the 65% subset were randomly drawn from the initial population, we should still have equal proportions of treated versus non-treated but in fact, we have 70% treated versus 30% not treated. Said differently, the not-treated patients are more likely to cross over than the treated patients.

Cross-over isn't something that happens randomly. Patients are assessing their own health during the experiment, and thus, the opting out is frequently related to the observed (albeit incomplete) outcome.

***

In the article, Frakt states that the study of Humphreys et. al. "corrects for crossover by focusing on the subset of participants who do comply with their random assignment". I call this "filtering" rather than "correcting".

Does analyzing this subset lead to an accurate estimate of the treatment effect? I don't think so.

By filtering out the cross-overs, the researchers introduce a survivorship bias. If the cross-overs do so because they are unhappy about their assigned treatment, then these patients, if forced to continue the original treatment, are likely to have below-par outcomes compared to those who did not cross over. In a PP analysis, this subset is removed. Practically, this means that the treatment effect (PP analysis) is too optimistic.

Frakt is careless with his language when it comes to discussing the downside of PP analysis. He says (my italics):

it’s not always the case that the resulting treatment effect is the same as one would obtain from an ideal randomized controlled trial in which every patient complied with assignment and no crossover occurred. Marginal patients may be different from other patients...Despite the limitation, analysis of marginal patients reflects real-world behavior, too.

"Not always" leaves the impression that PP analysis is usually right except for rare situations. Note how he uses the word "limitation" above (paired with "despite"), and below, when discussing ITT analysis:

For a study with crossover, comparing treatment and control outcomes reflects the combined, real-world effects of treatment and the extent to which people comply with it or receive it even when it’s not explicitly offered. (If you want to toss around jargon, this type of analysis is known as “intention to treat.”) A limitation is that the selection effects introduced by crossover can obscure genuine treatment effects.

The choice of words leaves the impression that ITT is more limited than PP when both analyses suffer from problems arising from the same source: patients with worse outcomes are more likely to cross over.

***

Many readers of the NYT article link to a much longer article in The Atlantic. It appears that the scientific evidence on AA is very weak.

 

 

What popular baby names teach us about data analytics

In my latest piece for Harvard Business Review (link), I tackle this common problem in the interactions between data scientists and business managers:

A typical big data analysis goes like this: First, a data scientist finds some obscure data accumulating in a server. Next, he or she spends days or weeks slicing and dicing the numbers, eventually stumbling upon some unusual insights. Then, a meeting is organized to present the findings to business managers, after which, the scientist feels disgruntled or even disrespected while the managers wish they could take the time back.

Using analyses of the popular baby names dataset as an example, I contrast the kind of analysis that generates click bait (e.g. the most "poisoned" names, the most "trendy" names) with the kind of analysis that generates potentially real business value.

More here.

Surveymonkey's unipolarity and I

I was creating an online survey using Surveymonkey earlier this week. They asked me to try their new design, and so I did. There appeared to be a bug in one of the features. It kept preventing me from displaying the questions in a certain way. I tried a bunch of tricks but after ten minutes, decided to switch back to the old design. I clicked on their Feedback link and after describing my problem, they asked me to answer a few questions.

Here is one question:

This question is as standard as they come in a customer satisfaction survey.

My mood at the time was slightly unhappy. Just as I was about to click on that second-to-last radio button, I stopped. Can you see why? (Look at the choices more carefully.)

 

The fourth button is labelled "Slightly Satisfied". I was expecting it to say "Slightly Dissatisfied"!

Then I realized Surveymonkey is using a unipolar scale. All five answers are varying levels of satisfaction. I'm more used to a bipolar scale, such as:

• Extremely satisifed
• Somewhat satisfied
• Neither satisfied nor dissatified
• Somewhat dissatisfied
• Extremely dissatisfied

The bipolar scale is centered in the middle and allows answers in both positive and negative directions.

 

I was debating between the last two choices. Was I "slightly satisfied" or "not at all satisfied"? Surely, I wasn't 100 percent unhappy, far from it. But "slightly" was also inappropriate. The mirror image of "slightly dissatisfied" should be "mostly satisfied", which meant I should be debating between the second button and the last. 

However, "very satisfied" didn't fit with my mood, even though technically it was the mirror image of it. I wanted to express a negative sentiment, albeit minor, not a positive sentiment, albeit qualified. (Since I couldn't bear to pick either, I abandoned the survey at that point.)

 

I am not a fan of unipolar scales for many applications. For example, if you are measuring political attitudes (conservative and liberal), would your choices be:

• Extremely conservative
• Very conservative
• Moderately conservative
• Slightly conservative
• Not at all conservative

or would they be

• Extremely conservative
• Somewhat conservative
• Neither conservative nor liberal
• Somewhat liberal
• Extremely liberal

?

The unipolar scale automatically creates the problem of which pole to feature in those answers. Conservatives probably won't have an issue with that unipolar conservative scale but it's difficult for a "somewhat liberal" person to think he/she is "moderately conservative" or "very conservative"; vice versa.

The criticism of bipolar scales is that people (and I think this means Americans, and I doubt it generalizes to other cultures) tend to bias toward the positive direction relative to the negative. I don't see that as a big problem if a 7-point scale is used, or have the scale re-centered.

Yet another popular nutrition headline doesn't stand up to scrutiny

Are science journalists required to take one good statistics course? That is the question in my head when I read this Science Times article, titled "One Cup of Coffee Could Offset Three Drinks a Day" (link).

We are used to seeing rather tenuous conclusions such as "Four Cups of Coffee Reduces Your Risk of X". This headline takes it up another notch. A result is claimed about the substitution effect of two beverages. Such a result is highly unlikely to be obtained in the kind of observational studies used in nutrition research. And indeed, a glance at the source materials published by the World Cancer Research Fund (WCRF) confirms that they made no such claim.

The headline effect is pure imagination by the reporter, and a horrible misinterpretation of the report's conclusions. Here is a key table from the report:

The conclusion on alcoholic drinks and on coffee comes from different underlying studies. Even if they had come from the same study, you cannot take different regression effects and stack them up. The effect of coffee is estimated for someone who is average on all other variables. The effect of alcohol is estimated for someone who is average on all other variables. The average person in the former case is not identical to the average person in the latter case. So if you add (or multiply, depending on your scale) the effects, the total effect is not well-defined.

In addition, you can only add (or multiply) effects if you first demonstrate that the two factors do not interact. If there is interaction, the effect of alcohol is different for people who drink less coffee relative to those who drink more. The alcohol effect stated in the table above, as I already pointed out, is for an average coffee drinker. Conversely, the protective effect of coffee may well vary with alcohol consumption.

***

The reporter also misrepresented the nature of the analysis. We are told: "In the study of 8 million people, cancer risk increased when they consumed three drinks per day. However, the study also found that people who also drank coffee, offset some of the negative effects of alcohol."

The reporter made it sound like a gigantic randomized controlled study was conducted. This is a horrible misjudgment. WCRF did not do any study at all, and certainly no researcher asked anyone to drink specific amounts of alcohol or coffee. The worst is the comment on people who drank coffee as well as alcohol. I can't find a statement in the WCRF report about such people. It's simply made up based on the false logic described above.

***

At one level, the journalist misquoted a scientific report. At another level, the WCRF report is rather disappointing.

The authors of the executive summary repeatedly use the language of causation. For example, "There is strong evidence that being overweight or obese is a cause of liver cancer." Really? Show me which study shows obesity "causes" liver cancer?

Take one of their most "convincing" findings: "Aflatoxins: Higher exposure to aflatoxins and consumption of aflatoxin-contaminated foods are convincing causes of liver cancer." The causation is purely an assumption of the panel who reviewed prior studies. In Section 7.1, readers learn that this cause-effect conclusion comes from "four nested case-control studies and cohort studies" for which "meta-analyses were not possible". So not a single randomized trial and no estimation of the pooled effect.

What is nicely done in the report is the inclusion of "mechanisms" which are speculative explanations for the claimed causal effects. It's great to have thought carefully about the biological mechanisms. Nevertheless, these sections are basically "story time" unless researchers succeed in establishing those unproven links.