This is a supplement to the previous post about a new research paper on the effect of Alcoholics Anonymous, and an NY Times exposition that I commented on. A misreading of that article led me to complain about per-protocol analysis, which wasn't the methodology behind the Humphrey et. al. research. I will explain their methodology in this post (known as instrumental variables analysis).
In the last post, I showed this hypothetical situation, involving patients who "cross over" (disobey treatment assignment) in a randomized experiment.
In the paper, actual treatment is measured by the change in frequency of attending AA meetings (relative to baseline).
Because initial treatment assignment (rows) is random, one expects that equal proportions of people would have moved out of state, got married, got divorced, etc. Similarly, one expectas equal proportions of people would have increased AA attendance. But in the table above, 90% of people in the treatment arm upped attendance while only 60% of those assigned to no treatment increased attendance. (The researchers use a continuous scale of frequency rather than proportion but the concept is the same.)
Of course, the random assignment to treatment itself is a cause of higher relative attendance. People are told to go to AA meetings. But there are other reasons for increased attendance, such as self-motivation leading those in the no-treatment arm to cross over.
In ITT analysis, you ignore the actual attendance, and analyze how treatment assignment affects the amount of drinking.
Alternatively, one can run a regression of frequency of AA meetings on amount of drinking (relative to baseline). This will yield a result such as "the more meetings someone attends, the less they drink". The problem with this analysis is that while the initial assignment is random, the actual attendance is tainted by selection bias.
Instead of using the actual frequency of AA meetings as a regressor, the instrumental variables (IV) analysis uses a predicted frequency of AA meetings. The prediction is itself a regression of treatment assignment and demographic variables on the actual frequency of AA meetings. In other words, we only care about the proportion of the variability in AA attendance that can be explained by the random assignment (controlling for the demographic variables). The remaining variability (due to self-motivation, etc.) are left on the table.
This is the "correction" that Frakt inferred in the New York Times article. I think Frakt is correct that the conclusion can be applied only to those who obey the protocol but I don't think the researchers drop all non-compliers from the dataset.
Also, Humphrey, et. al. seem to be at odds with the author of The Atlantic article, as they say "The long-established positive association between AA involvement and better outcomes was therefore consistent with, but did not prove, causation."