« Gelman's Museum of Scholarly Misconduct | Main | With adjustments comes responsibility »


Feed You can follow this conversation by subscribing to the comment feed for this post.


"The 30% reduction in colon cancer risk" Maybe did you confuse with the marketing example?
I'm wondering how to solve this problem.
Order the law enforcement to take people randomized to experimental group to hospital and force them to have the colonoscopy?
Randomize people getting the colonoscopy and submitting half of them to a "placebo" colonoscopy?
(obviously, not two feasible options)


Antonio: The 30% reduction is the "adjusted per-protocol analysis"; the intent-to-screen which respects randomization shows 18%.
There is nothing to solve - if the experiment is used to find a practical solution. In practice, a proportion of people will ignore guidance to do a colonoscopy so the current setup mimics that. (Analogously, even something non-invasive like "wear a mask" is shunned by a large proportion of Americans.)
Put differently, an intervention may fail because it is not effective but it can also fail despite being effective if adoption rate is low.
Here's a design that measures only the effect on people who choose to do a colonosopy: invite everybody, then randomize those who accept the invite, so that half of those people are held out of the intervention. The problem of course is that this measures the effect on those who choose to do it but the overall effect will be muted by the subset who refuses the treatment.


Inviting everyone, and then only randomising the ones who accept the invite is perfectly reasonable. All our population now is people who want to be screened and now we are doing a randomised control on them. The main worry with doing this type of trial is that the people who are randomised to usual care may decide to have a colonoscopy because they are now aware of them.

The reason the observational studies showed an effect would be that people who had colonoscopies are inherently different from ones that don't. They are probably more careful about their health, for example diet, and may be more likely to see a doctor if minor symptoms occur. It is similar to the problem with Covid, where people who wear masks are more likely to adopt other measures to reduce the risk of Covid, like spending the last couple of years at home.


Ken: Good point. In the current study, because in those countries, colonoscopy is not the usual care, they were sure (and they verified) that the usual-care group didn't get colonoscopies on their own. In the randomize on accepting invitation setup, there is higher risk that someone violates the treatment protocol; but that is a risk with every RCT, including the Covid-19 vaccine trials. In fact, as I have pointed out here, the fact that close to 100% of all placebo participants in those vaccine trials almost immediately took up the vaccine when they were offered one actually proved the point that those trials had incredibly biased populations.

The comments to this entry are closed.

Get new posts by email:
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.


  • only in Big Data
Numbers Rule Your World:
Amazon - Barnes&Noble

Amazon - Barnes&Noble

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Principal Analytics Prep

Link to Principal Analytics Prep