« To prove inflation, get some inflated data | Main | Gelman's Museum of Scholarly Misconduct »


Feed You can follow this conversation by subscribing to the comment feed for this post.


Did you read the actual study though? The media reports about it are quite wrong! The study found that people actually screened for colonoscopy had a 50% lower chance of dying from colon cancer.




Cris: As I mentioned, I will post about the study's findings separately. However, I did scan through it before writing this post, and the finding you cited belongs to category (d) of the common reactions when studies don't confirm prior beliefs.

Here is the summary found on the first page of the published paper:

"During a median follow-up of 10 years, 259 cases of colorectal cancer were diagnosed in the invited group as compared with 622 cases in the usual-care group. In intention-to-screen analyses, the risk of colorectal cancer at 10 years was 0.98% in the invited group and 1.20% in the usual-care group, a risk reduction of 18% (risk ratio, 0.82; 95% confidence interval [CI], 0.70 to 0.93). The risk of death from colorectal cancer was 0.28% in the invited group and 0.31% in the usual-care group (risk ratio, 0.90; 95% CI, 0.64 to 1.16)"

The particular result you mentioned is the "per protocol" analysis which is not the primary analysis pre-registered in the trial. I will explain in the upcoming post why "per protocol" analysis is not considered the proper analysis (except in Covid-19 emergency-use studies!). As I wrote in this post, the biggest problem with coming up with alternative analyses is that when one method is found to deliver the comfortable/desirable outcome, then there is a tendency to throw away the proper or other analyses with the opposite results, which is exactly what's happening right now.

At best, one can regard this as something to think about but in no way should the "per protocol" analysis replace or overrule the "intention to screen" analysis.


Interesting! Nevertheless, your remarks on "per protocol" may have some merit, yet the study still has never been an RCT of colonoscopy but an RCT of "invitation to colonoscopy". Pre-registering or not does not change this basic fact, doesn't it? You should definitely address this common sense objection in your upcoming discussion of the study.


gg: Yes, I'll be addressing those as well. The post is being drafted as I write this. The key is to understand the process of doing science. Pre-registering means that the community has thought through many of these issues, and agreed to a specific way of running this experiment. What I'm objecting to in this post is changing the rules of the game after seeing the results - in particular, when the same arguments would not have been applied if the results came out differently.

Further, the proponents of what you call the "common sense" argument tout the "adjusted per-protocol" analysis which is also performed on the "invitation to screening" experiment.

The comments to this entry are closed.

Get new posts by email:
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.


  • only in Big Data
Numbers Rule Your World:
Amazon - Barnes&Noble

Amazon - Barnes&Noble

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Principal Analytics Prep

Link to Principal Analytics Prep