« To prove inflation, get some inflated data | Main | Gelman's Museum of Scholarly Misconduct »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Cris

Did you read the actual study though? The media reports about it are quite wrong! The study found that people actually screened for colonoscopy had a 50% lower chance of dying from colon cancer.

https://www.linkedin.com/posts/anthony-d-cox-ph-d-234a3456_effect-of-colonoscopy-screening-on-risks-activity-6986007045565018112-ztZ9?utm_source=share&utm_medium=member_ios

https://www.linkedin.com/posts/toyand_new-study-examines-the-effectiveness-of-colonoscopies-activity-6985664085434540033-ZBFp?utm_source=share&utm_medium=member_ios

Kaiser

Cris: As I mentioned, I will post about the study's findings separately. However, I did scan through it before writing this post, and the finding you cited belongs to category (d) of the common reactions when studies don't confirm prior beliefs.

Here is the summary found on the first page of the published paper:

"During a median follow-up of 10 years, 259 cases of colorectal cancer were diagnosed in the invited group as compared with 622 cases in the usual-care group. In intention-to-screen analyses, the risk of colorectal cancer at 10 years was 0.98% in the invited group and 1.20% in the usual-care group, a risk reduction of 18% (risk ratio, 0.82; 95% confidence interval [CI], 0.70 to 0.93). The risk of death from colorectal cancer was 0.28% in the invited group and 0.31% in the usual-care group (risk ratio, 0.90; 95% CI, 0.64 to 1.16)"

The particular result you mentioned is the "per protocol" analysis which is not the primary analysis pre-registered in the trial. I will explain in the upcoming post why "per protocol" analysis is not considered the proper analysis (except in Covid-19 emergency-use studies!). As I wrote in this post, the biggest problem with coming up with alternative analyses is that when one method is found to deliver the comfortable/desirable outcome, then there is a tendency to throw away the proper or other analyses with the opposite results, which is exactly what's happening right now.

At best, one can regard this as something to think about but in no way should the "per protocol" analysis replace or overrule the "intention to screen" analysis.

gg

Interesting! Nevertheless, your remarks on "per protocol" may have some merit, yet the study still has never been an RCT of colonoscopy but an RCT of "invitation to colonoscopy". Pre-registering or not does not change this basic fact, doesn't it? You should definitely address this common sense objection in your upcoming discussion of the study.

Kaiser

gg: Yes, I'll be addressing those as well. The post is being drafted as I write this. The key is to understand the process of doing science. Pre-registering means that the community has thought through many of these issues, and agreed to a specific way of running this experiment. What I'm objecting to in this post is changing the rules of the game after seeing the results - in particular, when the same arguments would not have been applied if the results came out differently.

Further, the proponents of what you call the "common sense" argument tout the "adjusted per-protocol" analysis which is also performed on the "invitation to screening" experiment.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name is required. Email address will not be displayed with the comment.)

Get new posts by email:
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.

Search3

  • only in Big Data
Numbers Rule Your World:
Amazon - Barnes&Noble

Numbersense:
Amazon - Barnes&Noble

Junk Charts Blog



Link to junkcharts

Graphics design by Amanda Lee

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Principal Analytics Prep



Link to Principal Analytics Prep

Community