« Gelman's Museum of Scholarly Misconduct | Main | With adjustments comes responsibility »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Antonio

"The 30% reduction in colon cancer risk" Maybe did you confuse with the marketing example?
I'm wondering how to solve this problem.
Order the law enforcement to take people randomized to experimental group to hospital and force them to have the colonoscopy?
Randomize people getting the colonoscopy and submitting half of them to a "placebo" colonoscopy?
(obviously, not two feasible options)

Kaiser

Antonio: The 30% reduction is the "adjusted per-protocol analysis"; the intent-to-screen which respects randomization shows 18%.
There is nothing to solve - if the experiment is used to find a practical solution. In practice, a proportion of people will ignore guidance to do a colonoscopy so the current setup mimics that. (Analogously, even something non-invasive like "wear a mask" is shunned by a large proportion of Americans.)
Put differently, an intervention may fail because it is not effective but it can also fail despite being effective if adoption rate is low.
Here's a design that measures only the effect on people who choose to do a colonosopy: invite everybody, then randomize those who accept the invite, so that half of those people are held out of the intervention. The problem of course is that this measures the effect on those who choose to do it but the overall effect will be muted by the subset who refuses the treatment.

Ken

Inviting everyone, and then only randomising the ones who accept the invite is perfectly reasonable. All our population now is people who want to be screened and now we are doing a randomised control on them. The main worry with doing this type of trial is that the people who are randomised to usual care may decide to have a colonoscopy because they are now aware of them.

The reason the observational studies showed an effect would be that people who had colonoscopies are inherently different from ones that don't. They are probably more careful about their health, for example diet, and may be more likely to see a doctor if minor symptoms occur. It is similar to the problem with Covid, where people who wear masks are more likely to adopt other measures to reduce the risk of Covid, like spending the last couple of years at home.

Kaiser

Ken: Good point. In the current study, because in those countries, colonoscopy is not the usual care, they were sure (and they verified) that the usual-care group didn't get colonoscopies on their own. In the randomize on accepting invitation setup, there is higher risk that someone violates the treatment protocol; but that is a risk with every RCT, including the Covid-19 vaccine trials. In fact, as I have pointed out here, the fact that close to 100% of all placebo participants in those vaccine trials almost immediately took up the vaccine when they were offered one actually proved the point that those trials had incredibly biased populations.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name is required. Email address will not be displayed with the comment.)

Get new posts by email:
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.

Search3

  • only in Big Data
Numbers Rule Your World:
Amazon - Barnes&Noble

Numbersense:
Amazon - Barnes&Noble

Junk Charts Blog



Link to junkcharts

Graphics design by Amanda Lee

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Principal Analytics Prep



Link to Principal Analytics Prep

Community