« Reading guides for instructors, book clubs, etc. | Main | What can we learn from Freakonomics? »


Feed You can follow this conversation by subscribing to the comment feed for this post.


Kaiser do you think this is a fundamental problem with RCTs? I am a huge supporter of them, but it seems if they are done in massive amounts of quantities you will continue to find (X, Y, and Z) as a cause for the outcome. I guess what I am really asking is, when do we stop investigating the claim?

Bruce Stephens

This wasn't, of course, a RCT. IIUC almost no such nutritional studies are RCTs. It was observational (epidemiological).

With RCTs I guess complaint 1 still stands: if you're studying one outcome then you're likely to end up presenting that one outcome and perhaps ignoring others. And the meta-complaint still stands, that the statistics tend to be presented in relative risk terms which (IIUC) are well accepted as being one of the worst ways to communicate such statistics. Complaints 2 and 3 would usually be avoided, presuming everything goes well (the study is effectively blinded, sufficiently large, etc.).


Let's talk about RCTs. RCT is typically used in clinical trials to test new drugs. As Bruce pointed out, complaint #1 applies to clinical trials; you keep hearing that the trials are not designed to detect potential harms because the primary goal is to detect potential benefit. Complaint #2 also applies because typically a drug cures one thing (say, reduce cholesterol) but the disease (heart disease) has multiple causes.

Complaint #3 is partially offset by the randomization - the point of randomization is to balance the unknown factors that may be skewing the outcomes. However, randomizing doesn't help the interpretation of regression coefficients. Controlling for other variables means that we have assumed the existence of someone who has average values of all the control variables. That person probably doesn't exist in the test sample.

Jeff Weir


We need a self-sufficiency test for comments, gents.

Jeff Weir

Red meat surely has some benefits. So we must balance both the benefits and the harms in order to decide how much to eat.

Fair enough. But I don't take issue with the headline "Study gives more reasons for passing on red meat", because it's not incorrect...the study did in fact give one more reason to pass. There's still plenty of reasons not to pass, like for instance the obvious enjoyment many readers of the study get from eating it (the meat, not the study!)

THis headline certainly doesn't say "Red meat is on balance bad for you". It's just prompting the reader to revise their intake to another indifference curve given this one more bit of information to hand tips the scale a little away from meat. That's how markets work, isn't it? And that's how the null hypothesis works, isn't it? It's obviously much more possible to quantify one well-defined harm than the entire possible range of harms and benefits to form a robust opinion of net benefit.

"Each extra serving was also tied to a 16 percent higher chance of dying from cardiovascular disease". If this was so, with 5 extra servings, we'd all be dead.. Without being familiar with the study, this isn't the only way you can read this stat...they might mean tha for each extra serving, your risk increases by 16% relative to the last level of risk.



RCT = randomized controlled trials (wiki)

this is a setup typically used to test the effectiveness of a new drug. in the simplest case, a group of patients known as controls are given a placebo or the prevailing treatment while a separate group gets the new drug. Assignment of any given patient to one of the two groups is by random lottery. The beauty of random lottery is that it equalizes all other factors including unknown factors. Real life is not so simple because many factors we'd like to study cannot be randomly assigned e.g. you can't force someone to smoke. Also, patients and/or doctors may decide to ignore the protocol, say if the traditional treatment is failing for someone in the control arm, there is a temptation to "cross over" and give the patient the new drug.




Jeff Weir

Cool. I just googled IIUC, and ironically now I do!

Hey, you got a comments RSS feed for this baby yet that readers can subscribe to? I'd like to follow the action in the Recent Comments bit by RSS...heaps of good learning here (and not just about internet slang).

The comments to this entry are closed.

Get new posts by email:
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.


  • only in Big Data
Numbers Rule Your World:
Amazon - Barnes&Noble

Amazon - Barnes&Noble

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Principal Analytics Prep

Link to Principal Analytics Prep