Three complaints about medical journalism

It's very frustrating to read journalism about health and medicine. They repeat bad advice from papers that are published with narrowly-defined research objectives.

Here's a recent example about red meat (link; the study itself). The headline gives us a prescription: "Study gives more reasons for passing on red meat". The key paragraph tells us:

Each extra serving was also tied to a 16 percent higher chance of dying from cardiovascular disease in particular, and a 10 percent chance of dying from cancer -- even after taking into account other aspects of health and lifestyle that could influence the chance of dying, such as weight, smoking, the rest of their diet and socioeconomic factors.

To summarize: Food X is linked to disease Y. Therefore, eat less (or don't eat) X.

***

Complaint #1: there is no such thing as "good food" or "bad food"

Every food has harms and benefits. These studies are almost always looking for "causes" of disease. Rarely are studies reported to prove that an extra ounce of red meat, etc. will enhance our intelligence by 10% (I'm making this up.) Red meat surely has some benefits. So we must balance both the benefits and the harms in order to decide how much to eat.

 

Complaint #2: there are multiple causes for cancer/heart disease/etc.

There is an unspoken implication that if one eats less of food X, one will not get disease Y. This is wrong on several levels. First, a 10 or 12 or 16 percent reduction in mortality rate may sound impressive but what is the base rate? The average age of 120,000 people being analyzed was about 45-50. In following them for 22-28 years, about 24,000 died of any cause, that's 20 percent. A 10 percent improvement of this probability is 2 percent. Is this practically meaningful? Only you can tell.

Another problem is that cancer/heart disease etc. has multiple causes. Most people are exposed to multiple risks that could lead to these diseases. Reducing one type of risk most likely will not stop you from getting disease Y.

 

Complaint #3: you can only control for things you know about

One typical defense of such analysis is the use of control variables. In the above quote, we read something like "after taking into account other aspects of health and lifestyle that could influence the chance of dying, such as weight, smoking, the rest of their diet and socioeconomic factors."

Impressive, right? The problem is for most diseases like cancer, we don't yet, and may never, know the entire causal structure of how one gets it. We can't control for variables that we don't know are related to the disease, or ones we mistakenly thought to be unrelated, or ones that are not being measured.

Secondly, these regressions almost never take into account the fact that all those control variables are not independent; for example, poorer people are more likely to have less healthy diets, someone who eats more meat are more likely to eat less vegetables. Say, someone reduces meat intake. It is likely this person would be eating more vegetables and/or fruits. The single coefficient for meat in the regression would underestimate the impact.

***

I cannot end without pointing out that it is misleading for the reporter to say "Each extra serving was also tied to a 16 percent higher chance of dying from cardiovascular disease". If this was so, with 5 extra servings, we'd all be dead. It should really say "Have one extra serving per day every day you're alive", which was what the researchers were measuring.

Comments

[Jonathan]

Kaiser do you think this is a fundamental problem with RCTs? I am a huge supporter of them, but it seems if they are done in massive amounts of quantities you will continue to find (X, Y, and Z) as a cause for the outcome. I guess what I am really asking is, when do we stop investigating the claim?

[Bruce Stephens]

This wasn't, of course, a RCT. IIUC almost no such nutritional studies are RCTs. It was observational (epidemiological).

With RCTs I guess complaint 1 still stands: if you're studying one outcome then you're likely to end up presenting that one outcome and perhaps ignoring others. And the meta-complaint still stands, that the statistics tend to be presented in relative risk terms which (IIUC) are well accepted as being one of the worst ways to communicate such statistics. Complaints 2 and 3 would usually be avoided, presuming everything goes well (the study is effectively blinded, sufficiently large, etc.).

[Kaiser]

Let's talk about RCTs. RCT is typically used in clinical trials to test new drugs. As Bruce pointed out, complaint #1 applies to clinical trials; you keep hearing that the trials are not designed to detect potential harms because the primary goal is to detect potential benefit. Complaint #2 also applies because typically a drug cures one thing (say, reduce cholesterol) but the disease (heart disease) has multiple causes.

Complaint #3 is partially offset by the randomization - the point of randomization is to balance the unknown factors that may be skewing the outcomes. However, randomizing doesn't help the interpretation of regression coefficients. Controlling for other variables means that we have assumed the existence of someone who has average values of all the control variables. That person probably doesn't exist in the test sample.

[Jeff Weir]

RCT. IIUC. WTF?

We need a self-sufficiency test for comments, gents.

[Jeff Weir]

Red meat surely has some benefits. So we must balance both the benefits and the harms in order to decide how much to eat.

Fair enough. But I don't take issue with the headline "Study gives more reasons for passing on red meat", because it's not incorrect...the study did in fact give one more reason to pass. There's still plenty of reasons not to pass, like for instance the obvious enjoyment many readers of the study get from eating it (the meat, not the study!)

THis headline certainly doesn't say "Red meat is on balance bad for you". It's just prompting the reader to revise their intake to another indifference curve given this one more bit of information to hand tips the scale a little away from meat. That's how markets work, isn't it? And that's how the null hypothesis works, isn't it? It's obviously much more possible to quantify one well-defined harm than the entire possible range of harms and benefits to form a robust opinion of net benefit.

"Each extra serving was also tied to a 16 percent higher chance of dying from cardiovascular disease". If this was so, with 5 extra servings, we'd all be dead.. Without being familiar with the study, this isn't the only way you can read this stat...they might mean tha for each extra serving, your risk increases by 16% relative to the last level of risk.

[Kaiser]

Jeff:

RCT = randomized controlled trials (wiki)

this is a setup typically used to test the effectiveness of a new drug. in the simplest case, a group of patients known as controls are given a placebo or the prevailing treatment while a separate group gets the new drug. Assignment of any given patient to one of the two groups is by random lottery. The beauty of random lottery is that it equalizes all other factors including unknown factors. Real life is not so simple because many factors we'd like to study cannot be randomly assigned e.g. you can't force someone to smoke. Also, patients and/or doctors may decide to ignore the protocol, say if the traditional treatment is failing for someone in the control arm, there is a temptation to "cross over" and give the patient the new drug.

[John]

http://garytaubes.com/2012/03/science-pseudoscience-nutritional-epidemiology-and-meat/

http://www.zoeharcombe.com/2012/03/red-meat-mortality-the-usual-bad-science/

[Jeff Weir]

Cool. I just googled IIUC, and ironically now I do!

Hey, you got a comments RSS feed for this baby yet that readers can subscribe to? I'd like to follow the action in the Recent Comments bit by RSS...heaps of good learning here (and not just about internet slang).