These journal editors probably forgot to drink their morning coffee

Here we go again. Another useless study published in a peer-reviewed journal (Mayo Clinic Proceedings) with a relatively high impact factor and promoted as "Breaking News from the Editor" to the press who then attached a sensational headline and reported it as "science".

This is what caught my eye: "Drinking more than 28 cups of coffee a week may be harmful for people younger than 55, according to a study." I saw this on USA Today (link)... but many other outlets also carried the story, including NPR, CBS, AJC, Guardian, etc.

Just the headline should sound alarm bells. Why 28 cups a week? Is it ok to have 27.5 cups a week? Why 55 years old? On your 55th birthday, should you stop worrying about the number of cups?

Of course, even that speculation betrays a causation creep. Nothing in this study, nor any study of this type, can prove a causal link between the accused food and harm.

Now let's check out the summary by the authors:

Conclusion: In this large cohort, a positive association between coffee consumption and all-cause mortality was observed in men and in men and women younger than 55 years. On the basis of these findings, it seems appropriate to suggest that younger people avoid heavy coffee consumption (ie, averaging >4 cups per day). However, this finding should be assessed in future studies of other populations.

Note that construction: "a positive association between .. and .. was observed ... on the basis of these findings, it seems appropriate to suggest that". All those weasel words. What they really mean is "we have data showing a positive association and we make an assumption that correlation = causation, therefore you should ..."

Do you think you'll learn anything in the journal paper about the biological or chemical mechanism by which coffee causes death? Take a guess.

***

USA Today commits the other sin of health reporting: failure to explain the level of harm, and the context to interpreting such. USA Today tells its readers:

Men younger than 55 who drank more than 28 cups of coffee a week (four cups a day) were 56% more likely to have died from any cause.

56% compared to what? Turns out it's compared to men younger than 55 who do not drink coffee at all. There is a wide gap between drinking over 28 cups a week and drinking zero.

What's also missing is the error bar. According to the paper, the 95% interval is 30% to 87%. Not kidding, it's 25% in each direction.

Absent is the context for understanding what 56% means. How many additional deaths for every 10,000 heavy coffee drinkers? Amusingly, you can't figure this out even after reading the entire paper. The authors got away with presenting data in aggregate (33,900 males, 2198 male deaths, etc.) without showing age group breakouts (Where were the editors??) Stymied, I glance at their other result, the one for "all" men.

In men, those who drank more than 28 cups of coffee weekly had a 21% higher risk of dying compared with their non-coffee-consuming peers.

By the way, the error bar on this result is 4% to 40%. Now, I can't interpret this result either. The baseline death rate in the study was 6.48%. Nowhere in the paper does it break out the number of deaths by the level of coffee drinking. There is no way to know how many of those 2,198 male deaths were men who did not drink coffee at all.

While the important data on the outcomes being analyzed are not published, the authors of the paper regale us with numbers such as the N=11 people who were excluded from the population because they had a history of stroke.

So I ask again: where were the editors? how did they miss this?

***

Readers should also check out Andrew Gelman's recent rant about the excessive reliance on statistical significance, and the neglect of data quality in journal editing (link). It's not that hard to draw up a list of data required for publication.

***

I may write another post about the other issues with the analysis. But here is the most important one in case I don't get to it. This is the chart that supports their finding about men under the age of 55 who drink more than 28 cups of coffee a week. They want us to look at the blue bar on the far right of the chart.

Look at the other blue bars. Need I say more?

 

 

Comments

[Ken]

I don't mind that they are using observational data, as it would be almost impossible to run a trial. Unfortunately it all goes downhill from there. It wouldn't be surprising if coffee had an effect on cardiovascular mortality, as caffeine is a stimulant, but they didn't find that, they found an effect for all cause mortality, but then they didn't look at what particular form of mortality.

From the paper there are obvious differences in the coffee groups, which indicates that there are probably employment, educational and lifestyle differences, which they don't have as covariates. Several of their covariates aren't very accurate. Physical activity, alcohol consumption and smoking deserve more than a binary. Year of entry to the study can also be an important confounder. There are probably other things. I dislike categorising things unless necessary, so using actual cups of coffee as a covariate would be my preference, although lots of medical journals seem to be OK with the idea that at 28 cups per week people suddenly start dying.

One thing that will amuse anyone with a good knowledge of survival analysis is "proportional
hazards assumption was tested by Martingale-based residuals". I hope not.

[Jon Peltier]

My own quick and admittedly dirty analysis:
I can draw a single horizontal line that passes through all ten sets of error bars in that chart.
This tells me the effect isn't very strong or particularly significant, despite any patterns I may think I see.

[Andreas]

I currently drink about 11 cups of coffee a week. I should increase this to 18 cups if I want to live!