Some statistics about nutrition statistics

I only read nutrition studies in the service of this blog but otherwise, I don't trust them or care. Nevertheless, the health beat of most media outlets is obsessed with printing the latest research on coffee or eggs or fats or alcohol or what have you.

Now, the estimable John Ioannidis has published an editorial in BMJ titled "Implausible Results in Human Nutrition Research". John previously told us about the crisis of false positives in medical research.

Oops, here are some statistics on nuitrition "science":

• In 52 attempts at using randomized experiments to validate findings from observational studies, the number of times the findings were replicated: 0
• In the NHANES questionnaire (the basis of all those findings), two-thirds of the participants provided answers that imply an energy intake that is "incompatible with life". I haven't read this paper; seems like worthwhile reading.
• There are at least 34,000 papers on PubMed with keywords "coffee OR caffeine" which means this one nutrient has been associated with almost any interesting outcome.
• Almost every single nutrient imaginable has peer reviewed publications associating it with almost any outcome. A statistician should never give the advice "If at first you don't succeed,..."
• Many findings are entirely implausible (and still get published in top journals)... for example, the idea that a couple of servings a day of a single nutrient will halve the burden of cancer is clearly "too good to be true," even more so for anyone who is familiar with this literature
• "Big datasets just confer spurious precision status to noise"
• Randomized experiments offer hope but are woefully undersized (like requiring 10 times the current sample).

Just to nail home the point, John concludes: "Definitive solutions will not come from another million observational papers or a few small randomized trials."

 

Comments

[Ken]

At one time I worked with some people who were doing a study in about 3,000 older subjects with evaluations of subjects every 3 years. I looked at the nutritional data for what they were interested in and there were some strange results, so I then looked at total energy. For some subjects this was varying by a factor of 3 from one period to the next. Some went up by a factor of 3, some went down by a factor of 3. When they were low they were theoretically starving and when high they were absolutely gorging themselves on food, as someone in training for an sporting event might.

My conclusion was that they didn't have a clue what they were eating and were often retrospectively filling in the diaries.

My impression is that the body copes quite well with what you eat provided that it isn't extreme, so most effects will be small.

[Basil Geoffrey]

I think this is a somewhat oversimplified view of nutritional studies (yes - there are many poor ones, I don't think anyone denies this - but not all); diet is incredibly difficult to assess, but it can have a considerable impact on health because it is consumed continuously over a very long time. The difficulties and limitations of different dietary assessment methods are well known, and a of effort is invested in refining these methods or finding alternatives (e.g. biomarkers, online-assessment tools etc).

Nutritional research is made difficult not only because of the limitations of dietary assessment, but also because the effect size is usually fairly small and therefore very hard to detect. Furthermore, it is not always clear whether an observed association is caused by a specific bioactive, nutrient, food, dietary pattern or something completely different. Take fibre as an example - whose intake is very difficult to measure (there exists more than one definition), but with sufficiently well designed studies it is possible to show an effect.

Do we need better studies in nutrition? No doubt we do - and we can learn from the medical profession and follow their example with standards and regulations. (but they should remember that a couple of years ago drug trials had some limitations too ...). However, it is important to understand that nutrition is fundamentally different from medicine - very different intervention (long term, complex) and very small effect size, but with a possibly huge impact on public health (eg trans-fats).

[Kaiser]

BG: Thanks for the comment. Yes, we need better studies. Ioannidis makes that point too and mentioned a few ideas, including the shift from focusing on assessing single nutrients to measuring diets. Trying to establish long-term, small effects of bundled stimuli is also a difficult problem in a business setting (multiple marketing actions taken over time); and we have not really come to terms with the difficulties.

It doesn't help when poor studies get reported in big media, with screaming headlines claiming you will get healthier by drinking another cup of coffee or whatever. Like you said, most of these effects, even if found, will be small.

[Ken]

One problem with nutritional trials is that people tend to be non-compliant over the years that the studies need to be run. Maybe someone will stick to a diet for a short period, but basically unless someone is given the choice of food that they enjoy eating and is easy to prepare they will end up with whatever they used to eat.