If it's an RCT, it can be trusted

I've written often about the problems with conducting observational studies. We like to hold up randomized controlled trials (RCTs) as the "gold standard," and if a study is described as an RCT, and especially if its design is pre-registered, we don't ask many questions.

This point of view is becoming unsustainable, based on a recent Nature article.

The gist of the article is that experts now believe at least one out of every four RCTs contain so many problems that their findings should be ignored, leaving the possibility also that some of these RCTs may be entirely fake. In coming to this conclusion, these experts sought and received the underlying individual-level datasets.

These problematic or fake RCTs make their way to systematic reviews (aka meta-analyses), and throught those, they sometimes influence medical practice. Systematic reviews are summary articles that weigh evidence from all studies on a given topic, e.g. whether sugar helps reduce death after head injuries.

In a particularly egregious example, a late Japanese bone-health researcher has had over 100 studies retracted due to fraud, but his papers have appeared in 88 different systematic reviews, half of which would have arrived at a different conclusion if his studies were omitted.

Besides, there is no universal requirement for authors of systematic reviews to revise their analyses should included studies be later retracted. What a mess!

One researcher estimated 20-30 percent of the papers included in systematic reviews in women’s health are suspect. Even when the authors of these systematic reviews have been alerted to the retracted studies, many have ignored pleas to update their findings.

When editors reject studies because of data problems from one journal, the same studies almost always eventually get published in a different journal, “sometimes with different data to those submitted [to the first journal].”

Fixing the RCT problem requires a cultural shift, which does not seem to be forthcoming.

***

The Nature article catalogs a number of objections to research that exposes RCT fraud. Here is a list:

• Privacy concerns of releasing individual-level data
• Participation agreements that do not explicitly include the right to share data
• The logistics of archiving the datasets
• The effort is not worth their time
• The potential harm of false positives
• Ethical concerns related to excluding studies, e.g. wasting the participants' time

Efforts to mandate data sharing have so far been thwarted.

***

Sad to say, the proportion of bad or fake RCTs is substantially higher than 25%. How do I know?

One expert examined over 500 RCTs, and only successfully obtained the individual-level datasets from 150 (30%). In the 30% with detailed data, he found serious problems with 25% of them.

He also looked for problems in the other 70% of the studies, but could only find issues in 1 or 2 percent of them. Paradoxically, this suggests data non-disclosure is positively correlated with trust. But not really. This result points to the fact that typical data summaries in publications hide major problems, including fake data. There isn't much the expert could have done to verify the study findings when the underlying data were not made available. I encountered this issue frequently when looking at Covid-19 vaccine studies.

If we generalize the 25% fraud from studies with data disclosure to those without, then we could conclude that overall 25% of RCTs are suspect. This conclusion assumes that whether the researchers agreed to disclose data is independent of whether their studies are fraudulent. I don't like this assumption. A more sensible assumption is that fraudulent researchers are highly unlikely to disclose their data. Thus, among the studies that don't disclose their data, we should expect a higher proportion of fraud.

Comments

[Dale Lehman]

Can you provide more detail on the author that examined the 500 RCTs? I find it hard to believe that someone attempted to get the data from 500 different RCTs - when I try to get such data, each attempt is rather time consuming. Further, I find it hard to believe that they successfully got data from 150 of these RCTs. My success rate is much closer to 0% than to 30% - in fact, it is quite rare to get the data. So, I have doubts about this - and, more importantly, am especially interested in finding out more about this particular study.

[Kaiser]

DL: The researchers were interviewed for the Nature article (the link is above). This is the 500-study paper:

Carlisle, "False individual patient data and zombie randomized controlled trials submitted to Anaesthesia" (link)