Another Covid19 victim is science practice

[Yes, that's a mango. It's not a random picture. Keep reading.]

Gilead's remdesivir has grabbed a few headlines lately. The company just issued a press release saying a new analysis showed that the drug saves lives. This might surprise some readers - hasn't the drug been approved already? Does this news mean Gilead won approval without showing remdesivir saves lives?

In this pandemic world, lots of things have turned upside down, and the practice of science has become a casualty as well. This post focuses on the "emergency use" practices that have creeped into science.

The Gold Standard of Science Loses Luster

Getting immediately to the question I raised above. Indeed, remdesivir was approved even though the clinical trial did not show a statistically significant effect on mortality when compared to a placebo. The approval was based on an improvement on a different metric, the so-called "recovery ratio". Patients are graded on an 8-point scale, with a cutoff level below which they are considered "recovered". Thus, at the end of the analysis period, each patient is classified as either dead, recovered, or still in treatment; a certain proportion of patients have "recovered".

The press release from Gilead cites yet another metric, median recovery time of 15 days for recovered patients on placebo versus 11 days for those on remdesivir. Probably this metric is considered fit for public consumption. This result is not very exciting, when one considers the just-announced price point of over US$3,000 per patient. (Actually, in the clinical trial, patients received a 10-dose treatment, which maps to a price point of US$5,800. While the 5-dose treatment is priced lower, it's not clear to me that the clinical trial result would hold using the lower dosage.)

Against this background, the news that remdesivir did save lives after all is newsworthy. The casualty I alluded to earlier is of science practice. This new conclusion isn't based on a new clinical trial but a "deeper dive" analysis of the previous data. Lots of data analysts are asked to do "deep-dives," which frequently amounts to a salvage operation. I'm reminded of the lemon of a mango I just sliced open (it did taste like a lemon). Parts of the flesh have already turned black, I should have thrown the whole fruit out but I carefully carved out the little orange bits, and savored them.

Of course, Gilead already compared mortality rates of treatment versus placebo groups in the preliminary analysis and that did not reach statistical significance so where's the magic?

We don't have much information. In a press release, Gilead described the new analysis as a "Comparative Analysis of Clinical Recovery and Mortality Outcomes from the Phase 3 SIMPLE Trial Versus Real-World Cohort of Severe COVID-19 Patients Receiving Standard of Care." In other words, they abandoned the clinical trial - the gold standard of medical statistics, and adopted a methodology for observational studies, usually resorted to when we don't have the gold standard.

Effectively, Gilead ditched the placebo group in the trial and replaced it with a self-selected placebo group, constructed by finding patients not in the trial, did not receive remdesivir and match the characteristics of the patients in the trial. This is comedic, actually, the kind of thing that comes with a serious health warning in every statistics class. Why? Because the placebo group in the trial already met all those criteria. The placebo patients did not receive remdesivir, and matched characteristics of those patients who received remdesivir (by design via randomization). These patients were just as "real world" as those others. In addition, the randomly assigned placebo group does not suffer from unknown selection bias, neither by patients nor by analysts.

Don't get me wrong - I love matching methodologies as a principled, explainable way to analyze observational data. But I can't think of a reason to use a matched cohort to overturn the conclusion of a clinical trial. Just as we're approving drugs for "emergency use", lowering the usual standards, we're practicing emergency-use science.

Press Releases Topple Peer Reviews in Scientific Communications

The highly unorthodox analysis was announced to the science community in an equally unorthodox method. A press release, with barely any details. Science in the pandemic era has been communicated as a series of leaks and press releases. The previous result was leaked to the press by Dr. Fauci, whose organization is a sponsor of the remdesivir study, about a month before the scientific study was published in the New England Journal of Medicine. The journal dropped the Gilead study Friday evening on Memorial Day weekend in the U.S.

Notably, the published study described the treatment as "Remdesivir was administered intravenously as a 200-mg loading dose on day 1, followed by a 100-mg maintenance dose administered daily on days 2 through 10 or until hospital discharge or death." This corresponds to a 10-dose treatment. If this is the full disclosure of how the trial was conducted, then the patients who received 5 doses were those who recovered early.

A major problem with the specific press release on a matching analysis is that nobody can evaluate the work. Matching is a highly subjective exercise that involves a lot of assumptions. We don't even know what the entire set of variables patients were matched on. So, the situation is similar to New York State antibody testing, for which we still have zero information about how it was conducted.

After selected numbers and headline findings are leaked, the media has no ability to fact check these results and no experts can comment on them without seeing the study. This creates a vaccum in which the results are propagated and unchallenged. When the study is published, it feels like old news but in fact, the results have not previously been vetted by outsiders. Until this day, lots of reporters, and also other scientists are citing Cuomo's numbers even though no data from the antibody testing program has been published anywhere by the scientists running the program.

If the clinical trial has rock-solid results, there is no need to play the PR game. Just get the study through peer review and out in NEJM as soon as possible, and let the world endorse it on its merits.

Cherry-picking Become the New Pastime

Prior to the leak of the positive result eventually published by NEJM, the media reported negative results from another study of remdesivir, from Wuhan, China. That study is also a gold-standard, randomized controlled trial. It has now been conveniently forgotten.

Every result adds to the collection of knowledge we have about the drug. It would be sad if the new "comparative analysis" result grabs the headline while the unsatisfactory result from the clinical trial gets the boot.

[P.S.

I had the pleasure to be a guest on the CSAI Podcast, hosted by John Reid and Rusen Aktas. We literally chatted for many hours about Covid-19 models, data science, science communications, uncertainty, etc. The best bits are found here:

Apple: podcasts.apple.com/gb/podcast/csa

Spotify: open.spotify.com/episode/2Ejv3y

Anchor fm: anchor.fm/csai-podcast/e

]