The worst kept secret of the pandemic year is Pfizer's vaccine efficacy. The second worst kept secret is... the Moderna vaccine, of course. Today, the FDA committee will recommend approval of this vaccine for emergency use as expected.
The general outline of Moderna's application is the same as Pfizer's so I'm more or less repeating my notes from the previous post about Pfizer's vaccine.
- The vaccine efficacy up to mid November (with about 50% of the trial participants having been observed for two months or longer) is above 90% in the trial population. This does not mean the VE is 90%; DJ Lemahieu of the Yankees batted .364 in 2020, what's the chance that he bats .364 or higher in 2021? Nevertheless, the VE even if it is 80% is better than good. Note again that the FDA's success metric is VE > 30%. The classical statistical analysis (outlined here) addresses this narrow question. When the FDA determines that VE > 30%, it is not simultaneously endorsing that VE = 94.5%, which is the number constantly paraded in Moderna's press releases.
- The duration of protection is unknown, and there isn't even full evidence for two months yet. The rush to catch up with Pfizer (and front-run the holiday) meant that today's FDA decision is based on an early analysis in which only 25 percent of the participants have passed the two-month-after-the-second-dose requirement. The maneuver to get around this rule is the submission of a supplement, two weeks later, of the full analysis that meets the two-month requirement. The FDA scientists have not had a chance to review the supplement in detail so today's decision is based on (a) the previous analysis and (b) Moderna's submission that the supplementary analysis is not materially different from the previous analysis.
- Any subgroup analysis is silly. The VE analysis only counts five cases on the vaccine arm so any subdivision by race, age group, severity of cases, etc. creates many subgroups in which the vaccine arm has not yet registered a single case. Be careful about people claiming "directional" evidence of these subgroups; such claims are bogus with these sample sizes.
- Moderna defines cases in an even stricter way than Pfizer in terms of symptoms. In addition, Moderna only starts counting 14 days after the second dose, which is given one week later than Pfizer's second dose. So the starting day of counting for Moderna is 42 days after the first dose while for Pfizer, it's 28 days after the first dose. In both vaccine trials, the gap between the vaccines and the placebos emerges after two weeks (some people will get infected after the first dose and before the second dose).
- There is also noise about a single-dose treatment. Unless it is assumed that the second dose is useless, I don't see how that conclusion can be acceptable. See this post.
***
That's not much more to say beyond those expected findings. I want to turn now to register my frustration with how the top-line VE number is being exaggerated for unscientific motives.
Recall that Pfizer has been yelling 95 percent while Moderna replies with "94.5%". The purpose is to have us round 94.5% up to 95% so that we think they are equally effective.
Why didn't Moderna just round up 94.5% themselves to 95%?
Turns out they can't! The 5 cases in the vaccine arm and 90 cases in the placebo arm lead to a VE of 94.46%. They then round this to one decimal place to 94.5%. But if they drop another decimal, they have to round it to 94%.
Moderna was clinging on to that second decimal for dear life. I realized this when on p. 23 of the briefing document, it is disclosed that "there were an additional 18 COVID-19 cases ... but were not able to be adjudicated in time for the interim analysis. Of these.., one was in the vaccine group, and 17 were in the placebo group." Let's assume these cases passed adjudication. So now, we have 6 cases in the vaccine arm vs 107 in the placebo arm. What's the VE? It's 94.41%. With the second decimal below 5, they would have to report 94%.
Don't forget that all of these calculations is for the earlier analysis in which only 25% of the participants have crossed two months of observation. In the supplement, Moderna reported 11 cases in the vaccine arm versus 185 cases in the placebo arm. What's the VE now? 94.1% so even further from 95%.
This bit of decimal mischief unnecessarily sullies the otherwise highly impressive conduct of these vaccine developers. The margins of error around the VE number is in the order of 10 percent so gaming the second decimal point is entirely pointless.
***
One issue I hope the FDA will address is the rapid jump in cases between the first and second analysis. Only two more weeks have passed between the two analyses. Yet, the number of cases doubled (95 to 196). It's hard to know why without the underlying data. Are these new cases happening on the front or the back part of the time line?
Comments