Bending to outside pressure, Moderna has made public one of the documents submitted to the FDA that provide more details on how they are running the vaccine trial. (PDF here) This is a reassuring move, and I applaud them for taking this step.
In a previous post, I raised two key questions I would like to find answers to, and after reading all 135 pages of the protocol, I can tell you the answers.
How is efficacy measured?
Vaccine effectiveness (technically, efficacy) is the reduction in infection rate among the vaccinated relative to the baseline infection rate among the unvaccinated. A 50% effective vaccine is one that halves the baseline infection rate. (For more on vaccine effectiveness, see this other post.)
An important detail is how this infection rate is measured. According to the trial design documentation, Moderna will confirm infection using PCR nasal swab tests (primarily). To be precise, what they measure is confirmed cases rather than infections.
What is the analysis schedule?
Until the documentation is made public, there has been a lot of sloppy reporting about when results will be available. Now we know. The primary analysis will be conducted when the total number of confirmed cases (across both test and placebo arms) exceeds 151. The expected time of this analysis is six and a half months after the last trial participant receives the second dose. By my estimate, this will be mid June 2021.
The trial started enrolling participants at the start of August. Assume that enrollment is complete by the end of September, the second dose of the last participant should occur at the end of October, the data for the primary analysis will be frozen by mid May 2021, and the trial result will be ready by mid June 2021.
Why do pharma CEOs keep saying they expect results by the end of 2020? That's because they are hoping one of two interim analyses might produce results so stellar that the FDA will have seen enough. The first interim analysis is triggered when the number of confirmed cases exceeds 53, which is 35 percent of the full sample. If it takes 6.5 months to accumulate 151 cases, it would be 2.5 months to get to 53. (In both instances, the half month is added because they don't start counting cases until 14 days after the second dose.)
For the first interim analysis, assume they only analyze participants through the end of August, the second doses should be completed by the end of September, the data will be frozen by mid December 2020, and so the earliest date by which the vaccine might be declared effective is mid January 2021, if everything else falls into place.
***
What to Look Out For
I read through the entire 135-page document. I prepared a list of notes about the clinical trial, mostly about statistical matters. These are placeholders for now. How much these issues matter depends on what happens during the trial.
Blinding
The trial is observer-blind, less than the gold standard of double-blind. Blinding is hiding whether a participant has gotten the vaccine (or drug) or the placebo. Double-blind means neither the researchers nor the participants are told who got what shot, until the data are ready for analysis. Blinding prevents humans from misbehaving as a result of knowing the treatment assignment. In this Moderna trial, the person administering the shots is not blinded. There may be a scientific reason for why some researchers cannot be blinded for this study but I didn't find an explanation in the documentation. (I checked the Pfizer-Biontech trial, and it is also observer-blind, not double-blind.)
Many outside scientists and reporters have missed this detail, and mis-reported the Moderna trial as double-blind (e.g. here, here, here).
Analysis Population
The primary analysis is "per-protocol", which is less than the gold standard of intent to treat. This is always a controversial subject so let me explain the disagreement. In a canonical clinical trial, half the participants get the vaccine (or drug), and the other half receive the placebo (or comparator). Researchers take great care to randomize assignment to these two arms. If each participant has equal chance of being assigned to either arm, then we have created the condition of "all else being equal" so that when we compare the average person across the two arms, any difference can be attributed to the vaccine (or drug).
In practice, some participants will drop out of a trial. It is quite possible that the dropout rate is not the same across the two arms. For example, if the treatment causes irritation, more people in the treatment arm might quit. The statistical debate is whether to include the dropouts in the analysis. The gold standard of intent-to-treat analysis says to stick as close to the original population as possible. Dropouts are therefore included (up to the time of dropout). In a per-protocol analysis, the dropouts may be excluded as if they never existed. The proponents of per-protocol argue that the final outcomes of these dropouts are not known, and so including them might be "unfair" to the treatment arm.
By contrast, supporters of intent-to-treat (including me) don't buy that the act of dropping out is independent of treatment, and worry that the loss of randomization invalidates the "all else equal" condition. For me, the most important argument is the practical point of view: the per-protocol findings apply only to people who take the treatment without dropping out!
Dropouts are not the only subset of participants who might be excluded in a per-protocol analysis. It's just easier to explain.
Back to the Moderna trial. It appears that dropouts prior to the second dose will be excluded from the per-protocol analysis but those who discontinue after the second dose will be included.
Infections Before Finishing Treatment
In the Moderna trial, another interesting subset consists of participants who get infected and then are excluded from the per-protocol analysis. In the primary analysis, Moderna starts counting infections 14 days after the participants get their second dose.
The trial design stipulates that people who have confirmed infection before the second dose will be ineligible for the second dose. All such people therefore will not complete the protocol and be excluded from the per-protocol analysis. The good news is that Moderna has committed to performing an intent-to-treat analysis for comparison. However, the final decision will be based on per-protocol, and there is no guarantee that the intent-to-treat analysis will be published for outside evaluation.
Let's say we're behind at the end of a(n American) football match, and our coach decides to go for an on-side kick and then drive for a field goal. To win the game requires executing both steps. If the on-side kick fails, the other team will run out the clock. (Last year, on-side kick success rate in the NFL was only 6 percent.) If the on-side kick fails, the per-protocol analysis excludes this match because the team didn't execute the entire plan. If the on-side kick is successful and the team gets into field-goal position, the per-protocol analysis includes this match. So, the estimate of the success rate of the two-step plan will be over-optimistic.
Here's the practical point of view. If the vaccine requires two doses, and someone gets sick after the first dose, the vaccine is ineffective for that individual. It doesn't matter that the said individual has not received the second dose; it is meaningless to administer the second dose.
Excluded Infections
Then, there is the subset of participants who get infected during the first 14 days after the second dose. For an unexplained reason, Moderna isn't counting infections until two weeks after the second dose.
This subset will be included in the per-protocol analysis because they took the full two-dose program. However, the outcome for these participants will be "censored," meaning instead of being treated as infected, they are treated as if they have dropped out of the study just before their infections are confirmed.
Again, I take a practical view on these matters. If, after the vaccine is approved, someone gets infected a few days after receiving the second dose, the vaccine would have failed for this individual.
So when the results come out, keep an eye on whether there is a differential likelihood to get early infections on the vaccine arm relative to the placebo arm. The good news is Moderna has committed to conducting additional analysis that counts infections from the first day after the second dose for comparison. However, the final decision does not have to take into account this analysis, and it may not be publicly available.
Counting Infections
Measuring infections in a vaccine trial is always challenging because the participants are healthy. Contrast this with a clinical trial testing a treatment on sick people. Since it's unethical to inject virus into healthy subjects, we have to wait for infections to crop up via community spread.
Moderna isn't very proactive in monitoring for infections. They rely on self-reported symptoms, which then trigger confirmatory PCR tests. For example, the only way researchers will learn about suspected infections during the first 14 days after the second dose is if the participants report symptoms. Not everyone who report symptoms will get tested; the researchers have some discretion for certain types of symptoms.
That's why we will never know the true infection rate. The reported rate should be considered the rate of confirmed cases.
***
These are the most salient information I learned from the just-released documentation on the Moderna vaccine trial. I have several more notes that will be written up in my next post.
Comments