I spent a good chunk of my career designing, running and analyzing industrial experiments, and as policymakers stray further away from the underlying science, I have a bad feeling about the rollout of the Covid-19 vaccines
At issue is a push to deviate from the treatment protocol that produced the highly promising clinical trial results. During the clinical trial (by Pfizer), the vaccinated participants were given one dose of the vaccine on day 1, and a second dose on day 21, the vaccine was then given one week to take effect, so that the 95% efficacy was computed by counting cases from day 28 (i.e. nullifying cases between day 1 and 28). Moderna and AstraZeneca had a similar treatment protocol, all specifying two doses, but the number of days separating the doses, and the waiting period for the effect to take hold depended on the protocol.
When the outcome of a carefully-designed experiment returns positively, the straightforward next step is to roll out the treatment "per protocol". Any deviation from the protocol requires assumptions (coming from intuition, experience, and gut feelings) modifying the science. The point of running a randomized controlled trial is to follow the science rather than one's guts.
***
The specific alterations to the treatment protocol are as follows:
- the elimination of the second dose
- extending the timing of the second dose (if a supply shortage were to materialize, the delay of the second dose may become infinite, folding this into the first item)
- the size of the first dose (in the case of the AstraZeneca trial, they had initially trumpeted a half-dose followed by full-dose treatment, which has been quietly forgotten after the U.K. government decided to authorize a one-dose treatment. Does anyone know if the single dose is a half dose or a full dose?)
- mixing and matching different vaccines (such as Moderna and Pfizer)
- the success metric now based on (very small?) immunological samples and not based on cases confirmed by PCR testing
So far, the U.K. is the only country that adopts most of these as official policy while the U.S. and others currently claim they allow them for "exceptional cases". The U.S. announced that second doses are no longer reserved for those who have taken their first shots, increasing the chance that people would not receive their second doses at the prescribed time, if ever.
***
A fundamental best practice of running statistical experiments on random samples of a population is that once the winning formula is rolled out to the entire population, the scientists should look at the real-world data and confirm that the experimental results hold.
This post-market validation is hard even if properly done. That's because on rollout, everyone is eligible for the treatment, and those who have received the vaccine up to the time of analysis do not form a random sample of the entire population. So any difference between the vaccinated and unvaccinated groups may not be a pure effect of vaccination. (This difference is why we conduct randomzied controlled trials, which allow scientists to isolate causes.)
The action of the U.K. government (and others who may follow suit) has severely hampered any post-market validation. It is almost impossible to compare real-world evidence with the experimental result, because most people are not even getting the scientifically-proven treatment per protocol!
Even if the rollout was perfect, the real-world outcomes are likely to deviate from the experimental findings. Statistical science gives us confidence that any such deviation is immaterial.
With the altered protocol, the real-world data come from a variety of dosing schedules - some with one dose, some with two, some with two doses 21 days apart, some with two doses 10 weeks apart, etc.
What happens when the aggregate real-world outcome falls below expectation? How will scientists figure out what are the reasons for the under-performance?
First, the scientists can't even say what the expected performance is because the experiment yielded data on just one dosing schedule, which could be the least used in the rollout.
Second, there may be several simultaneous changes to the dosing schedule. For example, one might be comparing someone with 1 dose of Pfizer on day 1 with another who got 1 dose of Pfizer on day 1 and 1 dose of Moderna on day 36. Is the difference of outcomes due to mixing and matching, or number of doses, or the gap between doses? How do we learn which factors are contributors, which are not, and the relative importance of these factors?
Third, many post-market validation studies are doomed by focusing exclusively on factors that the investigators can control, such as the factors listed above. It's possible that the policymakers are correct - that none of the dosing changes would affect the vaccine efficacy. In that case, an observed difference in outcomes is due to other factors, such as viral escape by new variants, as-yet unclear duration of protection, compliance to mitigation measures after vaccination, which people choose to take the vaccine, etc. Notably, these are factors outside the control of scientists.
Why do we need answers for those questions? If the rollout does not meet expectations, we need to correct the course. To rectify the problems, we need to know what they are. If we can't even diagnose the potential problems, as will be the case here, we will be swinging in the dark.
***
Actions have consequences. The decision to deviate from the treatment protocols that delivered the promising vaccine trial results makes it very challenging, if not impossible, to measure the impact of these vaccines. One of the key lessons of managing this pandemic so far is that good data drive good decisions, and bad data drive bad decisions. Unfortunately, policymakers have signed up for bad data, so no one should be surprised if future policies turn out badly.
P.S. On Twitter, some are offering this UK government document as evidence that the scientists have a plan for post-market validation. I recommend reading this document to learn about what appropriate validation steps would have been in normal circumstances when the rollout of the treatment is per protocol. In the vaccine efficacy section, it addresses unanswered questions from the trials such as the duration of protection, subgroup analyses, other outcome metrics, etc. I didn't find anything in that document that estimates how deviations from the treatment protocol affect outcomes.
Comments