This is part 2 of my review of the FDA briefing document about the Pfizer coronavirus vaccine Phase 3 trial. Part 1 was published here.
Transmissions vs. Symptoms
You may have heard that the Pfizer vaccine (and any of the other ones that are being studied) does not necessarily stop transmission of the coronavirus from person to person. So what is the point of these vaccine trials?
The answer is in the success metric - comparing the case rates of the vaccine group relative to the control group. About 1 percent of the placebo group have been confirmed infected by mid November versus 0.05 percent of the vaccine group. Pfizer does not know how many people were infected but displayed no or mild symptoms.
This metric is chosen in part because of ethics. It would have been easier to track asymptomatic people and transmission paths if trial participants were locked up in a closed environment deliberately infected with the virus. But most scientists revolt at the thought of such an experiment.
Nevertheless, if fewer people are getting sick, fewer will succumb to Covid-19. So it's still a net benefit. Public health officials continue to recommend all mitigation measures to minimize any unnecessary deaths.
Vaccine uptake
I know some people are worried about vaccine uptake. I'm not that worried because of expected word of mouth. If this vaccine actually works as it did during the trial, then within a few weeks and by two months after the first vaccinations, people are going to notice the drop in cases, and the word will spread quickly. Of course, the flip side of the coin also holds: if cases don't start declining, Pfizer has a problem!
TV stunts
Please don't ask government officials to take the shots on TV. It convinces no one. People who believes in vaccine science don't need reassurance, and the anti-vaxx segment will assume it's a setup anyway. TV stunts juice up TV ratings, and that's it.
Unexpected Sightings
Here are a few things I read in the document that surprised me.
I already gave you one in the last post, which is that there were 325 total cases detected among the Pfizer trial participants, not 170 cases in the headline number. The gap is because Pfizer's result discounts anyone who got sick prior to seven days after their second dose. As I described in that post, after including those cases, the overall efficacy achieved during the trial was still a robust 82%.
Secondly, there were six deaths so far among trial participants, including two on the vaccine arm. Evidently, Pfizer did a better job of not drawing attention to deaths than AstraZeneca (link). Of course, deaths may not be caused by the vaccine.
Finally, Pfizer speculates that there may be no immunity coming from prior infection. This is their interpretation of a surprising observation from the trial itself. In one of the analyses of vaccine efficacy, Pfizer excludes people who showed evidence of a prior Covid-19 infection (e.g. via antibody testing) from calculating the case rates. In the placebo arm, there were 670 exclusions, nine of whom were subsequently confirmed infections during the trial (1.3 percent). This 1.3 percent is the same rate as for the remaining placebo arm of people who were not previously infected.
It is widely assumed before that the reinfection rate is zero or close to zero. Another possible explanation is problems with antibody testing; maybe the prior infections were misclassified. That's why one should not exclude data based on assumptions. The analysis that does not exclude the prior infected turns out to be more credible here.
***
The rush to a decision leaves a number of questions unanswered. Here are a few issues I'd be tracking going forward.
Potentially Relevant Factor 1: Duration of observation
Because of the rush, the vaccine developers are conducting analyses when only half the participants have reached two months after treatment (second dose). That was a stipulation by the FDA, discussed here. Many statisticians would have preferred the criterion to be all participants reaching two months. But that was not to be.
The briefing document provides more details on the distribution of follow-up time. Pfizer admitted that they have not quite met the FDA requirement. As of November 14, when the data were frozen for analysis, 44 percent of the participants reached two months of follow-up (below the 50% requirement but we'll let that pass). Thirty-six percent were in the second month of follow-up while 20% were in the first month of follow-up (after the second shot).
Why do statisticians want all participants to reach two months? That's because we want to draw the conclusion that the Pfizer vaccine yields protection for at least two months. The problem with requiring only the median participant to reach two months is that we can conclude only that the median protection is two months, that half the vaccinated people may have protection that lasts fewer than two months.
Next time you hear some talking head say we have proven efficacy for two months, remember that is not quite true. As noted above, 1 in 5 participants have not even reached one month of follow-up, and 1 in 2 participants have not reached two full months of follow-up.
The rule being applied to the median participant rather than all participants also damages any subgroup analysis. There were about 2,000 blacks in the vaccine arm. Does the median black participant have at least two months of follow-up? We do not know because that information has not been released.
If you ask any statisticians who design experiments, we'll tell you decisions have consequences.
Potentially Relevant Factor 2: Site-level differences
The trial is being conducted in the U.S. (77% of enrollees), Argentina (15%), Brazil (2%) and South Africa (2%). Nowhere in the briefing document does it break down the data by site. For example, I am curious where the cases were detected - all in the U.S. or spread out across countries? Also, they mislabel blacks as "African Americans," which implies there were no black participants in South Africa or Brazil or Argentina; strangely, Asians are described as Asians, not Asian Americans, when it appears that most if not all Asians would be Asian Americans.
Potentially Relevant Factor 3: Confirmation of cases
Most cases are self-reported by participants to Pfizer, who then confirms the infections. While the PCR test is the preferred method, the protocol allows for "quick tests" to be used in cases where PCR testing could not be performed. I'd like to know what percent of the cases were confirmed by PCR.
Further, Pfizer disclosed that a case is a case only if (a) the participant reports at least one of 9 symptoms and (b) a test confirms the infection within "4 days of the symptomatic period". I'm not a doctor so why the positive test must happen in those 4 days. Ordinarily, these little details are not important but for a vaccine trial in which we are talking about dozens or fewer cases, even shifts of 3 or 5 cases may matter.
Not Forgetting
Finally, it's useful to list some design decisions that were made to accelerate the trials. Any of these can have unintended consequences - if not, we would have done away with these practices in ordinary trials.
- Extra-short period of observation
- Single blind, not double blind
- Per protocol analysis, noting that in many of these trials, two doses are required weeks apart
- Exclusions both planned and unplanned
- Surveillance is not continuous, relatively infrequent, so that most cases are self-reported
- Not measuring asymptomatic transmission
- Not measuring the level of exposure to the virus (if someone hasn't been exposed to the virus, s/he won't get sick whether vaccinated or not.)
This is not to say these are bad decisions. It's that some risks are knowingly taken, and if anything doesn't meet expectations, these factors might be the reason.
Comments