The worst kept secret will be revealed today

Today's FDA meeting will reveal the worst kept secret ever - that the committee will approve the Pfizer vaccine for Emergency Use Authorization (EUA). Given the severity of the pandemic, the urgency of relief, and the promising initial data from the vaccine trial, there really isn't any other possibility!  

Pfizer has submitted a Briefing Document to the FDA to secure EUA, and I'm highlighting parts of this briefing that should interest you.

 

Top-line Conclusion: Trial results are good, no need to embellish

 

Pfizer has been selling this conclusion: the vaccinated participants had a case rate (not the same as infection rate) that is 95 percent lower than the placebo (unvaccinated) participants. 

However, focusing on this selling point is likely to harm the reputation of the vaccine because real-world experience will almost surely refute that. As I pointed out in this post, the PR derby by Pfizer, Moderna and others have moved our expectation from 50 percent VE to 90 percent VE. If such a level of VE is achieved, then people would be right to expect the pandemic to disappear in short order - but that simply isn't the belief of most public health experts.

There are two notable takeaways from the document that I bet you won't hear from the mainstream press:

a) For FDA approval, "The criterion for success was met if the posterior probability that true vaccine efficacy >30% conditioning on the available data was >99.5% at the final analysis." It doesn't matter whether the VE in the trial is 50% or 70% or 90%, they will all be approved based on the 30% threshold. (See yesterday's post to learn why Pfizer's trial performance would have to exceed 50% in order to prove VE > 30%.)

b) We keep hearing that a total of 170 cases were detected during this trial, split 8 cases in the vaccine arm and 162 in the placebo arm. These are cases that were detected within the analysis window. The start of the analysis window is not the day of enrollment or the day in which the treatment began (i.e. the first shot). The analysis window starts seven days after the second shot and ends at the time of analysis (which was November 14). The briefing disclosed that 325 cases, not 170, were documented during the trial. The other half occurred after the first dose and before seven days after the second dose, and are excluded from the headline analysis.

As I previously pointed out (link), this case definition is impractical. Someone who got sick after the first dose before getting the second dose is a failed vaccination. Pfizer submitted a secondary analysis that included all 325 cases, and in this calculation, the VE declines to 82%. There is a margin of error around these VE estimates; the lower bound is 76%. This is still a good result, and I trust this number more than 95%.

The difference between the two analyses (with and without the early cases) tells you that the vaccine wasn't performing anywhere close to 90% efficacy after one dose. During the 28 days after the first dose, and before Pfizer starts officially counting cases, the vaccinated participants have roughly half the chance of getting sick as the placebo participants. Remember that 50% VE in the trial will translate to VE = 30% in general so barely meeting the minimum FDA standard (explained here).

The 50% VE has led to some ill-informed commentary (all over the news channels) that a one-dose regimen should be approved. I address this folly next.

 

Open Issue 1: Should a one-dose treatment with 50% efficacy be approved?

 

This seems to be one of the talking points in the media, most of whom appear not to have read the briefing document, or if they did, they decided they know more than the scientists at Pfizer. Because the Pfizer briefing literally states this:

The efficacy observed after Dose 1 and before Dose 2, from a post-hoc analysis, cannot support a conclusion on the efficacy of a single dose of the vaccine, because the time of observation is limited by the fact that most participants received a second dose after three weeks. The trial did not have a single-dose arm to make an adequate comparison.

There are several problems with concluding that rolling out a one-dose treatment would result in cutting case rates by half. First, as explained in yesterday's post (link), reaching 50% efficacy in a specific trial only shows VE = 30% in general. So at best, the single-dose treatment can be predicted to cut case rates by 30%, not by half.

Second, we have no idea whether the effect of the first shot holds beyond the first three or four weeks because everyone who isn't already sick and hasn't dropped out of the study by day 21 was given the second shot. We are not sure that the top-line VE of 95% holds beyond two months since less than half the participants have reached that point; similarly, we cannot be sure about the single-dose VE beyond four weeks. Because of the trial design, we can answer the former question after more months of follow-up but we will not receive any further data to adjudicate the latter question.

Anyone arguing for a one-dose treatment is making the assumption that there will be no degradation in protection after the first month, without any data to support it. Note that Pfizer researchers elected to test two doses after a Phase 1 trial so a priori, they believe two shots are necessary. In my view, the one-dose treatment should be rejected. Pfizer should design a new trial to test the one-dose treatment against the two-dose control.

 

Open Issue 2: Should those who received placebo be given the vaccine?

 

Pfizer's position is clear: "The Sponsor plans to offer vaccination to participants > 16 year of age who originally received placebo and who become eligible for receipt of [the vaccine] according to local or national recommendations... these participants will be unblinded upon request and will have the opportunity to receive [the vaccine] as part of the study."

Pfizer does not comment on how this affects subsequent analyses, in particular, how this action affects our ability to understand long-term protection, severe cases, and effects on subgroups. There is no doubt that the proposed plan destroys the randomization in the vaccine trial, making it much harder to interpret the results. Those remaining in the placebo arm will clearly not be the same as those who got the vaccine.

But there are some practical and ethical issues that come into play. Is it possible to stop people from seeking the vaccine on their own (and these people will be violating the terms of enrollment)? Can Pfizer ethically turn away such requests?

Further, what is the domino effect on other ongoing trials? If placebo participants got the Pfizer vaccines, the protocol will drop them from the analyses; the claim is that the trials are blinded but it's only single-blind, and it's pretty clear that those who got vaccines may have mild symptoms of various types so bias can creep into these trials.

The randomized controlled trial is the gold standard for proving the efficacy of any new treatment. The randomization of treatment is a foundation to such trials. So it'll be interesting to see how the committee decides this issue.

 

Secondary Findings are Meaningless

Since the trials were designed to get a top-line result as quickly as possible, they have too few samples to draw any reliable conclusions on any of the following topics:

a) period of protection

b) subgroups, such as age groups, comorbidities, racial groups

c) severe cases

d) rare but potentially fatal adverse events

There were only 9 total cases on the vaccine arm used in the analysis. Any kind of breakdown by subgroup means in many subgroups, there would be zero cases on the vaccine arm. Having observed zero cases on a small sample does not merit the conclusion that the vaccine is 100% protective. Pretty much all the margins of errors reported in the subgroup analyses are laughably huge, such as -159% to 99.5% for Asians. Nothing to see there.

Same thing with respect to protection against severe cases. There were only 4 severe cases used in the analysis (split: 1 vaccine, 3 placebo). The interval estimate for VE is -125% to 96%. The statistic screams you don't have enough data.

The absurdity of these intervals has not deterred commentators who just ignore them, and tell us, for example, that the vaccine efficacy for Asians is 74%. What those intervals tell us is that if we ran the Pfizer trial again and again and again, the observed VE for Asians in any one trial would be a number between -125% and 96% (extreme values are rarer but still possible); what they tell us is that we should not draw conclusions from this observation.

The implication of d) is that we must accept an inherent risk of rolling out a new vaccine. The trial would have to be unacceptably huge to detect rare adverse events. With 20,000 participants on each arm, you'd expect on average one instance of a rare problem with 1 in 20,000 chance of occurring (meaning in many such trials, you'd observe zero such events). If the chance is 1 in 100,000, for example, there is basically no chance of finding it during a trial of this size. We will however learn of these rare events when the vaccine is rolled out to millions so active monitoring is essential.

 

Exclusions are Meaningful

There are two types of exclusions - participants who enrolled in the trial but are not included in the analysis.

The first type is designed exclusions. The protocols have stated all kinds of restrictions. Pfizer initially did not allow HIV-positive people in the trial, for example, and then it did enroll some. The types of people who were excluded upfront may seek the vaccine now, and if so, we know neither the efficacy nor the potential harm on them. [P.S. I pulled down the list of exclusions from the Pfizer protocol. You can check the list here.]

The second type are part of the trial but subsequently got thrown out for any number of reasons. For example, someone drops out. Also, anyone who got sick after the first dose is not given the second dose, and so by definition, did not complete the treatment program. In a per-protocol analysis (which is what the FDA accepts as primary), those people are excluded from the analysis population.

The statistical question is if these people had been included (i.e. if they had not dropped out, if they had not gotten sick before the second dose,...), how many would have gotten sick? For example, by my calculation, 533 people were dropped from the vaccine arm after the first dose while 293 were dropped from the placebo arm, and it appears that the reason for dropping these is "protocol deviations that happen up to 7 days after Dose 2". I'd be interested in learning what those deviations were, and why there were 80% more frequent in the vaccine arm.

 

***

Overall, the data present a good story and so I don't doubt that the Pfizer vaccine will be approved.

 

P.S. [12/11/2020: While working on the next blog post, I realize another reason why the subgroup analyses must be viewed with great skepticism for now, and why we must wait for a longer observation period. Recall that under FDA guidelines, these results include up to half of the participants to have been observed for fewer than 2 months. That condition is applied to the aggregate population, which is roughly 20,000 on each arm. Now, when you drill down to subgroups, the subpopulation sizes are much lower, and therefore you're going to have subgroups in which more than half of the participants have been observed for fewer than 2 months. Pfizer can publish the breakdown by length of observation to allay my concerns. In any case, if we wait longer, then everyone would be observed for 3 months, 6 months, etc. so the concern will eventually go away.]

 

 

Comments

[Michael Droy]

It was only a secret as long as Trump was promising it before year end.
As soon as the election was over, Biden was claiming credit for it.

[Kaiser]

MD: The politicians should join hands and thank the scientists, doctors, statisticians, business managers, etc. who did the hard work!

[just askin]

no comment on the large suspected but unconfirmed group?

[Kaiser]

JA: Is that something new? I thought anyone who reports symptoms are tested and confirmed. I haven't heard of such a group but of course if it exists, we should know about it.