Regarding the Pfizer vaccine trial results, statisticians are tempted to say that the signal (the vaccine's share of cases) was so strong that we can let down our myriad defenses. As I explain today, this is not quite true. It depends on which signal we're talking about.

If the question is whether the Pfizer vaccine is at least 50 percent effective, then we don't have much to worry about. Nevertheless, the press release derby has created extremely high expectations - if the question is whether the Pfizer vaccine is at least 90 percent effective, then the finding is highly sensitive to shifts of just a few cases. I'll show below that we can be secure in making a statement such as that the Pfizer vaccine is at least 80 percent effective.

***

The basic strategy of a Bayesian analysis yields a probability estimate of the efficacy of the vaccine being tested (VE), given the results from the vaccine trial. The outcome is a probability estimate for any value of VE. According to the Pfizer protocol, the U.S. regulators are interested in a specific estimate: the chance that VE is higher than 30 percent.

The result from the vaccine trial is expressed as the vaccine's share of cases (VSC), which is the proportion of detected cases that came from the vaccine arm of the trial (the other arm being the placebo). Intuitively, the better is the vaccine, the lower its share of cases.

Enrollment in the Pfizer trial can be stopped when the trial records 170 cases. The most recent press release from Pfizer reported that out of those 170 cases, only 8 came from the vaccine arm, thus the observed VSC was 8/170 = 5 percent. This translates to 95% VE: in other words, the case rate in the vaccine arm was merely 5% that in the placebo arm.

(Notice that the average case rate is very low: 170/43000 = 0.4% about 2 months since enrollment, therefore it is wrong to say that 95 percent of vaccinated people will be protected - most people in the trial have not yet been exposed to the virus!)

If any of the above is unclear, please review my prior posts about the Pfizer analysis (here and here).

***

I was wondering how sensitive the result is to the observed VSC. If the trial had found 9 or 10 cases, instead of 8, among the vaccine arm, how much would our conclusion change?

I use the chart below to answer this question:

First, look at the red line. This shows the probability that vaccine efficacy (VE) is over 90 percent, given the trial results. The actual trial result is indicated by the down arrow - eight cases in the vaccine arm out of 170 total cases. The corresponding dot above says there is a 98% chance that VE > 90%. That's what the Pfizer press release wants to tell us.

Now, notice how steeply the line drops as we move from 8 to 10 to 12 cases: 98% -> 93% -> 81%. To achieve the typical standard of 95% confidence, this probability has to clear 97.5%. This means that the 8 reported cases were sitting on the edge. One more case, and the claim of over 90% VE is shaken.

This might sound like bad news for Pfizer. If it does, it's Pfizer's own doing. Because of the press release derby, it feels like VE needs to be above 90%. Remember those days when we hoped VE is at least 50 percent?

So I did the same analysis for the probability that VE is over 50 percent, as the number of cases in the vaccine arm increases. This is shown as blue dots. Notice that these dots hug the 100% line for dear life. Even if the vaccine arm found 25 cases (out of 170), we can still say with complete confidence that the VE of Pfizer's vaccine is over 50 percent.

Let's double the number of cases coming from vaccinated participants to 16. If you draw a vertical line at 16, it will hit the light pink dots at 99 percent. Those dots represent VE greater than 80 percent. We are highly confident that VE is over 80 percent, even if the vaccinated participants suffered twice as many cases as observed in the vaccine trial.

***

When discussing these vaccine trial data, please bear in mind the following:

1) Recall that the 40,000 or so participants are not monitored every day. The cases are self-reported, and the participants who report symptoms are then tested. At most one mandatory follow-up of the entire set of participants has occurred so far.

2) The claim of 90% or above efficacy is delicate. Shifts of one or a few cases matter. Bear in mind that a small number of participants may be excluded from the analysis. These can have completely legitimate reasons; notstanding, such exclusions can swing the outcome. That said, claiming about 80% efficacy shuold be very solid.

## Comments

You can follow this conversation by subscribing to the comment feed for this post.