I previously described how Pfizer told the FDA they're analyzing their trial data, using Bayesian analysis (link). In this post, I describe how classical statisticians would have analyzed the same data.

The beauty of having simultaneous vaccine trials is our ability to compare and contrast the trial designs and outcomes. **Even though Pfizer and Moderna are testing vaccines using similar messenger RNA technologies, their trial designs have notable differences.** For example, Pfizer counts cases from the first day after the second dose for seven days while Moderna counts cases starting 14 days after the second dose for 14 days. (Remarkably, those two observation windows are disjoint; it would be interesting to hear why each company's researchers made those choices. I have notes on the Moderna protocol here. This timing difference explains why Pfizer was first out of the gate.)

Another major difference is the analysis method. While Pfizer uses a Bayesian analysis, Moderna's statisticians are following the traditional method (sometimes called classical or frequentist).

***

From the previous post, you learn that the Bayesian's answer is a probability curve that gives the probability that the vaccine efficacy (VE) takes on some specific range of values, given what was observed during the vaccine trial. I reproduce the key graph here:

In particular, the FDA is expected to look for the chance that VE is at least 30 percent, i.e., the proportion of the area under the probability curve to the right of the 30% mark. Given the hugely optimistic trial data, the chance is essentially 100% as shown above.

***

Classicists (i.e. Frequentists) start from a different place. They believe the efficacy of a given vaccine has a fixed value, but it is unknown, and is estimated using the data from a randomized controlled trial. The last released result from the Pfizer trial is that 8 of 170 detected COVID-19 cases have occurred in vaccinated participants. This result is reported in two ways: (a) the vaccine share of cases (VSC) is 5% (= 8/170) and (b) the vaccine efficacy is 95% (note, again, this means that vaccination reduces the case rate by 95 percent. It does not mean 95 percent of vaccinated people will be protected from infection.)

The simplest analysis is to take the result from the trial and run with it. This means we are generalizing data from 43,000 people to all 330 million Americans (I'm ignoring the fact that some participants in the trial are outside the U.S.). Statisticians believe this simple analysis is too simple.

Imagine parallel universes in which we would roll out the same Pfizer trial. In each universe, a different set of 43,000 people would participate in the trial, and produce a different number of cases in the vaccine arm - probably leading to a VE close *but not identical* to 95%. We call this **variability due to sampling**.

The question then is if we could run these parallel universes, and count the number of cases in the vaccine arm in each, what would the distribution look like?

The shape of this distribution depends on how effective the vaccine is. So a chart like the above must assume a value of VE. The natural assumption is VE = 30% which is the minimum standard to get FDA approval. That's what I assumed when making the above chart.

The featured distribution addresses a very particular question: if the vaccine has 30 percent efficacy, how many COVID-19 cases do we expect to find among the vaccinated? The most likely outcome is 67 cases in the vaccine arm, and this outcome is expected in 6 percent of the parallel universes. But anything is possible: in some scenarios, we would find only 1 case while in others, there might be 140 cases.

Notice how classicists adopt a precise *frequentist* definition of "chance" as the proportion of all parallel universes. Bayesians interpret "chance" as a subjective probability.

***

Another difference is that classicists provide a precise answer to a limited question: do we have sufficient evidence to show that the vaccine is at least 30% efficacious? Yes or No. Bayesian analysis, by contrast, has the flexibility of answering many different questions, with each answer being a probability. What's the probability that VE > 30%? or VE > 80%?

Classical analysis establishes a cutoff of 54 cases. If there were over 54 cases in the vaccine arm during the actual vaccine trial, then the data are consistent with the vaccine's efficiacy being 30 percent. In this case, the trial fails to show that VE is higher than 30%. If there were 54 or fewer cases in the vaccine arm, then the data are inconsistent with VE = 30%. In this case, we conclude that VE > 30%.

In Pfizer's trial, they found only 8 cases among the vaccinated population, so according to the chart above, the classical analysis comes to the same conclusion as the Bayesian analysis - that VE > 30% is more or less certain.

***

How come we keep hearing that the FDA requires vaccine efficacy of 50% while I keep saying it's 30% (according to the design protocols)?

In the above classical analysis, the cutoff is 54 cases in the vaccine arm - out of 164 total cases. So to show VE > 30%, the vaccine's share of cases (VSC) during the trial must be below 54/164 = 33 percent. This value of VSC can be translated back to VE: VSC of 33 percent is equal to VE of 50 percent. In other words, if the VE is 50 percent (or better) as measured in the Pfizer trial, then we have proven that VE > 30 percent in general.

Note that the first VE number (50 percent) is computed from one sample of 43,000 people, and the second VE number (30 percent) refers to (the average of) all parallel universes. If the first number is 45 percent or 40 percent or 35 percent, statisticians are not comfortable that the second number is over 30 percent. This is because of variability from sample to sample.

To reiterate, the FDA minimum requirement is not 50 percent but 30 percent. However, in order to prove that the vaccine has 30 percent efficacy (over the entire population, not just in a single sample), the vaccine must show 50 percent or higher efficacy during the vaccine trial.

This post about frequentist analysis is inspired by the blog post titled "A Vaccine Trial from A to Z" by Stephen Senn, contributed to Deborah Mayo's blog.

***

Underlying all of this is the logic of **hypothesis testing**. This is a topic I covered in more detail in **Numbers Rule Your World** in Chapter 5. Get your copy of the book here.

P.S. For those who wonder about deriving the cutoff of 54 cases in the vaccine arm: the Pfizer protocol states that 164 total cases triggers the full analysis. Showing VE > 30% is the same as showing VSC < 41.2%. During the trial, out of 164 total cases, how many will appear in the vaccine arm, given that there is 41.2% chance of each case being in the vaccine arm? That's a binomial distirbution with N = 164 and p = 41.2%. The cutoff is then based on a tail probability of 2.5%. This means if VE = 30%, there is still a 2.5% chance that 54 or fewer cases appear in the vaccine arm. According to our decision rule, we would have concluded that VE > 30%. This means we are allowing a 2.5% chance of falsely concluding that VE > 30%.

## Comments

You can follow this conversation by subscribing to the comment feed for this post.