While we eagerly await the scientific report on the coronavirus vaccine trial from Pfizer/BioNTech, some of us in the blogosphere have checked out the Pfizer protocol in which they have pre-registered the method of analysis. This post is intended for those who are curious how the data from such trials are analyzed.
In a vaccine trial, the key metric is vaccine efficacy (VE). Pfizer previewed the "interim" analysis in a press release in which they claimed their vaccine has a VE of "above 90 percent". (Regardless of the optimism of this announcement, science by press release is an undesirable development. Such pre-announcements with few details prevents experts from offering any critique - the only reasonable response is to accept the headline tentatively, allowing it to take root.)
VE of 90% means the vaccine cut infections by 90 percent. It does not mean 90 percent of those who are vaccinated will not get sick (link). That's because the coronavirus isn't spreading as quickly as you might presume. Inferring from Pfizer's press release, I think about 1 percent of the unvaccinated participants got infected within 30 days. So cutting infections by 90 percent means the case rate in the vaccinated group is 0.1 percent. Throughout this discussion, I refer to cases instead of infections because it's not clear from the protocol that Pfizer will detect every infection.
Since the participants are equally split between the vaccine and placebo groups, the average case rate is (1+0.1)%/2 = 0.55%. Out of every 10,000 participants, 55 have gotten sick. There were 10 times fewer sick people in the vaccine group than the placebo group. So 5 out of 55 are from the vaccine group, and 50 out of 55 are from the placebo group. The vaccine group accounts for 5/55 = 1/11 = 9 percent of the total cases. I'm going to call this the Vaccine Share of Cases (VSC) metric.
The Pfizer analysis does not work directly with the VE metric. It focuses on the VSC metric. (For those who've read the protocol, VSC is the entity known as theta.) Notice that the higher the VE, the more powerful is the vaccine, the smaller is the VSC. Those two metrics are inversely related. I did the calculation above to show you that any value of VE corresponds to some other value of VSC. Maximizing VE is the same as minimizing VSC. (The analytical reason for using VSC instead of VE will be explained below.)
The following chart shows the inverse relationship between VE and VSC.
The claimed VE of 90% implies that 9% of the cases were traced back to the vaccine group. The supposed minimum standard of 50% efficacy corresponds to cases split 1/3, 2/3 between the vaccine and placebo groups.
***
When the statisticians receive the data from the trial, they can estimate the VSC in the trial population. The Pfizer press release said at the time of the interim analysis, they recorded 94 cases in total. We just computed that 90% VE corresponds to 9% VSC, so 9% of 94 or roughly 8 cases came from the vaccinated group and 86 cases came from the placebo group. If the vaccine is completely useless, the cases would split evenly, 47 cases each.
That analysis is inadequate, though, because it does not take into account sampling error. Imagine re-running history with a different set of trial participants. It's quite possible that the VSC in this trial would be around 9 percent but it is highly unlikely that it would end up at exactly 9 percent.
Instead of concluding that VSC is 9%, as observed in the trial, Pfizer statisticians assign probabilities for the value of VSC in a Bayesian-style analysis. Like this:
In this chart (a probability density curve), the value of VSC runs along the horizontal axis, and the height represents the probability. The peak of this density curve gives the most likely value of VSC, which is 8.4%.
Statisticians focus on the middle 95% of the probability, i.e. the VSC range from 4% to 16%. The FDA scores the trial based on the chance that vaccine's share of cases is below 41%, which, if you glance back at the first chart of the post, corresponds to VE at least 30 percent. (This means that the FDA does not actually require at least 50 percent vaccine efficacy as frequently reported.)
Below is the same probability curve in terms of VE rather than VSC:
This chart is more intuitive. As you might suspect, the shape of the curve is influenced by the specific outcomes of the vaccine trial. The above chart assumes the cases split 8 and 86 between vaccine and placebo groups.
In the next chart, you can see how the probability curve changes if the trial had returned different results.
Some experts suggest that the vaccine cases could be as low as 3. For that trial outcome, the Pfizer analysis yields the blue dashed line shown below. The probability mass shifts to the right (higher VE) and also becomes tighter (lower uncertainty).
I include also the purple dashed line for the scenario in which the vaccine and placebo groups both produced 47 cases. As expected, the probability mass is now concentrated on the opposite end of VE. Even though the measured efficacy in the trial is 0%, the Bayesian model assigns probabilities of VE to be up to 50%.
If you're interested in even more technical details, Sebastian's post is your next stop.
***
As I explained the other day, the Bayesian analysis requires a "prior." The prior expresses the analyst's subjective expectation of the answer without the benefit of the data from the trial. Once the data become available, the prior is modified to yield the probability curves shown above.
The following chart compares the prior and "posterior" density curves.
You can see how far the prior has been pushed by the strong data observed in the Pfizer trial.
The shape of the prior (pink line) is rather curious though. The Pfizer statisticians seem to believe that the most likely VE is 100%, which is shockingly optimistic. They hedge by spreading the rest of the probability rather evenly across all possible values. The data from the vaccine trial pull the peak away from 100% (there were confirmed infections among the vaccinated), while concentrating all probabilities to between 70% and 100%.
Recall that the FDA wants to know the chance that VE is larger than 30%. The black line tells us that the strong data from the trial indicate an almost certainty. What was the researchers' prior expectation? That's asking how much of the probability mass of the pink line lies to the right of 30%. According to my software, it is 57%54%. [Oops, I forgot to switch from VE to VSC.]
In plain English: prior to running the vaccine trial, Pfizer believes their vaccine has a 65 in 10 chance of reaching at least 30% efficacy (the FDA minimum requirement), and based on the strong result in the trial, they now think it is almost certain that the vaccine has at least 30% efficacy.
P.S. I tied up a couple of loose ends in this next post.
[11/26/2020: In a new post, I put up some Python code that can be used to generate the charts shown on this post.]
Comments