It's the third time, and we've noticed a pattern. The FDA can't keep a secret. In late February, they approved the emergency use of the Johnson & Johnson (J&J) vaccine. On the day the Advisory Committee met to deliberate in public, there was zero suspense since the entire process was choreographed weeks in advance.
The headline result from the J&J trial are these:
J&J reported a vaccine efficacy of 67% during the trial. This result from a single multi-national clinical trial implies that the vaccine efficacy is above 60% in general. Emphatically, this statistic should not be interpreted as 6 out of 10 vaccinated people being protected (and 4 left unprotected).The baseline case rate after two months was about 2 percent during the J&J trial so 98 out of 100 participants did not get ill. Upon vaccination, everyone's chance of getting infected is cut by 60%. In addition, J&J also documented few side effects of concern. Given the FDA's stated intention of approving any vaccine that has over 30% efficacy (on par with the flu vaccine), the outcome of the meeting was never in doubt.
As you must know by now, the J&J VE number cannot be directly compared to VEs from other trials because the FDA allows each pharma to decide what it counts as a case, and which time window is used to count cases. J&J counts cases starting 14 days after the inoculation, and uses a more stringent set of rules to qualify a "case".
Since all vaccine trials are sized to deliver a preliminary verdict as fast as possible, none of these trials have sufficient data to draw strong statements about subgroups - especially not at the interim analysis. This is true also of J&J. In addition, it is premature to talk about the probability of rare events, such as "severe" cases, hospitalizations and deaths. This has not stopped the media from trumpeting "exploratory" and sometimes "post-hoc" findings in the same breath as the overall VE estimate, which is a more solid result.
(Notably, none of the post-hoc analyses comes with corrections for multiple comparisons, which is a basic tool to reduce the chance of spurious findings; the more things you look for, the more things you'll find, the greater the chance that your discovery is not replicable. This comment applies generally to all vaccine trials.)
The J&J vaccine is an adenovirus vector, so it's like the Astrazeneca-Oxford technology (link), and different from the mRNA technology used by Pfizer/Biontech (link) and Moderna (link). The adenovirus technology is also relatively new, having only had one prior drug approved for ebola by the EU while the mRNA has never had any (not counting the two EUAs granted for the novel coronavirus).
The J&J vaccine can be stored in standard refrigerators, which simplifies storage and transportation. However, the shots are still packaged in groups of 5 doses; once unpacked, the entire lot must be consumed within 2-6 hours.
The J&J vaccine treatment evaluated in this trial is a single shot. However, the pharma has an ongoing two-shot experiment that began in November (more later).
Exploring the case curve
Like the other pharmas, J&J provided a cumulative case curve starting from Day 0 (the day of the shot).
Let's review how this curve should be interpreted. In theory, the vaccine efficacy is eyeballed from the chart. If you pick a time, say Day 56, the cumulative incidence rate of the placebo group is roughly 2 percent (red line) while the cumulative infection rate of the vaccine group is just under 1% (blue line). So the vaccinated group experienced half of the incidence rate of the placebo group, which translates to 50% improvement, or VE of 50%. In short, the blue line is at half the level of the red line. (To be precise, the VE was 55% by Day 56.)
As with every trial, what's shown on the cumulative case curve does not lead to the headline VE number (which is 67% in the case of J&J). The headline number nullifies cases in the first 14 days. This is the same as resetting the count of cases on Day 14 to zero (while using the at-risk population on Day 0). I explained this procedure in greater detail in this blog post. Using the same technique as above, we see that the blue line is now at a third of the level of the red line. This two-thirds improvement corresponds to VE of 67%.
Since the primary success metric is VE, we should plot VE directly, which leads to the following chart:
I'm only drawing up to Day 56 because by that time, the analytically-ready population has already been cut by half - as most of the trial participants have not yet reached that amount of follow-up time. Remember that the FDA allows analysis when only half the participants have reached two months after treatment (currently interpreted as 56 days rather than 60 days). The data beyond that period is based on a small sample, which is also biased because of staged enrollment of lower-risk people first.
It is instructive to show J&J's curve on the same chart as Pfizer's.
Firstly, the Pfizer curve is much smoother throughout the first 56 days, with the difference particularly stark during the first 14 days. J&J's curve is a step function: VE is basically zero for two weeks, then VE magically jumps to 45 percent, and grows slowly thereafter.
On the one hand, someone can look at the J&J curve and say this justifies the presumption that VE = 0 in the first two weeks. The step function embodies the idea that the vaccine is useless for two weeks. On the other hand, someone can say a biological process affecting a large population over time should be smooth rather than a step function. My view is the latter so I wonder what happened with case counting in the first two weeks. Unfortunately, because J&J discounts cases in the first two weeks, the study report has very limited data for that period, e.g., it provides no information about how many "mild cases" were detected during the first 14 days.
Secondly, Pfizer's treatment has two shots, the second one administered around Day 21 while J&J's is one shot on Day 0. Interestingly, the two VE curves are closest together at Day 21 and then the Pfizer curve rises faster than J&J's. If I worked for Pfizer, I might see this as evidence that we timed the second dose just right. (Of course, there can be other reasons for the rate divergence.) When J&J has results for its two-shot trial, we'll have confirmation about this hypothesis.
Next, I look at weekly data, rather than cumulative.
The chart shows VSC (the vaccine's share of cases for a given week), and the timeline starts at 14 days after the first shot. The lower the VSC, the higher the VE. This view reveals the vaccine efficacy to be quite stable from week to week. The first data point for J&J (red) is the first week that J&J considers official, and has the lowest VSC (or peak VE), although we can't tell if these are just random fluctuations. The cumulative chart hides this information because the most recent week is weighed down by the data from previous weeks.
***
A good question to ask is whether every person requires the same level of protection from a vaccine. I believe the answer is no. People who have lower risk of exposure, people who are younger and fitter, people who take other mitigation measures more seriously, people who have fewer daily contacts with others, people who intend to wear masks after getting the vaccine, etc. have a lower baseline risk of infection, and so they may benefit from a one-shot vaccine.
I have quite a few pages of notes left, and cover some other interesting observations in the next post.
Comments