The role of prior infection in the vaccine trials

In the final analysis of the J&J vaccine trial (link), the study authors suggested that their vaccine benefits those who have previously been infected with Covid-19:

We observed that participants with previous asymptomatic infection (defined by serologic positivity for SARS-CoV-2 N protein and an absence of history of symptomatic Covid-19) can benefit from immunization with a Covid-19 vaccine. In a post hoc analysis, previous infection alone provided 90.4% protection against symptomatic infection, and after administration of Ad26.COV2.S in seropositive participants, 97.7% protection was observed in a comparison with seronegative placebo recipients.

Attentive readers should be raising their eyebrows as they digest this paragraph. The headline vaccine efficacy (VE) reported in this study was 56%. Against this backdrop, the above claim of 98% efficacy for people who have already caught Covid-19 before (aka seropositive) is quite simply astounding.

[The authors, nevertheless, chose their words carefully, as the statement does not actually make a causal claim, but merely promotes such an interpretation.]

***

Let's unpack the underlying data.

The randomized trial design included the usual pair of treatment groups (vaccine and placebo). However, FDA allowed excluding many subgroups post-randomization in the statistical analysis. One such exclusion are people who were seropositive at baseline (actually, it appeared that they may have excluded also anyone who got infected before 14 days after the shot, an additional complication I will ignore in what follows).

We can subdivide the trial participants into four subgroups, as shown in the following table:

Roughly 10% of the vaccine and placebo groups were deemed seropositive (previously infected) at baseline and removed from the headline analysis. The "post-hoc" analysis brings the seropositive subgroups back into consideration.

The headline analysis ignores the first row of the table, and reports on the difference between the case rates between the placebo and vaccine groups in the second row (expressed as a relative ratio). As mentioned already, the VE is 56%, which means that the case rate of the vaccinated was roughly 44% that of the placebo group. Since we are allowed a causal interpretation in the context of randomized trials, the vaccine roughly cuts the case rate by half, from 5.5% to 2.5%.

If the VE were to be 98% instead of 56%, the case rate of 5.5% would have to drop to 0.1%. This is the first place where the paper's aforementioned claim can easily be misinterpreted. Because the VE is expressed as a relative ratio, a 98% drop is not the same as a 98% drop - because the base rates are not equal.

In this example, the base rates cannot be more different. The average case rate for sero-positive (i.e. with prior infection) is only one-tenth of that for sero-negative people. As shown below, on the placebo arm, those with prior infections had a case rate of 0.6% (despite not receiving the vaccine) compared to 5.5% if they did not have a prior infection.

As the study authors acknowledged, natural infection acts like a vaccine with 90% efficacy. Well, they used more scholarly words:

Previous infection alone, in an analysis involving seropositive and seronegative placebo recipients, was found to provide 90.4% (95% CI, 83.2 to 95.1) protection against moderate to severe–critical Covid-19.

Now, where did the 98% claim come from? It turns out the authors were making a comparison between two unlikely subgroups: (seropositive + vaccine) vs (seronegative + placebo).

This comparison is odd in a number of ways. Two conditions are varied at the same time. The 98% VE represents the aggregate effects of the vaccine plus prior infection. The study makes no attempt to assign relative importance between these two changes. In theory, the effect of the vaccine can range from all of the 98% to none of it.

From a practical view, each person either has caught Covid-19 before or hasn't. Unlike the vaccine decision, serological status is typically not a manipulable treatment (unless someone deliberately attends a "Covid party".)

If you're unvaccinated with a prior infection, then your decision is whether or not to get vaccinated. The appropriate comparison is (seropositive + placebo) vs (seropositive + vaccine). This is the first row of the table.

If you're unvaccinated without prior Covid-19, then your decision is - also - whether or not to get the vaccine. The appropriate comparison is (seronegative + placebo) vs (seronegative + vaccine). This question is the primary endpoint in the trial.

The vaccine efficacy for seropositive people is 76% (first row). This is quite a bit higher than the 56% for seronegative people (second row). Remember though that 76% represents a reduction of case rate by 76% - off a base of 0.6% while the 56% represents a reduction of case rate by 56% - off a base of 5.5% so they are not directly comparable values. Besides, at the time of the trial, seronegatives outnumber seropositives 9 to 1.

In glorifying numbers > 90%, the study authors picked the larger case rate from the placebo column and the smaller case rate from the vaccine column to produce a meaningless 98% number. A fuller analysis shows that the vaccine is decently effective for those who have been infected but the improvement is on top of a tiny case rate.

***

If we are serious about learning the effect of vaccination on those who have already been infected, what experiment should we run?

It's straightforward. Recruit people who had previous infections. Randomly divide them into two groups, give one group the vaccine and the other group a placebo. In such a trial, you are effectively comparing (seropositive + vaccine) and (seropositive + placebo). This is precisely the first row of the table shown above.

The only difference is that the sample size of seropositive people was too small during the original vaccine trial. There were simply not enough people with Covid-19 when the trial was initiated.

***

At this juncture of the pandemic, experts have acknowledged that these Covid-19 vaccines do not stop infection or transmission - they say the purpose of these vaccines is preventing hospitalization and deaths.

In the vaccine trials, basically no one with prior infection subsequently was hospitalized for or died from Covid-19. In the J&J trial, only 15 participants out of over 42,000 were both seropositive at baseline, and subsequently counted as cases. Therefore, we have no data on how the vaccine affected the chance of hospitalization or death among the seropositive subgroup (but we know it's not a big number).

The 98% VE number mentioned in the opening quote of this blog is a metric about preventing infection, not severity of disease. Thus, we have been served incongruent messages that (a) previously uninfected people should not worry about getting infected (and only worry about getting severe disease) while (b) previously infected people should worry about getting infected, even though their case rate is 10 times lower than the previously uninfected.

 

Comments

[Paul]

I've just started to read the "final analysis" article and noticed something quite odd. In Table 1, under "Death from any cause", ">= 14 Days after Administration' there are 19 cases in the vaccine group (6787.0 person-years) compared to 45 cases in the placebo group (6669.3 person-years).

It appears that the J&J vaccine is an amazing longevity treatment!