Take 2 on the Johnson & Johnson vaccine trial results

In the previous post, I characterized the vaccine efficacy (VE) of Johnson & Johnson's vaccine (J&J) as 67%, which is the headline number in the FDA report.

The VE Constellation

If J&J had offered a single estimate of vaccine efficacy, we'd have been shocked.

They centered our attention on 67%, which is described as the VE for "centrally-confirmed moderate to severe/critical Covid-19 cases" with a case-counting window starting 14 days after the first shot (1D+14).

The following list is a partial run-down of other VE estimates that also appeared in the report, and how they relate to the headline number.

If the case-counting window is shifted forward to 1D+28, the VE becomes 66% (lower by mere decimals). This estimate throws out another two weeks of cases, and was added to the clinical trial protocol midway through the trial. I can't think of a reason to justify this metric, other than to compare it directly to Pfizer's headline number (2D+7 = 1D+28).

If that's the case, we also need J&J's VE starting the count at 1D + 35 to compare to Moderna's headline VE at 2D+14. This is not directly cited in the report but it's still 67%.

The VE is 55% if case counting starts the day after the first shot. This metric is not referenced in the text of the report but can be obtained from the cumulative incidence curve I featured in the last post.

The numbers above aren't quite the same as Pfizer's or Moderna's because J&J removes "mild cases." However, J&J reported only 4 "mild cases with onset >= 14 days post-vaccination". This makes the definition of "mild" pointless (unless there are many mild cases prior to Day 14, which we do not know). The VE including mild cases is still 67%.

The headline VE also does not contain all known cases by the time of data cutoff. That's because J&J requires "central confirmation" of positive cases. About 30% of cases have not been confirmed as of the FDA meeting. These cases were excluded from the analysis - which has the same effect as assuming they are false positives. The report said that if all the unconfirmed cases came back confirmed, the VE will be 66%, decimals fewer than the headline number but on the other side of 0.5.

J&J employs adjudicators to decide if a Covid-19 case is "severe." Therefore, they report another VE that ignores moderate cases. This secondary endpoint was added midway through the trial in an amendment to the trial protocol. When moderate cases are removed, the VE goes up to 77% at 1D+14, and 85% at 1D+28. The definition of "severity" includes ICU admission and deaths but neither is necessary, and the adjudicators have the final say. We do not know how many cases they adjudicated. It is not clear how clinicians can utilize such a finding unless "severity" is predictive of hospitalization, intubation, death, etc.

In describing the above result, the report says "Efficacy against severe disease appears to be greater when cases that occurred before 28 days are excluded." Mathematically, the severe cases on the vaccine arm are more front-loaded, and if the earlier cases were excluded, the VE rises. This exposes the very odd enterprise of reducing case counts through exclusions, which almost always leads to increased VE. This increased VE has a more restrictive interpretation (only severe cases, only cases occurring after 28 days, etc.) so nothing practical is gained. The vaccine is less effective, not ineffective, in those excluded areas.

Next, there is a VE for hospitalizations, which was 82% at 1D+14 and 100% at 1D+28. We are now getting into the territory of rare events, and the error bars on those numbers span 80%. Not surprisingly, this analysis is post-hoc, not originally specified in the trial protocol. There are several inscrutable sentences about what counts as a "hospitalization" on p. 33 of the FDA report, and if interested, I'd direct you there (link to PDF).

There is one other VE estimate that should interest my American readers. The J&J trial has participants from the U.S. (~50%), Brazil (~20%), South Africa (~10%) and other South America. For U.S. participants, the VE was 74% at 1D+14, and 72% at 1D+28. These estimates included the unconfirmed cases, which means it is only comparable to the number that includes those (66%). The size of the trial is not large enough to conclude that VE is higher for Americans (the confidence intervals completely overlap).

Double Zero

Recently, the media started describing the primary purpose of vaccines as preventing deaths, even though none of the clinical trials have been sized to measure the death rate with the required precision. In several trials, including J&J, there have been zero deaths attributed to Covid-19 in the vaccine arm, and people then erroneously concluded that the vaccine provides 100% protection against death.

Most lottery players have never won a lottery. However, all lottery players do not conclude that their chance of winning is 0%.

J&J reported 7 deaths on the placebo arm that have been adjudicated as Covid-19 related. All seven occurred in South Africa. So, in the U.S. section of the trial, the verdict thus far is 0 vs 0. If you're an optimist, that's 100% protection. If you're a fatalist, that's 0%. If you're a statistician, you'd wait for more data.

Double Testing

The central confirmation of positive cases is, as far as I know, unique to the J&J trial. This procedure is effectively a guard against false positives but since negative tests are not checked, the double testing always reduces the number of detected cases. The FDA report discloses that typically about 90% of the positive results are confirmed by the central lab.

The central lab uses the Abbott Alinity RT-PCR test, the result of which over-rides local test results. It appears that this type of tests has a false-negative rate of at least 20% (link). The other trials do not re-confirm positives.

Let's assume 10% of all positive test results are thrown out by the central lab. The report tells us nothing more. It is not at all clear that the confirmation rate is even across all cases. Is the probability of confirmation the same between placebo and vaccine cases? Is the rate of confirmation the same regardless of the date of onset of cases? Any such imbalance affects how we interpret the data.

Double Blind

The J&J trial is double-blind while Pfizer/Biontech, Moderna and Astrazeneca-Oxford were all single-blind. This rules out a potential source of bias, and raises the question of why the other pharmas did not conduct double-blind trials.

Blinding reduces the chance that participants know their vaccination status and thus behave differently. Despite being double-blind, there are hints that some people in the J&J trial may have inferred whether they got the vaccine or the placebo. On Table 4, we learned that there was double the number of people who withdrew from the study on the placebo arm (72) than the vaccine arm (32) - this difference is highly significant, even though the rate of dropout is small.

Double Jeopardy

As with all trials, J&J tests each participant on the day of the first shot before innoculating him or her, and throws out anyone who tests positive. Because of this reason, J&J removed about 4,000 people, which seems quite high in a trial with 40,000 enrollees.

While these participants are no longer part of the trial, they presumably took the shot before their test results were available, and were thus followed up. Surprisingly, the vaccine appeared to be ineffective in this subgroup at both 1D+14 (3 vs 4) and 1D+28 (1 vs 2), and also ineffective against "severe" disease (1 vs 0).

Double Alibi

In a side analysis about "asymptomatic cases," J&J disclosed 3 cases that were originally deemed asymptomatic but later re-classified as symptomatic. The investigators said they analyzed symptomatic cases first since it's the primary endpoint, at which time those cases were deemed asymptomatic and thus excluded. When it came time to look at asymptomatic cases, the cases have been re-classified, and thus were excluded from the secondary analysis. In other words, those cases fell through the cracks, neither here nor there.

Perhaps only statisticians find this little episode amusing. We're talking about a few cases. This shows the high stakes in the vaccine competition among the pharmas; every case matters.

 

Comments

[A Palaz]

So I notice some things in this report. They seem to change between a confirmed and unconfirmed group in tables and also change age groups. Do you know why@

On the seroonverted there seem to be 2 sets those that were ecludedcat start band another for the asymptomatic section in the protocol set.

In the asymptomatic set should we add periods together assuming someone positive once remains seropositive or should we treat as separate between 1-29 and 29 ?

Thanks!

[Kaiser]

AP: Yes, those items are in my notes too but didn't make the cut for the blog.

The unconfirmed cases appeared briefly in the VE Constellation section. I don't accept the excuse that there wasn't enough time to do confirmatory tests on those samples. We are in a global emergency, they are moving things along at "warp speed" and they just couldn't find the time to do 200 "rapid" tests. I pointed out what they also claimed, which is if those cases were treated as if confirmed, their conclusions don't change. But here's the thing: the aggregate number may not change but different cuts of the data surely do change. Otherwise, we'd have to believe that the unconfirmed cases is a perfect random sample of all suspected cases (which I find highly unlikely). That may be why those unconfirmed cases sometimes show up in the tables, and sometimes don't. They are showing the best-case scenario for each analysis. What this practice does is to shift the game from trust the methodology to trust the researcher. I'd add that J&J is not the only pharma that plays this game. For example, Moderna's entire application was based on data in which much fewer than half of the participants have reached the required 2-month median follow-up time - with the promise that a completed but not "confirmed" analysis of the required data would have not changed the finding. That's trust the researchers again.

I did not discuss the asymptomatic analysis because the only trial that even made an honest attempt at that was the Astrazeneca-Oxford. You can't learn anything about asymptomatics when you only test participants once every month. They also did not disclose compliance rates on the testing, nor compare the characteristics of those who did the tests, and those who refused. In addition, the error bars on those estimates are huge to the point of meaningless. They re-defined the base population and the time periods because this entire analysis was not pre-specified in the protocol. The "after Day 29" analysis is an example of why I have criticized the arbitrary choice of case-counting windows. It makes the numbers look better but nothing changed practically except throwing out a bunch of cases - and not changing the denominator.

[A Palaz]

So I agree the range is big. I am more interested in the validity of looking at the asym data from the view of seroconverted.

I think their definition in sub samples seem reasonable if we consider the time to conversion for IgG meaning we can take as reasonably indicative the VE value for this sub sample? (c and d )?

So also I am not sure why the ranges are so wide. Any ideas? Is it sample size or maybe days at risk? Something other?

I did not see Astra section, will take it a look

Thanks!

[Kaiser]

AP: Validity - there were 19,600 at risk by Day 28, which means 30% of the participants did not have serological test results. Why? Is the serology risk set representative of all at risk? They didn't tell us.

Day 29 - I think it's unreasonable to define symptomatic cases as after Day 14 and then total cases (including asymptomatics) as after Day 29.

Ranges - it's just a game of numbers. The result they want us to look at is based on <1,400. It appears to have a less ridiculous error bar just because in that case-counting window, the ratio of infection rates shows a far larger difference (went from 15% higher to 3x higher on placebo). Not only is this result based on 1,400 but J&J used a staged enrollment (so did Astrazeneca), always enrolling healthier, younger participants first, so those who have a longer follow-up period are healthier and younger.