The Johnson & Johnson vaccine is being treated as a third-class citizen due to some safety concerns (link). Interestingly, such concerns did not surface in the "final" analysis of the single-shot Covid-19 vaccine, published in February, 2022 in New England Journal of Medicine (link). In this report, the scientists concluded "Ad26.COV2.S was associated with mainly mild-to-moderate adverse events, and no new safety concerns were identified". Since in the interim report, the FDA said "the analysis supported a favorable safety profile with no specific safety concerns," we were supposed to believe that the J&J vaccine is completely safe.
On the efficacy side, the final report headlined 56%. This number is meaningless unless we know how VE is defined. The 56% number comes from a counting window that starts 14 days after the shot (1D+14). During the interim report, the headline number was 67% (link).
However, these numbers are not comparable for reasons I'll get into in this post. The FDA has allowed pharmas to keep shifting the goal posts so nothing can really be compared like for like. The situation illustrates clearly why statisticians won't give you an answer unless we know all the details about how things have been measured!
***
First, the interim report was conducted when only about half of the participants have reached 2 months of follow-up - note that the largest number on the time axis was 126 days.
In the final analysis, about half the participants have reached 6 months of follow-up - the largest number on the time axis is now about 280.
So, the comparable value from the final report is not the headline VE which concerns up to 280 days of follow-up but the VE at day 126... and that number is 55% (1 - (436-100)/(938-185)). According to Table 2B (final report), the error bar around this number is 20% to 70% so the trial did not enroll enough participants to give a precise estimate of VE.
Harmonizing the follow-up period is not sufficient. Between the initial and final analyses, J&J ditched the requirement that all cases be "centrally confirmed". In the interim report, cases were counted only if a central laboratory confirmed the local PCR test positives. In the final report, cases accrued based on local test results.
***
Another major development is letting the placebo group get vaccinated. After J&J's vaccine was granted EUA at the end of February 2021, all participants reaching 6 months of follow-up were effectively unblinded so that those in the placebo group could get the vaccine if they so chose. Almost everyone did.
Since enrollment ended October 2020, and the data cutoff for the final analysis occurred in July 2021, everyone could have been followed for at least 6 months, if the above plan were implemented to the letter.
But that was not what happened. Some participants were unblinded prior to reaching the 6-month point because they requested getting one of the by-then authorized Covid-19 vaccines.
As a result, only 23% of the study participants had an observation window of at least 6 months. (By design, 100% should.) In fact, half of the participants in this trial had a follow-up period of less than four months. Because the follow-up period was truncated by the placebo participants receiving the vaccination, effectively destroying the placebo group, there will be no further useful data.
Refer back to the case curve shown above, notice that the at-risk population has dropped by more than half by day 126. The further out one went in time, the fewer participants contributed to the incremental case numbers, the wider the error bar. Such dropoff does not happen in a normal clinical trial.
Those who reached the longest follow-up period are also the earliest enrollees in the trial. In most vaccine trials, the pharmas first enrolled the lowest risk people - in the case of J&J, the earliest batch of enrollees were young people with no comorbidities. So the right hand side of the case curve lacks both validity and reliability.
***
When I wrote about the J&J trial, I pointed out that it was one of the few double-blind trials - the Pfizer and Moderna trials were single-blind. Blinding removes a source of potential bias - a kind of self-fulfilling prophecy in which the investigators or the participants behave in a way that advantages the treatment against the placebo.
We now have solid evidence that blinding did not work well, and confirmation bias was a real concern. To begin with, given that about 30% of Americans have obstinately refused to get the vaccine, we know that this subset is not represented in these vaccine trials. When the placebo groups in these trials were offered the vaccine, almost all of the participants took advantage, implying they were vaccine enthusiasts, not skeptics.
Blinding is supposed to neutralize partisan feelings about vaccines but it didn't work as planned. In Figure S1 (in the appendix), we learned that 1,222 placebo participants "received other vaccine before unblinding", compared to 670 on the vaccine arm. That's 80% more on the placebo arm than the vaccine arm. If the treatment groups were properly randomized, and if participants were properly blinded, they would not know whether they received the placebo, and so the proportion of people who got another vaccine against trial rules should be balanced.
The number of participants who withdrew during the trial was also disproportionately higher on the placebo arm (338 vs 224), about 50% more. These are huge gaps.
***
One of the big mysteries of Covid-19 vaccine trials is what counts as a case. The case-counting window is one variable I've discussed often. But there's more. Most of these early trials were designed to measure symptomatic, PCR-confirmed cases. J&J added a definition of mild, moderate and severe cases, although as I pointed out before, the definition of "mild" disease is meaningless since almost every case has been classified as moderate or severe.
In most studies, including the J&J trial, all participants are given serological tests at regular time intervals. Seropositivity indicates that someone has been infected in the past, and thus, it is presumed that said infections were asymptomatic. This device has been used by scientists to claim the vaccines also prevented asymptomatic infections.
Apparently, J&J now believes that someone without a positive result from either PCR or serological testing can also be infected. This is how I interpret one of the footnotes from Table 1 that corresponds to a concept called "any SARS-CoV-2 infection", which includes:
"undetected cases that were subsequently detected through a positive serological result (according to the clinical severity adjudication committee), which did not count as either symptomatic cases (becqause they were RT-PCR-negative) or asymptomatic cases"
I'm not sure how the adjudicators got wind of the "positive serological result" since if such a result was present, the case should have counted as "asymptomatic", which is a different line in Table 1.
(Also, the severity adjudication committee is a misnomer as the committe members concerned themselves here with asymptomatic infections that would be milder than "mild".)
What is shocking about this line of numbers is the magnitude. There were 1,038 such infections in the vaccine group and 1,699 in the placebo group. These two numbers are much higher than the counts that made it to the headline VE calculation. In particular, only 42% of the vaccine infections, and only 52% of the placebo infections, ended up as "cases". This calculation also shows that an "any" infection on the placebo group has a higher chance of being counted as a case than an "any" infection on the vaccine group.
"...we know that this subset is not represented in these vaccine trials."
I was thinking about this myself awhile ago. The participants of a clinical study are randomly assigned a treatment and control, but the participants themselves are not randomly recruited, they are self-selected. So they're not really "randomized". It makes sense that most of the placebo group would go on to get the vaccine; the anti-vax or vax-hesitant are much less likely to sign themselves up for a clinical trial, I imagine.
I guess this means that clinicians need to apply some form of sampling bias correction, even for "randomized" trials, if they aren't already.
Posted by: Anonny | 05/12/2022 at 03:50 AM