Those readers who have taken my invitation to jump in the deep end with the Astrazeneca-Oxford (AO) vaccine trial results have to hang on the pool's edge for another day. I'm getting distracted by another headline-grabbing supplementary analysis just advanced to the media, which makes two new claims:
- that the AO vaccine works better with a longer delay of the second dose,
- that a single dose of the AO vaccine confers 76% efficacy, which suggests that the second shot is not needed (even though, so far, all those who publishes such findings on the one hand advises taking the second shot, which suggests they don't believe their own results).
As I described it in the last post, when it rains, it pours. This is a pair of supplementary analyses in which the researchers once again slice and dice the data to fit their questions. Even though the data came from a randomized controlled trial, none of these supplementary analyses addresses comparisons of randomized subgroups.
***
First, let's examine the following finding:
In the SD/SD group, after the second dose, efficacy was higher with a longer prime-boost interval: VE 82% [...] at 12+ weeks, compared with VE 55% [...] at < 6 weeks.
If you read my last post on the Lancet paper, you know that they've already performed two other ad hoc analyses in a determined campaign for longer dose intervals. Those prior analyses considered cutoffs of 6 weeks and 8 weeks. Now, the goalpost has been moved to 12 weeks. This new analysis included more data: longer follow-up time for the people previously studied, additional participants (primarily from the Brazil cohort) who did not meet the minimum follow-up time requirement last time, and participants of other trials not included before, including Phase 1 cohorts (excluded in the prior analyses), and a newer trial in South Africa.
I did not see a discussion of U.K. and South African variants, nor why those datasets are similar enough to be pooled. Even if we ignore that difference, we know that the South Africa cohort will only affect the front part of the case trend because the trial started later.
Are you ready to get in the deep deep end with me? Hold tight.
***
This type of analysis is highly seducing. But it is filled with danger, and these analysts have fallen down the rabbit hole. It's not easy to explain the flaw of the analysis methodology. Let me start by visualizing the time-line of the trial experience. Imagine the following schedule for a single trial participant:
This participant enrolls and receives the first shot during Week 1, and then returns for a second shot in Week 4. S/he cannot contribute to the case count until Week 6, at least 14 days after the second dose. The case-counting window ends when the data are locked down for analysis. In this diagram, I assume this happens in Week 25. Thus, this participant will be counted as a confirmed case if it happens between Week 6 and 25 (a window of 19 weeks).
What if the participant gets sick in Week 5? According to the rules set by the pharmas, this case will be ignored. The participant is in effect counted as not infected despite being infected. This is the pink part of the diagram.
What if the participant gets sick before the scheduled second dose? According to the trial protocols, this participant will be excused from the second shot - this means s/he does not have a complete treatment, and so will be dropped from the analysis population. This is shown in the red part of the time-line.
Because this participant took his/her second shot during Week 4, s/he will be included in a low-delay subgroup (interval <= 6 weeks) in the dose interval analysis.
Now let's compare this participant to another one in the long-delay subgroup (interval >= 12 weeks). Notice the bottom time-line: this person will contribute to the case count between 14 days after the second shot and the data lockdown date, which totals 11 weeks.
The diagram exposes serious problems with the analysis methodology.
First, the case-counting window for the low-delay subgroup is consistently longer than that for the high-delay subgroup. The vaccine efficacy formula uses cumulative counts. So, the analysis is biased against the low-delay subgroup because longer windows lead to higher counts. This is a great real-world example of why it is necessary to fix the duration of case-counting for every participant.
Now, consider someone from the placebo group who gets sick during Week 13. Imagine this participant has been re-randomized into the vaccine group. If she gets her second shot in Week 4, then she will be counted a confirmed case. If she takes her second shot in Week 12, then she counts as an exposed but uninfected individual! This is why I disagree with designating that pink area in the timeline. It is much better to count cases from Day 1 of the trial - something known as an intent-to-treat (ITT) analysis.
It gets worse. Imagine someone who would get sick in Week 10. If this participant takes the second shot during Week 4, she counts as a case. If the same person was scheduled to take the second shot during Week 12, she would have been disinvited, and removed from the trial population. This exposes a surviorship bias. Anyone who gets sick during the first 12 weeks after the first dose will be removed from the efficacy calculation in the long-delay subgroup so it is not surprising that the long-delay subgroup showed higher efficacy. It's a mirage of how they defined vaccine efficacy.
***
Because of the definition of the case-counting window as starting 14 days after the second shot, a different way to look at how the VE numbers are computed is shown below:
Since VE is computed on cumulative counts, the short-delay subgroup is bound to have more cases just because the case-counting window is longer (all else being equal). In the following annotation of Figure S-2 from the new study, I indicated several other factors that can explain the apparent counterintuitive finding that the longer-delay subgroup has a higher VE than the lower-delay subgroup.
Sharp-eyed readers may notice that this new analysis includes the LD/SD subgroup while the two dose-interval analyses in the Lancet paper don't. The reason given for excluding the LD/SD subgroup last time was that since the second dose was added months after the enrollment began, the dose interval was greater than 8 weeks for everyone. The fact that this subgroup reappeared would not alter the dose interval. So the LD/SD subgroup resides on the right side of the above chart, to the right of 56 days. The further to the right, the higher the weight of the LD/SD subgroup. What do we know about the LD/SD subgroup? It's the subgroup that showed VE of 90%, which is very likely to be just a statistical anomaly. (In fact, in the latest study, incorporating a longer case-counting window, the VE of that subgroup has dropped to 83%.)
***
Similar issues plague the analysis of a single-dose treatment, so I'll make this short. Remember that the trial design was switched from testing single dose to two doses so there is officially no treatment arm of single dose. This group consists of "participants [who] chose not to receive the second dose" and people who experienced "delays in the administration of the second dose" (for whom they observe up to the second dose).
I have already explained before that this is no longer a randomized trial. The single-dose subgroup differed from the two-dose subgroup significantly in "age, sex, health and social care worker status, dose(LD/SD, SD/SD), country, ethnicity, and follow-up time (p < 0.001)."
The position the researchers take regarding these facts is like most scientific research reports. Disclosure is an act of exorcism. They proceeded as if such biases did not exist, after they acknowledged they do exist.
***
The new paper has not passed peer review. Reviewers should consider the problems outlined here when they evaluate its merits.
[P.S. 2/9/2021. The Oxford team has announced this week that the South African trial has shown their vaccine has no efficacy, especially against the dominant virus variant there. This raises the ethical question of why they included the data from South Africa in this analysis of the dose interval. It is clear that the vast majority of the South African cohort received their second dose before 6 weeks, which means they are over-weighted in the short-delay subgroups relative to the long-delay subgroups. Since this cohort has much worse overall efficacy than the other country cohorts, this is a significant factor explaining the variations by dose interval.]
Comments