Raphael Lataster has written a response to our JECP paper, which discusses several biases in Covid-19 vaccine observational studies that have not been properly accounted for in many notable studies.
He augmented our first example to show another source of bias in later studies.
In our example, we demonstrated how the "case counting window bias" could cause vaccine effectiveness estimates to be over optimistic. In typical clinical trials, one starts counting cases (events) from the day the treatment is finished, so in the case of two-shot Covid-19 vaccines, this should be the day after the second dose. However, none of the Covid-19 VE results you have read about include all these cases!
All studies have imposed a case-counting window that always starts X days after the second dose. X is most often 14 days but sometimes it's 7 days, sometimes it's 28 days, or whatever the researchers deemed appropriate, frequently after computing VE for different window sizes.
The problem is that in observational studies, for the unvaccinated group, there is no such thing as 14 days after second shot (because they did not get any shots, unlike those in the clinical trial who got placebo shots). In many studies, the case-counting window is applied asymmetrically to the vaccinated group (reducing the case count) but not to the unvaccinated group.
Lataster pointed out another problem that creeped in with later studies. Many prominent studies started to define the "fully vaccinated" group as those having two doses (within some allowable dose duration) and who did not get infected prior to 14 days after the second shot, and then label the complement of the fully vaccinated group as the "unvaccinated" group.
This creates a host of problems. For the sake of discussion, let's focus on one subset: the group of doubly vaccinated people who got infected during the first 14 days after taking the second shot. In our original example, these cases are taken out of the case count in the vaccinated group. Effectively, these cases disappeared from the analysis.
In Lataster's example, these cases did not disappear. They got shifted into the unvaccinated group because that group now included doubly-vaccinated people who got infected in the first 14 days.
***
The decision to apply a case-counting window is tragic. It lands researchers into a dead end. This is a point Peter and I made in our reply to Lataster's response (here, paywalled).
The key of understanding our argument is to pay attention to both numerators and denominators in the case infection rates, which underlie the VE formula. The numerators are the cases, and the denominators are the subjects from which cases accrue.
In typical clinical trials, the treatment populations are fixed in advance. In an intent-to-treat analysis, one must use the treatment groups as originally randomized. Even in a per-protocol analysis, we remove subjects who did not complete the treatment (at the risk of adulterating the randomization). This procedure is different from removing subjects who completed the treatment (2 doses), and then got infected (i.e. have the primary outcome) within X days of finishing the treatment.
However, if the analyst uses the per-protocol population, and also applies the case-counting window, then these subjects who got infected during the first 14 days disappear from the numerator (case count) but not the denominator. In effect, they have become "immortal". Further, in observational studies, for which we cannot apply the case-counting window to the unvaccinated group, the analyst has made some immortals in the vaccinated group but not the unvaccinated group!
In Lataster's example, not only are the cases moved to the "unvaccinated" group but also the subjects. The analyst has taken the doubly-vaccinated person who got infected during the first 14 days out of the vaccinated group and added this person to the unvaccinated group. The case count follows along. This procedure makes me very uncomfortable. What happened here is that the analyst altered the treatment group assignment after looking at the primary outcome (Covid-19 cases).
This is why I called this a dead-end. If you just move the case not the group assignment, then you insert an immortal bias. If you move both the case and group assignment, you're manipulating the group assignment based on observing the primary outcome of the analysis. And, as Peter and I explained in our response, we are not convinced by arguments for case-counting windows.
***
Another part of Lataster's comment concerns empirical evidence that showed negative VE. I'm unwilling to go as far as he does because those negative VE numbers are observed in subgroup analyses for selected studies. Much more work needs to happen before we can establish generality of those results.
Comments