Today, I return to the Clalit booster study. I’ve written three prior posts about the study, which was the catalyst for regulatory approval of booster shots of mRNA vaccines for Covid-19 at the turn of 2022. In the first post, I showed how the methodology - common among Covid-19 vaccine studies - contains obvious biases that researchers and their peer reviewers overlooked. (link) The booster study featured a further unexplained wrinkle – some subjects are excluded after their infection status is known (link). In a third post, I developed a sports analogy to ponder what kind of effects such analytical procedures may have on the results.
This post describes further obvious biases in the study design. There are two classes of biases that affect observational studies in which we cannot randomly apply treatment to subjects. The first class of biases occurs when the two treatment groups are not statistically identical, except for the presence or absence of treatment. The second class of biases occurs when the study population does not look like the population to which the treatment if approved would be applied.
As explained in the first post, this study is unusual in that 90% of the study population appeared first in the no-booster group and later in the booster group. Thus, if we investigate covariate balance in the usual way, the two groups would look close to identical.
So the more interesting question is whether the study population represents people to whom Clalit intends to promote booster shots should they be approved. It’s tough to answer this question without access to the data. We can address a slightly different question: is the study population a random sample of Israelis 50 years old and above?
The answer is most likely no. According to OurWorldinData, the proportion of Israelis 50 year old and above who have received at least one booster dose during the study’s time frame was between 61-80%. And yet the study reported that 90% of the subjects got booster shots during the study period. So, the study population had greater inclination to take the booster shot than the general population.
In fact, fast forward to April 2022, seven months after the study ended, and the last month for which OurWorldinData compiled data, and we learn that the proportion of 50+ year old who have received at least one booster shot was stuck at 73-86 percent, still below the 90% recorded in the study.
Looking at the list of exclusions from these types of studies is highly informative. The key question to ask is whether people excluded from these studies will also be excluded from the rollout of the booster. Seemingly the answer is no – the booster will be recommended for essentially everyone.
This practice makes little sense to me. If the excluded people will be given boosters, why should they be excluded from the analysis?
To make this more concrete, here’s the first exclusion criterion: anyone who have ever been infected with Covid-19 prior to the study start date. I have never heard a public health official say don’t take boosters if you had been infected before; quite the opposite – they usually claim that prior infection does not confer immunity, and recommend prior infected to take the boosters.
What might be the effect of removing prior infected? If the public health claim is correct, then prior infection does not offer extra protection to this subgroup, and therefore including them doesn’t affect the effect of the booster shot. As a subgroup, those with prior infection may look different from those without, changing the baseline infection rate but that's not a reason to exclude. If the public health claim is false, that is to say, prior infection does confer some immunity, then any effect of the booster would be attenuated in this subgroup, and excluding them improves the reported result.
Another exclusion criterion involves those whose second vaccine shot (in the original two-shot Pfizer regime) was administered fewer than 5 months from the start of the study period. Once the booster is approved, is there any enforcement of this “rule”? Are people forced to wait at least 5 months before getting the booster?
Exclusions are always present in these studies, and not enough attention has been paid to the biases that exclusions create. Excluding prior infections mean only people without prior infections are in the study. This subset includes people who are health-seeking, and those who follow non-vaccine mitigation measures seriously (masks, distancing, stay home, etc.). Meanwhile, those who ignore mitigation measures and take risks are more likely to get infected, and thus get kicked out of the study.
The other exclusion also directly inserts a bias into the study – as it selects people who are less vaccine hesitant and thus take their vaccine shots earlier rather than later.
Comments