Saying it again: timing traps 1 2 3 4...

Previously, I posted some notes on the study of vaccine effectiveness by Israel's largest health fund (insurer). I mentioned that these researchers are dealt a tough hand. The fast pace of vaccinations makes it very hard to find unvaccinated people who are similar to the vaccinated individuals. This formidable condition does not just bias the analysis set but also affects the counting of cases.

After reading many studies, I continue to doubt the ability to nail down the timing of control infections, which is the basis for estimating VE of any interim case-counting window. In this post, I will describe several issues that caught my attention - these are not unique to the Clalit study. Without the dataset, it's not easy to know how much these issues matter but these questions are worth asking.

Timing Trap 1

Due to the fast progress of the vaccination campaign in Israel, the researchers adopted a “rolling” mode of processing in which matches are found for newly vaccinated people, one day at a time. This process allows a vaccinated individual to be used as an unvaccinated control prior to his or her vaccination. If the Clalit team had followed the Mayo Clinic’s process, in which an unvaccinated individual is someone who did not get any shots throughout the study period, I believe they would suffer from excessive drop-off.

But the rolling mode of processing has an undesirable side effect. Consider someone who is vaccinated on January 1. This same person may be matched to a vaccinated person on December 25, while he or she was unvaccinated. Then, six days later, the vaccination takes place, destroying the prior match. Therefore, the unvaccinated control has a follow-up life-time of only 6 days.

The Clalit protocol requires that the vaccinated person of the matched pair be censored as well, meaning that person is also only tracked till January 1, therefore the follow-up life-time for the vaccinated person has been curtailed to 6 days. As you no doubt know by now, all research teams pick VE metrics that start counting cases sometimes from 14 days after the first dose, sometimes from 21 days after the second dose, and when a follow-up period is snipped at 6 days, no cases could ever count for that pair.

This births an awkward scenario. If the vaccinated person subsequently get sick, even if the case appears 14 days after the second dose, it would not count (technically because of censoring). This is a form of asymmetric treatment: the vaccinated person’s follow-up can be stopped by an action of the unvaccinated but the unvaccinated person’s follow-up cannot be ended by an action of the vaccinated. Shortening follow-up always leads to lower case counts.

Toward the end of the paper, the authors acknowledge that a large contingent of unvaccinated controls quickly flipped over to the vaccinated group “often within a few days of matching.” As a result, the average length of follow-up in the Clalit study is only 15 days. The main result they reported on the front page of the paper is for the case-counting window of Days 14 to 20. Many matched pairs have stopped being tracked by this time. The other most commonly cited VE estimate is for a case-counting window starting 7 days after the second dose (around Day 28). Fewer than 25% of the analysis set have been followed for over 25 days.

Timing Trap 2

A common exclusion criterion is positive PCR test prior to the vaccination date (for vaccinated cases) or prior to study enrollment date (for unvaccinated controls), defined as the vaccination date of the matched vaccinated individual. As I will now show, this exclusion rule seems to create a bias in favor of the vaccinated group.

Consider an infection that occurred on January 1. If the individual is subsequently vaccinated, then this case is excluded from the analysis, preempting matching. If the individual remains unvaccinated during the study period, then s/he can only successfully match with someone who received his or her first shot prior to January 1. If matched, the case accrues to the unvaccinated group.

Now, if we classify all January 1 cases by subsequent vaccination status, those cases affecting the vaccinated group vanish from the analysis while some of the cases on the unvaccinated group remain in the analysis set. This asymmetric treatment may produce a bias in favor of the vaccinated group.

Clalit's exclusion criterion removes people who tested positive prior to the study start date, which is December 20. The policy in Israel is not to vaccinate people who previously had Covid-19. But see also the next section. The Mayo Clinic's study excludes people with positive tests prior to getting their vaccinations.

Timing Trap 3

The authors of the NEJM paper explains how they compute vaccine effectiveness for a case-counting window that does not start the day after the first shot. Their favored window is Day 12 to 20 after the first shot. You already know this means nullifying any cases that are reported during the first 11 days.

These researchers went one step further. For each matched pair, if one individual get sick prior to Day 12, the other member of the pair is censored, that is to say, the other member cannot be counted as a case whether or not s/he gets sick after Day 12. The researchers simply stop tracking that individual. This rule limits the ability to measure a vaccine's ability to delay the onset of disease.

We don't have the data so we can't know the size of the impact. We'd like to see how many cases occurred during the case-counting window but did not count because of censoring due to events that happen on the other group.

Timing Trap 4

I already mentioned another timing-related problem in the discussion of the Mayo Clinic paper. The matching process creates subgroups of individuals that are considered "similar" on account of their age, sex, and other matching factors. Within any such subgroup, effectively, the observational study treats them as interchangeable.

But they are no longer interchangeable when we assign cases to a time-line! Because the timing of cases depends on what day is Day 0. Day 0 for an unvaccinated control is determined by the matched vaccinated individual. Therefore, a case detected on a given calendar day can be interpreted as a Day 1 case or a Day 30 case or any Day X case, depending on who among the subgroup the unvaccinated person get paired with.

***

As you can see, real-world studies are complicated enough without these timing traps. The exclusions and matching process already requires many discretionary decisions. The insistence on creating arbitrary case-counting windows makes these real-world studies much harder to interpret, and prone to mistakes. 

 

Comments

[Gary]

Great analysis. I have been using brute force to try and work out the at risk numbers that match their detailed tables in appendix and just gave up. Can you tell with your experience how sample sizes might differ in their subgroups?

Note too that they managed to prematch 260k of controls which leaves 336 for rolling matching 90 of which were censored and rematched from pool on rematching.

Some macro issues.

Since older age groups were vaccinated first one might expect longer time windows to be older and more difficult to rematch. Not sure how they managed all this.

There was a lockdown on Jan 8th.

I think they should include tesing data and suspect that a confounding issue is that vaccinated individuals are less likely to test.

This is somewhat backed up by this article.

https://www.haaretz.com/israel-news/thousands-of-israelis-tested-positive-for-coronavirus-after-first-vaccine-shot-1.9462478

This paper attempts to address the issues related to the context of the study and does provide some insight. Not sure it convinces

Https://doi.org/10.1101/2021.02.08.21251325

Good post on a good issue because many studies "benchmark"against it probably due to sample size.

Thanks!

[Kaiser]

Gary: The challenge of real-world studies is always the unending well of potential biases. There is direct evidence about vaccinated people less likely to test after vaccination in the Mayo Clinic paper. The only biases that these studies so far adjusts for are factors related to the individuals, demographics and clinical information - but you're right that they do not take into account environmental factors.