Big Data, Plainly Spoken (aka Numbers Rule Your World)

Vaccines can do no wrong, sloppy researchers say

Washington Post devoted space to some more sloppy medical research last week when it printed an article about vaccinations for various ailments providing protection for Alzheimer's disease/dementia (link).

Without even reading the details, I already know this is not a randomized clinical trial (RCT) but another observational study based on analyzing databases. Having seen many sloppy observational studies during the Covid-19 pandemic, I was not surprised to find that this study is also fraught with fundamental problems.

The bar for proving cause-effect in observational studies is extremely high for the simple reason that there are too many possible explanations for any observed effects, other than the treatment under study. The RCT setup helps eliminate alternative explanations.

The key study being cited for this connection between vaccines and Alzheimer's is this one. The study uses the propensity score matching methodology to construct pseudo treatment and control groups. The pseudo groups are then compared based on their Alzheimer's case counts. I previously reviewed studies using propensity scoring here.

The researchers performed three separate analyses for three different vaccines, and reported the following results:

• Tdap/Td vaccination (tetanus, diphtheria): ~70% reduction in Alzheimer's risk, 7% of vaccinated vs 10% of unvaccinated
• HZ (herpes zoster/Shingles): ~75% reduction in risk, 8% of vaccinated vs 11% of unvaccinated
• pneumococcus: ~73% reduction in risk, 8% of vaccinated vs 11% of unvaccinated

[This research team has another study with similar results for flu vaccines.]

***

The setup of the observational study analysis is flawed, as it considers the three vaccinations as independent events. In reality, these three treatments are connected in two ways: first, it's likely that people who get one vaccine are more likely to have gotten the others; second, if each vaccine conferred benefits, what is the aggregate benefit of having had three vaccines?

Think about that for a moment. Let's go from top to bottom down the list of benefits. For someone without Tdap/Td vaccination, the chance of developing Alzheimer's was 10%; if the person had the Tdap/Td vaccine, the risk drops to 7%. What if the person also had the HZ vaccine?

The problem we run into is that the baseline risk for HZ unvaccinated is shown as 11%. Now, this group includes Tdap/Td vaccinated whom we expected to have a risk of 7% (not 11%). What is the risk reduction of HZ vaccination on top of Tdap/Td? The second listed benefit does not help.

If we simply apply the 75% risk reduction, then the new risk is 25%*7% = 1.8%. What did I just do there? I just made an assumption - that the risk reduction is 75% whether or not the person got the Tdap/Td vaccine. In other words, I assumed that the effects of Tdap/Td vaccination and of HZ vaccination on Alzheimer's are independent. Is this assumption justified? Absolutely not.

But let's persist. We now arrive at the third listed item. Again, the baseline risk is said to be 11%, obviously not for the group of people who had both the Tdap/Td and the HZ vaccines. We again assert independence. Now, the aggregate Alzheimer's risk of having had all three vaccines is 27%*1.8% = 0.5%.

Merely by invoking magical independence assumptions, I have shown the preposterous result that vaccinations would effectively wipe out Alzheimer's.

***

Can other researchers verify the results? Most certainly not. Under "data availability", the authors said they "cannot make data and study materials available to other investigators due to licensing restriction". They even cheekily added "interested parties can license the CDM by contacting Optum." So they are saying if you want to check our work, you have to pay up.

The remainder of this post is necessarily speculative because I'm not paying up.

***

The researchers made a false start in trying to deal with this issue of multiple, correlated treatments. In constructing the comparison groups, they not only used the typical matching variables such as demographics and comorbidities, but they also included vaccination statuses. For example, for the analysis of the Tdap/Td vaccination, they included vaccination status of HZ and of pneumococcus as matching variables.

For propensity matching, the goal is to ensure that the two groups being compared were as close to identical as possible before follow-up. Therefore, I'm going to assume that the vaccination statuses being measured are those prior to the follow-up period of the study. 

After matching, one group is labeled with Tdap/Td vaccination (during follow-up), and the control group is without Tdap/Td vaccination (during follow-up). These two groups are statistically identical on all the matching variables. Then, the researchers counted the number of Alzheimer's cases in each group during the follow-up period. That's how they got 7% and 10%, leading to an effectiveness of ~70%.

What about people getting vaccinated for HZ and/or pneumococcus during the follow-up period? The fact that the two groups started out with similar proportions of vaccinations for those two conditions does not imply that they would continue to maintain balance on such vaccinations! In fact, the balance is most likely lost during follow-up. Why? One key difference between the two groups during follow-up is that one group got the Tdap/Td vaccine while the other didn't. To the extent that there are correlations between vaccinations of different types, we expect the Tdap/Td vaccinated group to over-represent those who got one or both of the other vaccinations.

Given that those other vaccinations are also claimed to reduce Alzheimer's risk, now you have a hopelessly confounded situation. This research study has certainly not proven its thesis. We simply do not know whether the reported effect of Tdap/Td vaccination included the effects of the other two vaccinations.

***

Can we use the vaccination status for the entire study period during propensity matching, rather than just the vaccination status prior to the follow-up period?

This gets us into a different kind of methodological hell. The design of the comparison groups now requires data from the "future". In addition, these two variables from the future are believed to be highly correlated with the study's outcomes. So effectively, we would have used future knowledge of the study's outcome in selecting subjects into comparison groups. Definitely not a good practice.

***

Table 1 and Supplementary Table 2 provide some more details about the counts of people in the study. It appears that 1.6 million people were under consideration for matching. Under Table 1, prior to matching, the researchers reported 123K people who had the Tdap/Td vaccination during the follow-up period. In addition, they said 20K+3K = 23K people had HZ vaccination (aggregate of both Tdap/Td vaccinated and unvaccinated).

But then in Supplementary Table 2, the number of people with HZ vaccination during follow-up was 212K. This number is way larger that the 23K from Table 1. This is why I interpreted the 23K as those who got the HZ vaccination prior to the follow-up period.

Similar patterns hold for all three tables.

***

TLDR;

Causal inference is very hard. I have not seen evidence that medical researchers have the right skills to do it well.

 

Observational studies that don't include everyone intended to receive the treatment

Today, I return to the Clalit booster study. I’ve written three prior posts about the study, which was the catalyst for regulatory approval of booster shots of mRNA vaccines for Covid-19 at the turn of 2022. In the first post, I showed how the methodology - common among Covid-19 vaccine studies - contains obvious biases that researchers and their peer reviewers overlooked. (link) The booster study featured a further unexplained wrinkle – some subjects are excluded after their infection status is known (link). In a third post, I developed a sports analogy to ponder what kind of effects such analytical procedures may have on the results.

This post describes further obvious biases in the study design. There are two classes of biases that affect observational studies in which we cannot randomly apply treatment to subjects. The first class of biases occurs when the two treatment groups are not statistically identical, except for the presence or absence of treatment. The second class of biases occurs when the study population does not look like the population to which the treatment if approved would be applied.

As explained in the first post, this study is unusual in that 90% of the study population appeared first in the no-booster group and later in the booster group. Thus, if we investigate covariate balance in the usual way, the two groups would look close to identical.

So the more interesting question is whether the study population represents people to whom Clalit intends to promote booster shots should they be approved. It’s tough to answer this question without access to the data. We can address a slightly different question: is the study population a random sample of Israelis 50 years old and above?

The answer is most likely no. According to OurWorldinData, the proportion of Israelis 50 year old and above who have received at least one booster dose during the study’s time frame was between 61-80%. And yet the study reported that 90% of the subjects got booster shots during the study period. So, the study population had greater inclination to take the booster shot than the general population.

In fact, fast forward to April 2022, seven months after the study ended, and the last month for which OurWorldinData compiled data, and we learn that the proportion of 50+ year old who have received at least one booster shot was stuck at 73-86 percent, still below the 90% recorded in the study.

***

Looking at the list of exclusions from these types of studies is highly informative. The key question to ask is whether people excluded from these studies will also be excluded from the rollout of the booster. Seemingly the answer is no – the booster will be recommended for essentially everyone.

This practice makes little sense to me. If the excluded people will be given boosters, why should they be excluded from the analysis?

To make this more concrete, here’s the first exclusion criterion: anyone who have ever been infected with Covid-19 prior to the study start date. I have never heard a public health official say don’t take boosters if you had been infected before; quite the opposite – they usually claim that prior infection does not confer immunity, and recommend prior infected to take the boosters.

What might be the effect of removing prior infected? If the public health claim is correct, then prior infection does not offer extra protection to this subgroup, and therefore including them doesn’t affect the effect of the booster shot. As a subgroup, those with prior infection may look different from those without, changing the baseline infection rate but that's not a reason to exclude. If the public health claim is false, that is to say, prior infection does confer some immunity, then any effect of the booster would be attenuated in this subgroup, and excluding them improves the reported result.

Another exclusion criterion involves those whose second vaccine shot (in the original two-shot Pfizer regime) was administered fewer than 5 months from the start of the study period. Once the booster is approved, is there any enforcement of this “rule”? Are people forced to wait at least 5 months before getting the booster?

Exclusions are always present in these studies, and not enough attention has been paid to the biases that exclusions create. Excluding prior infections mean only people without prior infections are in the study. This subset includes people who are health-seeking, and those who follow non-vaccine mitigation measures seriously (masks, distancing, stay home, etc.). Meanwhile, those who ignore mitigation measures and take risks are more likely to get infected, and thus get kicked out of the study.

The other exclusion also directly inserts a bias into the study – as it selects people who are less vaccine hesitant and thus take their vaccine shots earlier rather than later.

A sports analogy for data processing rules

I'm been mulling over how to explain the potential impact of strange data processing procedures used in the observational studies of Covid-19 vaccines. These are by no means standard operating procedures.

And I think I've landed on one explanation that is tantalizing.

***

Imagine a football (soccer) match that is tied 0-0 and heading into overtime. Overtime is a 30-minute affair, played through, and not sudden death. The overtime period is divded into two 15-minute halves with a short break in the middle.

That is the standard operating procedure.

***

Now, we are going to change the rulebook.

The first additional rule is that any goal scored in the first 10 minutes of overtime doesn't count. This rule is inspired the "case counting window" used in the Covid-19 vaccine studies. We're justifying it because players need time to get back to their optimal condition after the exhausting 90-minute regulation time. Therefore, it is not fair to count goals scored at the start of overtime.

The second additional rule is that any goal scored in the first 10 minutes of overtime by the visiting team doesn't count but those scored by the home team counts. How should I justify this rule? Well, to establish the need for recovery time, we take vitals of the players. The measuring machines are only installed in the home locker rooms and not the guest locker rooms, and thus, we are unable to determine if the guest players are at their optimal condition. This rule is inspired by not applying the case counting window to non-vaccinated people since we can't determine when they took the second shot (unlike the situation in clinical trials, when non-vaccinated people are given placebo shots.)

The third additional rule divides the first 10 minutes into two parts, and stipulates that if someone scores in the latter part (i.e. minutes 6-10), then they are immediately kicked out of the match, and their goal doesn't count. This rule is inspired by the special exclusion criterion disclosed in the booster study I've been discussing (link).

This third rule is complicating as it modifies the previous rules. Taken together, we have the following scenarios: for the home team, if they score in the first 5 minutes, the goal counts; if they score in minutes 6-10, the scorer is kicked out of the match, and the goal is disallowed. For the away team, if they score in the first 10 minutes, the goal is always disallowed; in addition, if the goal is scored in minutes 6-10, the scorer is ejected.

***

One can easily get lost in the details of pharmaceutical studies and lose sight of the implications of the methodologies. Transferring the context to sports helps me grasp the effects of these data processing procedures. Hope this helps you too.

PS [8-4-2023] In my haste to put down my thoughts relating to this analogy, I skipped a rule. This is a Rule #2.5. This rule says that if a goal is scored in the first 10 minutes of overtime by the visiting team, it is counted as a goal scored by the home team. In Rule #2, the away-team goal is just disallowed; in Rule #2.5, the goal is added to the home-team tally. This rule is inspired by the booster study, in which they kept people in the no-booster group for 7 days after they took the booster.

 

 

Sloppy language misleads

One of the most frustrating aspects of reading Covid-19 related publications is reading the conclusions. A summary of results is supposed to provide readers with the minimally necessary information to understand the full contents of the paper - but what we encounter is "story time": almost always, these conclusions are stated in the most general sense, ignoring all the caveats, the exclusions, the imperfections of the methodologies and analyses. They are "stories" that go far beyond the limited evidence presented.

I've been reading the Clalit study that was a major force behind the approval of the first booster shots back at the end of 2021/start of 2022. This paper made the following one-line conclusion:

Participants who received a booster at least 5 months after a second dose of BNT162b2 [i.e. Pfizer] had 90% lower mortality due to Covid-19 than participants who did not receive a booster.

Is it possible to misread this sentence? Yes, if you literally read the sentence.

***

If we know something about vaccine efficacy/effectiveness, we immediately recognize that these authors are selling the same story - that the vaccine has VE 90%, the only difference here being the outcomes are deaths attributed to Covid-19, rather than reported cases. As such, this VE is driven by the relative ratio of the death rates of the booster group vs the no booster group. They didn't use the term VE as it typically connotes cases not deaths.

A death rate is a fraction composed of two parts: people who are counted as part of the comparison group, and people within that group who died during the study period. In this post, I'll pay attention to both these parts.

From my last post, we already know one issue with the above statement. If we take any participant at the start of the study and ask whether this person is counted as part of the booster group or the no-booster group during the study, the answer, for 90% of the participants, is both. 

Since at the start of the study, 100 percent of the participants had not received the booster, and by the end of the study period (a mere 54 days), 90 percent of them got the booster, the researchers could not have divided up the study population into two distinct groups of people - each person (except the 10% who didn't get the booster during the study period) first appeared in the no-booster group, and then transitioned to the booster group.

But the concluding statement reads like the study enrolled two distinct groups of participants. This misreading is even more likely for knowledgable readers because ordinary studies, such as well-designed clinical trials, use distinct groups.

Note that I'm not criticizing the methodology up to now, I'm describing what it is, and criticizing how the language in the summary statements sends readers the wrong message.

***

The above detail may sound trivial but it has major implications. Think about the regression adjustments. As with all observational studies, researchers claim that by adding standard covariates to their regression models (age, gender, ethnicity, prior health conditions, etc.), they have corrected substantively all biases. For example, if older participants are more likely than younger ones to want the booster, rather than being assigned a treatment at random as in a randomized clinical trial, then it's possible that the age distribution in the booster group does not mirror that in the no booster group, i.e. an age bias exists.

Look at the setup here: Ninety percent of the participants appear once in the treated group and once in the not treated group. By the design of this study, there is little variation across the comparison groups in the distribution of everything, including age. If we check "covariate balance", it's balanced because the two groups are by and large identical. So this type of studies underestimates the variability of everything - which leads to p-values that are too small, and regression adjustments that are meaningless.

***

In the paper's conclusion, the phrase "participants who did not receive a booster" is incorrect in more ways. We already know almost all of them ended up getting boosters before the end of the study period, which was fewer than two months long. I have previously said that a participant transitions into the booster group when s/he takes the booster shot.

This is in fact not true. That's because this study also imposes a "case counting window," under the claim that the booster shot takes 7 days to become effective. In other words, if someone takes a booster shot, and falls ill with Covid-19 within the first 7 days, this case is not counted as a case in the booster group.

Negating this case is one thing - but this study (and many others) took another step: the case is not negated but it is counted as a case in the no-booster group!

How do they pull this off? The study design says not to transition the participants to the booster group until the 8th day after the booster shot.

So, let's review this. Each participant starts off in the no-booster group, then the person took the booster shot; this person is kept in the "no-booster" group for an additional 7 days before being moved into the "booster" group. During these 7 days, if the person gets infected or died from Covid-19, the case or death is counted as "no booster".

Remember that we can only talk about exposure time for this type of study since almost everyone is eventually exposed to both conditions. The exposure time of the "booster" group therefore excludes the first seven days after the booster while the exposure time of the "no booster" group includes the first seven days after the booster.

This is an instance of the additional bias that Lataster described in his comment on our Three Biases paper.

***

As the story goes, even the above description is inaccurate. We haven't dealt with the following puzzling disclosure:

participants who received the booster and had a confirmed case of Covid-19 within 3 days before the effective-booster date (defined as 7 days after the booster was administered) were excluded

The analysts further subdivided the seven days after the booster shot is administered into two parts: Days 1-4 and Days 5-7. Let's follow a participiant who did not have the booster at the start of the study. During the study period, this person took the booster shot. If the person got sick during Days 1-4, the person is counted as a member of the no-booster group with infection as outcome. However, if the person got sick during Days 5-7, this person is excluded from the study, i.e. not counted as part of the no-booster group, and not counted as a case. Finally, if the person did not get sick by Day 7, the person transitions to the booster group.

There is a subtle but important difference. The Days 1-4 treatment is applied to every participant, regardless of outcome. If they got the booster, they continued to be called "no booster" for seven additional days. The Days 5-7 treatment is an exclusion based on the outcome. If the person got sick, the person is removed while if the person did not get sick, the person is retained. For me, it is never acceptable to re-define treatment groups after observing the outcomes. And yet, this paper (and many others) engages in this practice.

To understand the impact of these data processing procedures, you can use the same device we used in the Three Biases paper: imagine a vaccine that is ineffective, apply the analysis, does it result in no effect, or something else?

This is a great example of why researchers must disclose the code used to process the data. It's hard to believe these special adjustments did not get the attention of reviewers. But I can think of no other way to interpret those words. (In fact, the waterfall chart in the paper confirms the exclusions.)

***

Now that you have heard how they measure outcomes and define "no booster", it's time to revisit the conclusion:

Participants who received a booster at least 5 months after a second dose of BNT162b2 [i.e. Pfizer] had 90% lower mortality due to Covid-19 than participants who did not receive a booster.

Does this sentence correctly summarize the study?

 

 

One big problem with booster studies (and observational studies of Covid19 vaccines)

I'll be writing a number of posts about observational studies of the Pfizer booster, published by the Clalit health insurer in Israel (back in December 2021), which is the primary source supporting the effectiveness of the booster shot (3rd shot). These studies provide fodder forever for classes in observational studies because (a) there are many nooks and crannies in which residual biases hid, and (b) researchers made many assertions (wrong, of course) about having corrected all biases using the simplest, most conventional methods.

This first post points out the biggest problem with such studies, which is perhaps impossible to correct.

Let's first review the basic setup of such studies. (The devil is of course in the details, and I'll dive into those in future posts.)

Each study has a well-defined start and end date. This is not a clinical trial, so no one proactively enrolls in this study. Instead, the researchers had access to databases with relevant data and extracted the study population from it.

The start date is chosen close to the launch of the booster shot in Israel, and therefore, at the start, almost everyone was in the "no booster" group. Over the course of the study, most of these same people have migrated to the "booster" group - when they took the booster. Even though the study spanned fewer than 2 months, almost everyone (90%) received the booster shot by the end date.

Here is the first thing one must recognize. This is not a typical clinical trial in which one person belongs to either the treatment group or the control group but not both. In this study, nine out of ten people featured in both the "no booster" and the "booster" groups.

So when subsequently, researchers compare the "no booster" group and the "booster" group, they talk about exposure time, rather than number of exposed. The exposure time of the "no booster" group are incurred mostly by people who eventually got the booster within the study period. This is key observation #1.

If we now hone in on an individual who got the booster shot during the study period, the entire observation time is divided into two parts: from start of study to the day of the booster ("no booster" exposure), and from the day of the booster to the end of the study ("booster" exposure). It is 100% certain that the exposure time in the "no booster" group precedes the exposure time in the "booster" group. This is key observation #2.

Moreover, observation #2 deos not depend on the date of booster, which varies by person. Since individually, each person is exposed first to "no booster" and then to "booster", collectively, the exposure time of the "no booster" group is skewed towards the start of the study period while the exposure time of the "booster" group is skewed towards the end of the study period.

What else do we know about the booster study (or, the majority of Covid vaccine studies)? These studies almost always begin during a peak infection period, which means the infection rate dropped during the study. This is key observation #3.

Together, our three observations mean that the exposure time associated with the non-booster group is skewed towards the front end of the study period, which also happens to be a peak infection period while the exposure time associated with the booster group is skewed towards the back end, when infections are much lower.

For the Israel Booster study, the authors essentially confirmed the existence of such bias with the following sentences:

"During this time [ed: study period],the incidence of Covid-19 in Israel was one of the highest in the world."

"The incidence of Covid-19 and thus exposure to SARS-CoV-2 changed during the study period."

Given that at the start of the study period, Israel's infection rate was "one of the highest in the world" and that the incidence and exposure subsequently "changed," this change represents a decrease in incidence and exposure. (The data can be found here. One of the charts shows the 60 and over age group.)

Reflecting the current state of medical "science," the authors didn't make the above statements in order to explain a source of bias. Instead, they used the first statement to explain why their study could achieve statistical significance on mortality due to Covid, which is a rare outcome, and they cited the second statement in order to make the false claim: "we assume that after adjustment for all covariates, including socioeconomic status, these changes had a similar effect in the booster group and the non-booster group."

Their "assumption" I just showed to not hold.

***

This is another example of the "background infection rate" bias that we described in our bias paper (link). In that paper, we used an example of a Danish study which also started when the infection rate peaked and then by the end of the study period, the infection rate had dropped to almost zero.

If the booster were to be completely useless, the average case rate in the no booster group would be higher than the average case rate in the booster group because of differential exposure time against a decreasing background infection rate. This leads to a high vaccine effectiveness when the effectiveness is assumed to be zero!

Adjusting for "all covariates" does not cure this bias whatsoever.

***

What can be done about such bias?

For one thing, better data disclosure. All these studies should show the distribution of exposure time, by treatment vs control group, over the study period. On top of that, we can superimpose the background infection rate. Such an analysis discloses the source of bias.

Next, attempt to do the same analysis for a different time period, especially one in which the infection rate trends in the opposite direction. In practice, this may be hard to accomplish if the public health department decided to launch vaccines around an infection peak.

How hard can counting be?

Raphael Lataster has written a response to our JECP paper, which discusses several biases in Covid-19 vaccine observational studies that have not been properly accounted for in many notable studies.

He augmented our first example to show another source of bias in later studies.

In our example, we demonstrated how the "case counting window bias" could cause vaccine effectiveness estimates to be over optimistic. In typical clinical trials, one starts counting cases (events) from the day the treatment is finished, so in the case of two-shot Covid-19 vaccines, this should be the day after the second dose. However, none of the Covid-19 VE results you have read about include all these cases!

All studies have imposed a case-counting window that always starts X days after the second dose. X is most often 14 days but sometimes it's 7 days, sometimes it's 28 days, or whatever the researchers deemed appropriate, frequently after computing VE for different window sizes.

The problem is that in observational studies, for the unvaccinated group, there is no such thing as 14 days after second shot (because they did not get any shots, unlike those in the clinical trial who got placebo shots). In many studies, the case-counting window is applied asymmetrically to the vaccinated group (reducing the case count) but not to the unvaccinated group.

Lataster pointed out another problem that creeped in with later studies. Many prominent studies started to define the "fully vaccinated" group as those having two doses (within some allowable dose duration) and who did not get infected prior to 14 days after the second shot, and then label the complement of the fully vaccinated group as the "unvaccinated" group.

This creates a host of problems. For the sake of discussion, let's focus on one subset: the group of doubly vaccinated people who got infected during the first 14 days after taking the second shot. In our original example, these cases are taken out of the case count in the vaccinated group. Effectively, these cases disappeared from the analysis.

In Lataster's example, these cases did not disappear. They got shifted into the unvaccinated group because that group now included doubly-vaccinated people who got infected in the first 14 days.

***

The decision to apply a case-counting window is tragic. It lands researchers into a dead end. This is a point Peter and I made in our reply to Lataster's response (here, paywalled).

The key of understanding our argument is to pay attention to both numerators and denominators in the case infection rates, which underlie the VE formula. The numerators are the cases, and the denominators are the subjects from which cases accrue.

In typical clinical trials, the treatment populations are fixed in advance. In an intent-to-treat analysis, one must use the treatment groups as originally randomized. Even in a per-protocol analysis, we remove subjects who did not complete the treatment (at the risk of adulterating the randomization). This procedure is different from removing subjects who completed the treatment (2 doses), and then got infected (i.e. have the primary outcome) within X days of finishing the treatment.

However, if the analyst uses the per-protocol population, and also applies the case-counting window, then these subjects who got infected during the first 14 days disappear from the numerator (case count) but not the denominator. In effect, they have become "immortal". Further, in observational studies, for which we cannot apply the case-counting window to the unvaccinated group, the analyst has made some immortals in the vaccinated group but not the unvaccinated group!

In Lataster's example, not only are the cases moved to the "unvaccinated" group but also the subjects. The analyst has taken the doubly-vaccinated person who got infected during the first 14 days out of the vaccinated group and added this person to the unvaccinated group. The case count follows along. This procedure makes me very uncomfortable. What happened here is that the analyst altered the treatment group assignment after looking at the primary outcome (Covid-19 cases).

This is why I called this a dead-end. If you just move the case not the group assignment, then you insert an immortal bias. If you move both the case and group assignment, you're manipulating the group assignment based on observing the primary outcome of the analysis. And, as Peter and I explained in our response, we are not convinced by arguments for case-counting windows.

***

Another part of Lataster's comment concerns empirical evidence that showed negative VE. I'm unwilling to go as far as he does because those negative VE numbers are observed in subgroup analyses for selected studies. Much more work needs to happen before we can establish generality of those results.

 

Rat czarina claims numbers don't lie

One of the most astounding news of 2023 must be the creation of the position of the "rat czarina" by the New York City mayor. Just last week, a friend was asking if I have noticed any improvement since she began her reign. I could not think of a single thing she has done. Well, this New York Post article tells us her accomplishments (link). She said "numbers don't lie".

She apparently created a new rule that says garbage cannot be put out on the sidewalk before 8 pm. As a result, "rat complaints to 311" went down by 26 percent compared to last June.

I immediately have a few questions: first, what was the level of April and May rat complaints compared to last April and May? also, what was the level of June all complaints compared to last June? what is the year-on-year variability of these levels?

Assuming that the stated causal relationship holds, the question is why? Presumably, when the garbage arrives later, the rats show up later, fewer people are outside to witness them, and so fewer complain. Does that solve the rat problem? Or just the rat complaint problem?

***

The next new rule is going to be forcing buildings to put their bags of trash inside storage containers.

The NYC mayor explained the logic of this rule with this amusing quip: "rats love garbage bags." So I guess he thinks rats hate storage containers.

I'm not sure they understand the problem. The storage bags would have been fine if people aren't dumpster-diving. You put something out there on the streets, and someone will show up to tear open the bags looking for stuff - food, stuff that people throw out to make space, empty beer cans, etc. This won't change if the bags are put inside storage containers!

 

It's random because we say so

Some of you are wondering why I haven't written about Covid-19 studies recently. Lamentably, the quality of such studies has not improved. It may have deteriorated even more. I wrestle with whether it's a waste of time to read them. The problems with many studies are apparent from the first page: how do they pass the peer review process?

This post concerns a half page of a recent jourmal publication called "Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data" (link). It's only half of one page because there are already enough problems with what I read that I don't see the point of continuing.

A friend sent me this paper because the authors share my concern about significant misrepresentation of vaccine effectiveness in observational studies of Covid-19 vaccines. (Here is a post about my paper published last month with two co-authors that highlighted three common biases that are not properly handled in studies.)

My comments are limited to the section with the header "Sequential Target Trial Emulation" on page 2 of the paper. In this section, the authors describe the first of two proposed analytical devices for dealing with biases in observational data.

For any observational study, there must be a well-defined starting and ending date of study enrollment. For each day in this enrollment window (and for each region), they pretend that a new target trial is being launched. This is how they put it:

on each day and in each region, eligible persons who have not yet been vaccinated are randomly assigned to receive immediate vaccination or to remain unvaccinated throughout follow-up and are then followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first

They then suggest that each daily trial can be analyzed on its own, and then the results averaged to obtain an overall effectiveness. They claim this outcome is representative of the "intention to treat" effect in a hypothetical randomized clinical trial. That's a bold claim! (Not even Pfizer, Moderna, etc. claimed to have produced ITT vaccine efficacies even in their RCTs.)

I'm not making this up. They literally claimed that "The trial-specific estimates of the effect of vaccination from each sequential trial can then be combined to estimate an observational analogue of the intention-to-treat effect of assignment to the intervention."

***

Why am I shaking my head violently?

Their statement that people are "randomly assigned to receive immediate vaccination or to remain unvaccinated" is unserious.

We are talking about observational data - data analysts poking around in a database after the fact. It doesn't matter what analytical technique is used - it remains the case that the vaccinated people self-selected the treatment. They were not randomly assigned to receive immediate vaccination.

In fact, the authors contradicted themselves in the very next sentence when they explained (my bolding):

Each calendar day is considered as time zero for a new emulated trial, with persons who are determined to be eligible assigned to the vaccination group if they were vaccinated on that day or to the no-vaccination group if they were not vaccinated,

For the vaccinated, there is no "assignment", let alone random assignment. Whoever decided to get vaccinated lands passively in the treatment group for that day. Because the vaccinated are self-selected, the unvaccinated are self-selected as well. Whatever selection biases exist in the raw data persists in this ex-post experimental setup. This step absolutely does not support presumption of random treatment assignment.

If this procedure resulted in random treatment assignment, then the original dataset contains no biases, which annuls the central premise of the entire paper.

***

Of course, they aren't taking every unvaccinated person. So the data analyst actively selects who comprise the unvaccinated group. The authors give two mechanisms for selecting unvaccinated.

The first is random selection. I had to take a deep breath before reading on. I doubled back to see if I read this correctly. "One or more persons can be selected at random from all eligible persons who were unvaccinated on that day and in that region, with exclusion of unvaccinated persons selected for a comparison made on a previous day."

If they randomly selected unvaccinated, then the random sample of unvaccinated would have the same statistical composition as the population of unvaccinated. This ensures no correction for any kind of biases. And yet at the end of this process, they claim to have found the ITT effect as if a randomized clinical trial were run!

The second mechanism for selecting unvaccinated is by matching on key "characteristics that reflect Covid-19 risk". This method is definitely more acceptable as it at least attempts to make the two treatment groups more comparable.

In practice, the matching method faces several big challenges (many of which I have discussed before on this blog):

• the matchable population is a small proportion of the total population, e.g. most studies using matching heavily undersamples older people and oversamples younger people. The method is guaranteed to undersample demographic segments in which vaccination rate is high, and oversample demographic segments in which vaccination rate is low
  
  
• VE in the matched population may be different from VE in the general population. The estimated VE in the matched population is surely not the ITT for the entire population
  
  
• most important confounders are unmeasured (e.g. taking other mitigation measures, vaccine hesitancy, exposure to virus). Matching does not infer balance on non-matching variables, a key reason why ex-post matching on observational data cannot replace randomized treatment assignment
  
  
• most countries enforce highly targeted vaccination programs which generate irreducible biases, e.g. if older people are encouraged to get vaccinated first, and have done so in large numbers (say 80%), what's the chance that the 20% unvaccinated older people are comparable to the other 80% vaccinated older people?

***

There is another aspect of the observational studies that are buried away from the summary of the methodology. In the sentences cited above, they said that the study subjects were "followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first." That surely cannot be true!

Another event that happens is vaccination. For people in the unvaccinated group, by definition, they have not been vaccinated up to the day of enrollment; however, many or most of them would get the vaccine during the follow-up period. How did the data analyst deal with this event? We are not given a hint in this methodology section.

Upon vaccination, clearly the follow-up period for the unvaccinated person must end. We simply don't know the counterfactual: what would have happened if the person did not get vaccinated? If we follow the time series of vaccine efficacy from day of vaccination forward, more and more people in the unvaccinated group are dropped from follow-up because they got themselves vaccinated. The people who stay in the unvaccinated group for a long time are the vaccine holdouts, who are clearly different from the general population, e.g. they are much less likely to take other precautions such as wearing masks, distancing, avoiding crowds, etc. How well does matching on demographics and basic health data control for these differences?

If those differences are not adequately accounted for, then the right side of the VE curve cannot be trusted.

***

Also, note that variability is artificially suppressed in this type of study. Anyone who got vaccinated during the study period appear once in the vaccinated group on the day of vaccination, but also may appear in the unvaccinated group on any day prior to vaccination. Unlike a typical RCT in which there are no overlaps in the group membership, in this type of observational studies, the majority of the people show up in both groups. So, if we subsequently run regression analysis comparing the two groups, adjusting for demographics and other factors, the variance between groups is markedly under-stated!

 

 

Long story behind simple metrics

In a recent post on the dataviz blog, I featured a project by FiveThirtyEight on the impact of potential state-specific abortion bans. For comments on the visualization, please see the other blog post. This post discusses the analytics behind the visualization.

The data journalists measure the impact on women wanting abortions along two dimensions: "distance" and "congestion". Distance is the distance to travel to the nearest abortion clinic, and congestion is represented by a proxy, which is the service capacity of the said abortion clinics. (To be precise, the authors said they counted "the number of reproductive-age women that each clinic serves, which serves as a rough stand-in for appointment availability.")

I'd rather they call a spade a spade. I don't think much is lost by calling the second metric "capacity". Congestion is a result of matching supply and demand while they only have data on supply.

***

The smoothness and niceness of the dataviz hide some of its underlying assumptions (and thus biases). As consumers of dataviz, we must recognize them.

Look at the following pair of map excerpts. They both feature Arizona and its neighbors. The left panel shows the current state while the right panel is the projected state if Arizona should ban abortions. I have highlighted Pima county in which there is an abortion clinic.

In this example, the county went from the lower left corner of the color legend to the upper right. Women who have easy access to a clinic today would have to travel much further (266 miles) to find one in the future. In addition, that substitute clinic in New Mexico has a much lower capacity than the one in Pima county.

It appears that they are plotting the absolute values of distance and capacity. How about plotting the relative changes? How much further would women have to travel relative to the current state? What's the difference in capacity between the current clinic and the substitute clinic?

***

Clicking around, I dwelled on Coconino county, which showed something odd.

Notice that the color for Coconino county went from dark green (current) to lighter green (post-ban). It appears as if Coconino residents would benefit from an abortion ban. How could it be?

Digging into this reveals assumptions that the dataviz designer must make when producing such a chart. Clark county on the left panel is rated as low congestion (i.e. high capacity) while Maricopa county is rated as medium capacity. Both counties are rated low distance from Coconino county. And yet, the algo assumes that women in Coconino prefer the facility in Maricopa.

We don't have detailed data on every woman seeking service at these clinics. And if we do, we'll find that they don't all go to the same clinic. The designer is aiming for the average behavior here. Someone else may decide to connect Coconino to Clark county in the current state. These two designers would have justified their algos by arguing their assumptions.

In the case of FiveThirtyEight, there can be a number of reasons why Maricopa is preferred: state loyalty, insurer preference for in-state providers, the drive to Maricopa may be nicer, safer, and/or faster than Clark county, the capacity in Maricopa may be sufficient, the types of services offered by the two clinics are not the same, etc. etc.

It would be interesting to learn what proportions of the counties have the same predicted outcome as Coconino, i.e. the women would have benefited from an Arizona ban, at least according to the two criteria being considered.

***

There is another assumption revealed when comparing the pre- and post-ban Arizona. Notice that the colors of all surrounding states do not change at all.

This highlights what I pointed out before. "Congestion" is not congestion. It is "capacity". If it is congestion, then the fact that a whole county's women from Coconino are now expected to show up at Clark county's clinic would have caused more congestion in Clark.

***

In short, in order to make this data visualization, the designer must make assumptions about people's behavioral choices. There is an underlying optimization model that decides what women would do depending on where they live, and which states have bans. Not having such a model creates so many possibilities that the graphic would become too complex.

By how much did observational studies over-estimate Covid vaccine effectiveness

I'm happy to announce that our paper on limitations of observational (i.e. "real-world") studies of Covid-19 vaccines has been published in the Journal of Evaluation in Clinical Practice (link). I'd like to thank my co-authors (Peter and Mark) and reviewers for feedback that greatly improved the presentation, and the journal editors for the courage to print it.

In the paper, we offer three examples in which standard methods used to analyze observational data produce vaccine effectiveness (VE) estimates that are wildly off the mark. Specifically, we show that a hypothetical vaccine with 0% efficacy will be rated as 50 to 70 percent effective. The magnitude of the biases surprised us, and should surprise you too. That said, we are not claiming that Covid-19 vaccines are ineffective - as I explain below, the device of a 0% effective vaccine allows us to work around the lack of data disclosure.

The motivation for writing this paper is to encourage better science. Observational studies are hard, and the quality of Covid-19 related observational studies has been mediocre.

Two major obstacles to writing this paper were (a) the pathetic level of data disclosure for major studies that have had tremendous impact on public health policies and (b) the lack of a ground truth - no one knows the real vaccine efficacy. We found a way to work around these challenges.

For (b), we analyzed the case of the 100% ineffective vaccine. This device is very useful because if the vaccine is ineffective, it is ineffective for all subgroups of people, at all times, etc. If we assumed any other level of vaccine efficacy, say 50%, we'd have had to make a huge list of other assumptions as well. For example, there is no rationale to assume that if VE is 50% in aggregate, then it is 50% for every age group.

For (a), we opportunistically found papers that disclosed just enough data to prove the existence of specific biases. For example, to study the effect of background infection rate bias, we require data on weekly infection rates for certain subpopulations within an analysis window, which we found in a Danish study but are not typically disclosed by researchers. We didn't find a single paper that disclosed all the data required to study all three sources of bias, thus for each bias, we used data from a different reference.

In each example, we show that if the ineffective vaccine were tested in a randomized clinical trial, the typical analysis applied to RCTs would conclude correctly that VE is 0%; however, if the ineffective vaccine were analyzed using methods that were popular in early 2021, the resulting VE would erroneousy land in the 50-70% range! The magnitude of the over-estimation surprised us.

The full paper is here. Spread the word!