Big Data, Plainly Spoken (aka Numbers Rule Your World)

Running out of matches

There is one other curious thing about the matching procedure used to pre-process observational data in the paper analyzing the effect of the Omicron booster from my previous post (link).

As noted, in the table showing pre-matching statistics, there were 3.8 million people who didn't take the booster against 588,000 who did. So, there were about 6.5 non-boosted people for each boosted person. The matching task is to find one non-boosted person to pair up with each boosted person. We therefore expect the vast majority of the 3.8 million non-boosted people to be dropped from the study population.

Now, in the article itself, it said that 511,352 unique control participants were matched. This implies that over 10% of these matched non-boosted people were matched to multiple boosted persons. This usually happens when there aren't enough untreated persons available for matching.

However, they had on average 6.5 untreated available to match to each treated. So there didn't run out of unique untreated people. The next possibility is that the treated and untreated groups were sufficiently different along certain dimensions that the same untreated person must be used multiple times. For example, if there were 10 women aged 95+ with certain set of preexisting conditions in the boosted group, and there were only 2 such people in the non-boosted group, then the same two non-boosted women would appear on the other side of the pair for all 10 matched pairs. In other words, the more re-use there is, the stronger the suggestion that the two groups had some major differences.

In such circumstances, I'd prefer to drop the treated person from analysis rather than use an untreated person multiple times.

Think about this scenario. Let's say there are a particular demographic segment with 5 treated and 1 untreated. The one untreated person will be part of all five pairs. Now, assume that the untreated subject died. That death would have been replicated five times after matching. The matching turned a single death into a 100% death rate in that segment.

Alternatively, assume the untreated person did not get Covid during the observation window, but 2 of the five treated did. Replicated five times, the estimate for the untreated group will be zero cases, out of five.

Matching sounds simple, but details matter. Most published studies don't disclose nearly enough for an outsider to figure out the quality of matching.

Loose lips sink hearts

A reader sent me a new study from the Veterans Health Administration about the Omicron Covid boosters, introduced in September 2023 (link). These boosters were one of the first of many “boosters” that were reformulated vaccines targeted at new coronavirus variants, all of which were approved by the FDA without conducting clinical trials. The VHA researchers used real-world data to measure the vaccine effectiveness (VE) of these boosters. In short, they found these boosters lacking, with negative VE against cases, and low VE against hospitalization or death, topping out in the 30% range but waning to below 10% within the first six months.

Reading this report brings back unpleasant memories because they contain some of the same loose language that encourages sloppy thinking, misleads people, and hides iffy science. I’ll be liberally quoting from the above linked paper but the same type of writing can be found in many similar papers I have read during and after the Covid pandemic.

First, a quick refresher on observational data. Researchers pull health records, which include people’s vaccination and medical history. Without a clinical trial, people self-selected into treatment, that is to say, they decided on their own to take the booster shot (or not). Therefore, any observed difference in outcomes could be partly due to inherent differences in the comparison groups (technically called confounding). The random assignment of treatment, as implemented in a clinical trial, ensures that the comparison groups are statistically identical, removing this complication. A good analysis of observational data starts with a reasonable model of the self-selection of treatment, followed by effective adjustments to deal with these confounders.

***

Now, I’ll point out three of the worst cases of loose language in these Covid studies:

“Exact matching”

One of the more popular techniques for dealing with confounding is “matching”, a procedure designed to artificially create pairs of treated and not treated subjects, with the hope that they might behave as if treatment were randomly assigned. To match pairs of subjects requires a set of matching variables, typically variables such as gender, age, and preexisting health conditions.

There are many matching methods. One such method – popular because it is easy to describe – is known as “exact matching”. For example, if we have a vaccinated subject who is a 59-year-old male diabetic, we want to pair him up with an unvaccinated subject who is also a 59-year-old male diabetic. Each matching variable must match exactly, age to age, gender to gender, diabetes condition to same.

But … in the cited paper, what is tagged as “exact” is not actually “exact”. For example, matching on age is done using broad age groups, rather than single-year age. Only three age groups were used: 18-64, 65-74, and 75+. So someone who is 30 can be matched to someone who is 60, which is not what one can reasonably describe as an exact match.

Similarly, another matching variable is called “CAN score”, which is a predicted probability of dying within the next year. This probability score is also grouped into three broad buckets (0-50, 51-89, >=90) before matching so that someone who has 51% chance of dying can be matched to someone with 80% chance of dying. In this paper, that’s an “exact” match.

“Unvaccinated”

Every Covid study compares a “vaccinated” group to an “unvaccinated” group but in almost every case, the unvaccinated have taken at least one Covid vaccine shot – they just haven’t taken the particular vaccine shot that is the subject of the study.

In the above linked paper, they studied the Omicron vaccine shot. But to be eligible for inclusion, a person must have “received at least one documented Covid-19 vaccine in the VA healthcare system at any time in the past”. Therefore, what they label as “unvaccinated” are vaccinated people who decided not to get the Omicron booster. What’s wrong with calling the treated group “Had Omicron booster” and the untreated group “No Omicron booster”?

Take a look at this table header from the article’s Supplement:

If you haven’t read all the details in the paper, you’d assume that they are comparing one group of people who have taken an updated Covid-19 vaccine against another group of people who have had “no vaccination”. But then you’d have misread the study.

The conclusion

Almost all conclusions of these Covid studies mislead through over-generalization.

In this paper, the conclusion stated:

“COVID-19 vaccines targeting the XBB.1.5 variant of Omicron were not effective in preventing infection and had relatively low VE against hospitalization and death, which declined rapidly over time.”

This conclusion only applies to (a) the people in the VHA system who (b) meet the eligibility criteria who (c) look like someone in the matched groups.

If a clinical trial were done by the VHA with the same eligibility criteria, then the (a) and (b) restrictions would still hold while (c) could be dropped because there would have been randomization in place of matching.

One key fact that is always glossed over is that the matched subpopulation does *not* look like the population. In matching, each Omicron vacinee is matched to someone who didn’t take the booster. So after matching, the no-booster group looks just like the booster group. But, the no-booster group post-matching does not look like the no-booster group pre-matching! The magic of matching comes from dropping no-booster people who have no counterpart in the booster group.

All of the above is reflected in the statistics included in the paper.

The following table (relegated to the Supplement) displays the pre-matching statistics.

One immediately learns that the no-booster subpopulation is 4 times as large as the Omicron subpopulation, that self-selected Omicron subjects are much older, less likely to be female, much more likely to have pre-existing comordibities, more likely to have many recent primary care encounters, and also more likely to have gotten a Covid vaccine within the last year.

What happens after matching? The two groups now have the same number of subjects. The column showing the Omicron group doesn’t change while the no-booster column looks almost the same as the Omicron group. (This is found in a Table in the paper.)

Take age groups for example, the post-match, no-booster distribution is 27%, 33%, 40% for 18-64, 65-74, and 75+, which is exactly what it is for the pre-match, Omicron group. But the pre-match, no-booster group’s age distribution is 48%, 23%, 29%! Evidently, those dropped from the analysis who didn't take the booster don't look like those who were matched.

Importantly, the result of this analysis of matched pairs can shed light only on the effect of the Omicron booster on those who choose to take it, which is only about 25% of the entire population (under study). Matching is needed precisely because the no-booster group have different characteristics than the booster group (as seen in the pre-matching statistics); therefore, whoever is in the subgroup of no-booster who look like those in the booster group cannot be generalized to the unmatched no-booster people.

I'm not trashing the matching methodology. It's a very useful technique for analyzing observational data. It just doesn't generalize in the way that many researchers imply.

***

It's sad that paper after paper pushes misunderstanding by using loose language. We can and should do better.

 

Know your data 40: omnipresent listening

Apple markets itself as the tech giant that is most protective of user privacy. One reason to believe this claim is that unlike all the other tech companies, Apple charges premium prices for its products and is not as reliant on advertising as its revenue source. Nothing in this post disputes this assertion although we will wonder how low the bar is to quality as "most".

Apple recently settled out of court a lawsuit that alleges that Apple's Siri secretly records people's conversations, then extracts "insights" which are sold to advertising customers. It's a scenario that most of us recognize is possible, and perhaps many of us thought couldn't possibly be happening, not on an Apple device.

The allegation is that some users were delivered "eerily accurate targeted ads that appeared after they had just been talking about specific items like Air Jordans or brands like Olive Garden".

***

Of course, any interactive technology requires the tech to capture the user's instructions or reactions and thus we expect the tech to be recording, whether it's audio or video. This is true of Siri, other voice assistants, any home devices you can interact with, gaming and TV monitors, etc.

So, if one chooses to use these features, one should expect omnipresent listening.

Just think about the idea that "Hey, Siri" is used to activate Siri. If the phone is not constantly listening to you, how does it pick up "Hey, Siri"?

The concern is really about the lack of user control, or the disrespect of user intention (what has been euphemistically called "unintentional" here). I laugh every time the word "unintentional" is used. There is almost nothing that is unintentional in tech. The recording has to be triggered by some instruction and that instruction is written up in the code, a deliberate act. It's possible that the recording is set to be universal regardless of user intention or controls but that's not what is being admitted.

The allegation is that Siri was supposed to record conversations only if the user utters the words "Hey Siri" but in fact, conversations were (sometimes) recorded even when those words were not spoken.

Further, one doesn't typically hand over entire recordings to advertisers. The recordings must be turned into transcripts, then insights are extracted (e.g. person A mentioned brand X), the person A has to be identified so that the advertiser can send ads to that person's devices, etc.

The amount of code needed to execute the entire targeted advertising program, starting with recording the conversations and ending with delivering the ads, is substantial and involved. In my view, if such code exists, it is intentional.

***

It just so happens that while I was writing this up, a friend called me up and told me that his Instagram (that's a Facebook property) started flooding him with ads from a particular VPN brand after he recently asked me for VPN recommendations. He was quite alarmed and annoyed. He said he only wrote down the recommended names on a notepad so the only way Instagram knows we talked about that vendor would be through listening to our phone call.

That scenario is certainly possible but hard to prove since there is so little transparency when it comes to technology. In fact, any number of trackers may be collecting the data, which eventually find their way to Instagram.

Without whistleblowers or discovery, it's challenging to prove the case against Apple in court. But since Apple is settling, we might not hear about this again.

 

 

The science behind food recalls

McDonalds is having a moment. It's got our attention when Trump used it to make a political statement, then it got implicated in a food recall. The FDA thinks that at least 75 people have gotten sick and one person died after consuming burgers, and the culprit is thought to be the raw onions inside the burgers. (link)

***
In Chapter 2 of Numbers Rule Your World (link), I praised the science of tracing food outbreaks but readers also learn that these investigations are far from infallible.

The challenge arises because people eat a variety of foods, and when only a few people are getting sick, it's hard to tell where the E.coli has come from. The initial investigation is done using questionnaires to collect as much information as possible about what people consumed. Even this is hard, because of timing issues. Once the data are collected, then some math is used to infer which foods are most likely the culprit.

According to this CDC release, they were able to interview 42 of the sick people, and all of them have eaten at McDonalds prior (how long ago?) before getting ill. Further 86% of them ate a quarter pounder. That's all they have released so far. It sounds very damning and definitive. Yet it is misleading.

We are missing information on any control subjects i.e. similar people who didn't get sick. It's possible that similar people also have high probability of eating at McDonalds, and that a high proportion of McDonalds patrons order quarter pounders. We need a case-control study, and the press release only mentions the cases.

In a perfect world, everyone who got sick ate one common thing and everyone who didn't get sick didn't. But we don't live in a perfect world. So, the result of the investigation is an educated guess.

For example, as far as I could tell, no reporter has asked the hard questions yet: they were first blaming quarter pounders, then they said it's the raw onions inside the quarter pounders... but, does this imply that McDonalds have different supplies of raw onions, and they only use Taylor Farms on quarter pounders, and use a different source of onions on other items?

Also, according to the article, Taylor Farms also supply onions to other fast-food chains, so why is the current outbreak limited to McDonalds? How come the people who ate the same onions supplied to Burger King didn't get ill?

***

Sometimes, these investigations chance upon very good leads, and it is possible to trace the contamination back to a specific batch from a specific plot of land. Sometimes, they might even be able to find the source of E.coli at the farm.

This investigation isn't that kind, though. It seems like it's so far mostly guessing based on scarce data.

***

What else do we know about food recalls from Chapter 2?

Food recalls are very expensive because everyone is throwing stuff away based on "an abundance of caution".

The benefits of food recalls are hard to nail down.

Most contamination events are temporary, and affect only certain batches of produce during a limited time window. Thus, whether or not there is a recall, these perishable goods will leave the food supply eventually, and new cases will vanish. It's very difficult to ascertain if the recall prevented any cases. Indeed, given the time it takes to investigate, and organize recalls, it's quite likely that the crisis is over before the recalls even start.

That's not to say food recalls have zero benefits. It's just not as obvious as one thinks.

 

If it's an RCT, it can be trusted

I've written often about the problems with conducting observational studies. We like to hold up randomized controlled trials (RCTs) as the "gold standard," and if a study is described as an RCT, and especially if its design is pre-registered, we don't ask many questions.

This point of view is becoming unsustainable, based on a recent Nature article.

The gist of the article is that experts now believe at least one out of every four RCTs contain so many problems that their findings should be ignored, leaving the possibility also that some of these RCTs may be entirely fake. In coming to this conclusion, these experts sought and received the underlying individual-level datasets.

These problematic or fake RCTs make their way to systematic reviews (aka meta-analyses), and throught those, they sometimes influence medical practice. Systematic reviews are summary articles that weigh evidence from all studies on a given topic, e.g. whether sugar helps reduce death after head injuries.

In a particularly egregious example, a late Japanese bone-health researcher has had over 100 studies retracted due to fraud, but his papers have appeared in 88 different systematic reviews, half of which would have arrived at a different conclusion if his studies were omitted.

Besides, there is no universal requirement for authors of systematic reviews to revise their analyses should included studies be later retracted. What a mess!

One researcher estimated 20-30 percent of the papers included in systematic reviews in women’s health are suspect. Even when the authors of these systematic reviews have been alerted to the retracted studies, many have ignored pleas to update their findings.

When editors reject studies because of data problems from one journal, the same studies almost always eventually get published in a different journal, “sometimes with different data to those submitted [to the first journal].”

Fixing the RCT problem requires a cultural shift, which does not seem to be forthcoming.

***

The Nature article catalogs a number of objections to research that exposes RCT fraud. Here is a list:

• Privacy concerns of releasing individual-level data
• Participation agreements that do not explicitly include the right to share data
• The logistics of archiving the datasets
• The effort is not worth their time
• The potential harm of false positives
• Ethical concerns related to excluding studies, e.g. wasting the participants' time

Efforts to mandate data sharing have so far been thwarted.

***

Sad to say, the proportion of bad or fake RCTs is substantially higher than 25%. How do I know?

One expert examined over 500 RCTs, and only successfully obtained the individual-level datasets from 150 (30%). In the 30% with detailed data, he found serious problems with 25% of them.

He also looked for problems in the other 70% of the studies, but could only find issues in 1 or 2 percent of them. Paradoxically, this suggests data non-disclosure is positively correlated with trust. But not really. This result points to the fact that typical data summaries in publications hide major problems, including fake data. There isn't much the expert could have done to verify the study findings when the underlying data were not made available. I encountered this issue frequently when looking at Covid-19 vaccine studies.

If we generalize the 25% fraud from studies with data disclosure to those without, then we could conclude that overall 25% of RCTs are suspect. This conclusion assumes that whether the researchers agreed to disclose data is independent of whether their studies are fraudulent. I don't like this assumption. A more sensible assumption is that fraudulent researchers are highly unlikely to disclose their data. Thus, among the studies that don't disclose their data, we should expect a higher proportion of fraud.

Observational studies that don't include everyone intended to receive the treatment

Today, I return to the Clalit booster study. I’ve written three prior posts about the study, which was the catalyst for regulatory approval of booster shots of mRNA vaccines for Covid-19 at the turn of 2022. In the first post, I showed how the methodology - common among Covid-19 vaccine studies - contains obvious biases that researchers and their peer reviewers overlooked. (link) The booster study featured a further unexplained wrinkle – some subjects are excluded after their infection status is known (link). In a third post, I developed a sports analogy to ponder what kind of effects such analytical procedures may have on the results.

This post describes further obvious biases in the study design. There are two classes of biases that affect observational studies in which we cannot randomly apply treatment to subjects. The first class of biases occurs when the two treatment groups are not statistically identical, except for the presence or absence of treatment. The second class of biases occurs when the study population does not look like the population to which the treatment if approved would be applied.

As explained in the first post, this study is unusual in that 90% of the study population appeared first in the no-booster group and later in the booster group. Thus, if we investigate covariate balance in the usual way, the two groups would look close to identical.

So the more interesting question is whether the study population represents people to whom Clalit intends to promote booster shots should they be approved. It’s tough to answer this question without access to the data. We can address a slightly different question: is the study population a random sample of Israelis 50 years old and above?

The answer is most likely no. According to OurWorldinData, the proportion of Israelis 50 year old and above who have received at least one booster dose during the study’s time frame was between 61-80%. And yet the study reported that 90% of the subjects got booster shots during the study period. So, the study population had greater inclination to take the booster shot than the general population.

In fact, fast forward to April 2022, seven months after the study ended, and the last month for which OurWorldinData compiled data, and we learn that the proportion of 50+ year old who have received at least one booster shot was stuck at 73-86 percent, still below the 90% recorded in the study.

***

Looking at the list of exclusions from these types of studies is highly informative. The key question to ask is whether people excluded from these studies will also be excluded from the rollout of the booster. Seemingly the answer is no – the booster will be recommended for essentially everyone.

This practice makes little sense to me. If the excluded people will be given boosters, why should they be excluded from the analysis?

To make this more concrete, here’s the first exclusion criterion: anyone who have ever been infected with Covid-19 prior to the study start date. I have never heard a public health official say don’t take boosters if you had been infected before; quite the opposite – they usually claim that prior infection does not confer immunity, and recommend prior infected to take the boosters.

What might be the effect of removing prior infected? If the public health claim is correct, then prior infection does not offer extra protection to this subgroup, and therefore including them doesn’t affect the effect of the booster shot. As a subgroup, those with prior infection may look different from those without, changing the baseline infection rate but that's not a reason to exclude. If the public health claim is false, that is to say, prior infection does confer some immunity, then any effect of the booster would be attenuated in this subgroup, and excluding them improves the reported result.

Another exclusion criterion involves those whose second vaccine shot (in the original two-shot Pfizer regime) was administered fewer than 5 months from the start of the study period. Once the booster is approved, is there any enforcement of this “rule”? Are people forced to wait at least 5 months before getting the booster?

Exclusions are always present in these studies, and not enough attention has been paid to the biases that exclusions create. Excluding prior infections mean only people without prior infections are in the study. This subset includes people who are health-seeking, and those who follow non-vaccine mitigation measures seriously (masks, distancing, stay home, etc.). Meanwhile, those who ignore mitigation measures and take risks are more likely to get infected, and thus get kicked out of the study.

The other exclusion also directly inserts a bias into the study – as it selects people who are less vaccine hesitant and thus take their vaccine shots earlier rather than later.

A sports analogy for data processing rules

I'm been mulling over how to explain the potential impact of strange data processing procedures used in the observational studies of Covid-19 vaccines. These are by no means standard operating procedures.

And I think I've landed on one explanation that is tantalizing.

***

Imagine a football (soccer) match that is tied 0-0 and heading into overtime. Overtime is a 30-minute affair, played through, and not sudden death. The overtime period is divded into two 15-minute halves with a short break in the middle.

That is the standard operating procedure.

***

Now, we are going to change the rulebook.

The first additional rule is that any goal scored in the first 10 minutes of overtime doesn't count. This rule is inspired the "case counting window" used in the Covid-19 vaccine studies. We're justifying it because players need time to get back to their optimal condition after the exhausting 90-minute regulation time. Therefore, it is not fair to count goals scored at the start of overtime.

The second additional rule is that any goal scored in the first 10 minutes of overtime by the visiting team doesn't count but those scored by the home team counts. How should I justify this rule? Well, to establish the need for recovery time, we take vitals of the players. The measuring machines are only installed in the home locker rooms and not the guest locker rooms, and thus, we are unable to determine if the guest players are at their optimal condition. This rule is inspired by not applying the case counting window to non-vaccinated people since we can't determine when they took the second shot (unlike the situation in clinical trials, when non-vaccinated people are given placebo shots.)

The third additional rule divides the first 10 minutes into two parts, and stipulates that if someone scores in the latter part (i.e. minutes 6-10), then they are immediately kicked out of the match, and their goal doesn't count. This rule is inspired by the special exclusion criterion disclosed in the booster study I've been discussing (link).

This third rule is complicating as it modifies the previous rules. Taken together, we have the following scenarios: for the home team, if they score in the first 5 minutes, the goal counts; if they score in minutes 6-10, the scorer is kicked out of the match, and the goal is disallowed. For the away team, if they score in the first 10 minutes, the goal is always disallowed; in addition, if the goal is scored in minutes 6-10, the scorer is ejected.

***

One can easily get lost in the details of pharmaceutical studies and lose sight of the implications of the methodologies. Transferring the context to sports helps me grasp the effects of these data processing procedures. Hope this helps you too.

PS [8-4-2023] In my haste to put down my thoughts relating to this analogy, I skipped a rule. This is a Rule #2.5. This rule says that if a goal is scored in the first 10 minutes of overtime by the visiting team, it is counted as a goal scored by the home team. In Rule #2, the away-team goal is just disallowed; in Rule #2.5, the goal is added to the home-team tally. This rule is inspired by the booster study, in which they kept people in the no-booster group for 7 days after they took the booster.

 

 

One big problem with booster studies (and observational studies of Covid19 vaccines)

I'll be writing a number of posts about observational studies of the Pfizer booster, published by the Clalit health insurer in Israel (back in December 2021), which is the primary source supporting the effectiveness of the booster shot (3rd shot). These studies provide fodder forever for classes in observational studies because (a) there are many nooks and crannies in which residual biases hid, and (b) researchers made many assertions (wrong, of course) about having corrected all biases using the simplest, most conventional methods.

This first post points out the biggest problem with such studies, which is perhaps impossible to correct.

Let's first review the basic setup of such studies. (The devil is of course in the details, and I'll dive into those in future posts.)

Each study has a well-defined start and end date. This is not a clinical trial, so no one proactively enrolls in this study. Instead, the researchers had access to databases with relevant data and extracted the study population from it.

The start date is chosen close to the launch of the booster shot in Israel, and therefore, at the start, almost everyone was in the "no booster" group. Over the course of the study, most of these same people have migrated to the "booster" group - when they took the booster. Even though the study spanned fewer than 2 months, almost everyone (90%) received the booster shot by the end date.

Here is the first thing one must recognize. This is not a typical clinical trial in which one person belongs to either the treatment group or the control group but not both. In this study, nine out of ten people featured in both the "no booster" and the "booster" groups.

So when subsequently, researchers compare the "no booster" group and the "booster" group, they talk about exposure time, rather than number of exposed. The exposure time of the "no booster" group are incurred mostly by people who eventually got the booster within the study period. This is key observation #1.

If we now hone in on an individual who got the booster shot during the study period, the entire observation time is divided into two parts: from start of study to the day of the booster ("no booster" exposure), and from the day of the booster to the end of the study ("booster" exposure). It is 100% certain that the exposure time in the "no booster" group precedes the exposure time in the "booster" group. This is key observation #2.

Moreover, observation #2 deos not depend on the date of booster, which varies by person. Since individually, each person is exposed first to "no booster" and then to "booster", collectively, the exposure time of the "no booster" group is skewed towards the start of the study period while the exposure time of the "booster" group is skewed towards the end of the study period.

What else do we know about the booster study (or, the majority of Covid vaccine studies)? These studies almost always begin during a peak infection period, which means the infection rate dropped during the study. This is key observation #3.

Together, our three observations mean that the exposure time associated with the non-booster group is skewed towards the front end of the study period, which also happens to be a peak infection period while the exposure time associated with the booster group is skewed towards the back end, when infections are much lower.

For the Israel Booster study, the authors essentially confirmed the existence of such bias with the following sentences:

"During this time [ed: study period],the incidence of Covid-19 in Israel was one of the highest in the world."

"The incidence of Covid-19 and thus exposure to SARS-CoV-2 changed during the study period."

Given that at the start of the study period, Israel's infection rate was "one of the highest in the world" and that the incidence and exposure subsequently "changed," this change represents a decrease in incidence and exposure. (The data can be found here. One of the charts shows the 60 and over age group.)

Reflecting the current state of medical "science," the authors didn't make the above statements in order to explain a source of bias. Instead, they used the first statement to explain why their study could achieve statistical significance on mortality due to Covid, which is a rare outcome, and they cited the second statement in order to make the false claim: "we assume that after adjustment for all covariates, including socioeconomic status, these changes had a similar effect in the booster group and the non-booster group."

Their "assumption" I just showed to not hold.

***

This is another example of the "background infection rate" bias that we described in our bias paper (link). In that paper, we used an example of a Danish study which also started when the infection rate peaked and then by the end of the study period, the infection rate had dropped to almost zero.

If the booster were to be completely useless, the average case rate in the no booster group would be higher than the average case rate in the booster group because of differential exposure time against a decreasing background infection rate. This leads to a high vaccine effectiveness when the effectiveness is assumed to be zero!

Adjusting for "all covariates" does not cure this bias whatsoever.

***

What can be done about such bias?

For one thing, better data disclosure. All these studies should show the distribution of exposure time, by treatment vs control group, over the study period. On top of that, we can superimpose the background infection rate. Such an analysis discloses the source of bias.

Next, attempt to do the same analysis for a different time period, especially one in which the infection rate trends in the opposite direction. In practice, this may be hard to accomplish if the public health department decided to launch vaccines around an infection peak.

It's random because we say so

Some of you are wondering why I haven't written about Covid-19 studies recently. Lamentably, the quality of such studies has not improved. It may have deteriorated even more. I wrestle with whether it's a waste of time to read them. The problems with many studies are apparent from the first page: how do they pass the peer review process?

This post concerns a half page of a recent jourmal publication called "Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data" (link). It's only half of one page because there are already enough problems with what I read that I don't see the point of continuing.

A friend sent me this paper because the authors share my concern about significant misrepresentation of vaccine effectiveness in observational studies of Covid-19 vaccines. (Here is a post about my paper published last month with two co-authors that highlighted three common biases that are not properly handled in studies.)

My comments are limited to the section with the header "Sequential Target Trial Emulation" on page 2 of the paper. In this section, the authors describe the first of two proposed analytical devices for dealing with biases in observational data.

For any observational study, there must be a well-defined starting and ending date of study enrollment. For each day in this enrollment window (and for each region), they pretend that a new target trial is being launched. This is how they put it:

on each day and in each region, eligible persons who have not yet been vaccinated are randomly assigned to receive immediate vaccination or to remain unvaccinated throughout follow-up and are then followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first

They then suggest that each daily trial can be analyzed on its own, and then the results averaged to obtain an overall effectiveness. They claim this outcome is representative of the "intention to treat" effect in a hypothetical randomized clinical trial. That's a bold claim! (Not even Pfizer, Moderna, etc. claimed to have produced ITT vaccine efficacies even in their RCTs.)

I'm not making this up. They literally claimed that "The trial-specific estimates of the effect of vaccination from each sequential trial can then be combined to estimate an observational analogue of the intention-to-treat effect of assignment to the intervention."

***

Why am I shaking my head violently?

Their statement that people are "randomly assigned to receive immediate vaccination or to remain unvaccinated" is unserious.

We are talking about observational data - data analysts poking around in a database after the fact. It doesn't matter what analytical technique is used - it remains the case that the vaccinated people self-selected the treatment. They were not randomly assigned to receive immediate vaccination.

In fact, the authors contradicted themselves in the very next sentence when they explained (my bolding):

Each calendar day is considered as time zero for a new emulated trial, with persons who are determined to be eligible assigned to the vaccination group if they were vaccinated on that day or to the no-vaccination group if they were not vaccinated,

For the vaccinated, there is no "assignment", let alone random assignment. Whoever decided to get vaccinated lands passively in the treatment group for that day. Because the vaccinated are self-selected, the unvaccinated are self-selected as well. Whatever selection biases exist in the raw data persists in this ex-post experimental setup. This step absolutely does not support presumption of random treatment assignment.

If this procedure resulted in random treatment assignment, then the original dataset contains no biases, which annuls the central premise of the entire paper.

***

Of course, they aren't taking every unvaccinated person. So the data analyst actively selects who comprise the unvaccinated group. The authors give two mechanisms for selecting unvaccinated.

The first is random selection. I had to take a deep breath before reading on. I doubled back to see if I read this correctly. "One or more persons can be selected at random from all eligible persons who were unvaccinated on that day and in that region, with exclusion of unvaccinated persons selected for a comparison made on a previous day."

If they randomly selected unvaccinated, then the random sample of unvaccinated would have the same statistical composition as the population of unvaccinated. This ensures no correction for any kind of biases. And yet at the end of this process, they claim to have found the ITT effect as if a randomized clinical trial were run!

The second mechanism for selecting unvaccinated is by matching on key "characteristics that reflect Covid-19 risk". This method is definitely more acceptable as it at least attempts to make the two treatment groups more comparable.

In practice, the matching method faces several big challenges (many of which I have discussed before on this blog):

• the matchable population is a small proportion of the total population, e.g. most studies using matching heavily undersamples older people and oversamples younger people. The method is guaranteed to undersample demographic segments in which vaccination rate is high, and oversample demographic segments in which vaccination rate is low
  
  
• VE in the matched population may be different from VE in the general population. The estimated VE in the matched population is surely not the ITT for the entire population
  
  
• most important confounders are unmeasured (e.g. taking other mitigation measures, vaccine hesitancy, exposure to virus). Matching does not infer balance on non-matching variables, a key reason why ex-post matching on observational data cannot replace randomized treatment assignment
  
  
• most countries enforce highly targeted vaccination programs which generate irreducible biases, e.g. if older people are encouraged to get vaccinated first, and have done so in large numbers (say 80%), what's the chance that the 20% unvaccinated older people are comparable to the other 80% vaccinated older people?

***

There is another aspect of the observational studies that are buried away from the summary of the methodology. In the sentences cited above, they said that the study subjects were "followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first." That surely cannot be true!

Another event that happens is vaccination. For people in the unvaccinated group, by definition, they have not been vaccinated up to the day of enrollment; however, many or most of them would get the vaccine during the follow-up period. How did the data analyst deal with this event? We are not given a hint in this methodology section.

Upon vaccination, clearly the follow-up period for the unvaccinated person must end. We simply don't know the counterfactual: what would have happened if the person did not get vaccinated? If we follow the time series of vaccine efficacy from day of vaccination forward, more and more people in the unvaccinated group are dropped from follow-up because they got themselves vaccinated. The people who stay in the unvaccinated group for a long time are the vaccine holdouts, who are clearly different from the general population, e.g. they are much less likely to take other precautions such as wearing masks, distancing, avoiding crowds, etc. How well does matching on demographics and basic health data control for these differences?

If those differences are not adequately accounted for, then the right side of the VE curve cannot be trusted.

***

Also, note that variability is artificially suppressed in this type of study. Anyone who got vaccinated during the study period appear once in the vaccinated group on the day of vaccination, but also may appear in the unvaccinated group on any day prior to vaccination. Unlike a typical RCT in which there are no overlaps in the group membership, in this type of observational studies, the majority of the people show up in both groups. So, if we subsequently run regression analysis comparing the two groups, adjusting for demographics and other factors, the variance between groups is markedly under-stated!

 

 

By how much did observational studies over-estimate Covid vaccine effectiveness

I'm happy to announce that our paper on limitations of observational (i.e. "real-world") studies of Covid-19 vaccines has been published in the Journal of Evaluation in Clinical Practice (link). I'd like to thank my co-authors (Peter and Mark) and reviewers for feedback that greatly improved the presentation, and the journal editors for the courage to print it.

In the paper, we offer three examples in which standard methods used to analyze observational data produce vaccine effectiveness (VE) estimates that are wildly off the mark. Specifically, we show that a hypothetical vaccine with 0% efficacy will be rated as 50 to 70 percent effective. The magnitude of the biases surprised us, and should surprise you too. That said, we are not claiming that Covid-19 vaccines are ineffective - as I explain below, the device of a 0% effective vaccine allows us to work around the lack of data disclosure.

The motivation for writing this paper is to encourage better science. Observational studies are hard, and the quality of Covid-19 related observational studies has been mediocre.

Two major obstacles to writing this paper were (a) the pathetic level of data disclosure for major studies that have had tremendous impact on public health policies and (b) the lack of a ground truth - no one knows the real vaccine efficacy. We found a way to work around these challenges.

For (b), we analyzed the case of the 100% ineffective vaccine. This device is very useful because if the vaccine is ineffective, it is ineffective for all subgroups of people, at all times, etc. If we assumed any other level of vaccine efficacy, say 50%, we'd have had to make a huge list of other assumptions as well. For example, there is no rationale to assume that if VE is 50% in aggregate, then it is 50% for every age group.

For (a), we opportunistically found papers that disclosed just enough data to prove the existence of specific biases. For example, to study the effect of background infection rate bias, we require data on weekly infection rates for certain subpopulations within an analysis window, which we found in a Danish study but are not typically disclosed by researchers. We didn't find a single paper that disclosed all the data required to study all three sources of bias, thus for each bias, we used data from a different reference.

In each example, we show that if the ineffective vaccine were tested in a randomized clinical trial, the typical analysis applied to RCTs would conclude correctly that VE is 0%; however, if the ineffective vaccine were analyzed using methods that were popular in early 2021, the resulting VE would erroneousy land in the 50-70% range! The magnitude of the over-estimation surprised us.

The full paper is here. Spread the word!