Big Data, Plainly Spoken (aka Numbers Rule Your World)

The science behind food recalls

McDonalds is having a moment. It's got our attention when Trump used it to make a political statement, then it got implicated in a food recall. The FDA thinks that at least 75 people have gotten sick and one person died after consuming burgers, and the culprit is thought to be the raw onions inside the burgers. (link)

***
In Chapter 2 of Numbers Rule Your World (link), I praised the science of tracing food outbreaks but readers also learn that these investigations are far from infallible.

The challenge arises because people eat a variety of foods, and when only a few people are getting sick, it's hard to tell where the E.coli has come from. The initial investigation is done using questionnaires to collect as much information as possible about what people consumed. Even this is hard, because of timing issues. Once the data are collected, then some math is used to infer which foods are most likely the culprit.

According to this CDC release, they were able to interview 42 of the sick people, and all of them have eaten at McDonalds prior (how long ago?) before getting ill. Further 86% of them ate a quarter pounder. That's all they have released so far. It sounds very damning and definitive. Yet it is misleading.

We are missing information on any control subjects i.e. similar people who didn't get sick. It's possible that similar people also have high probability of eating at McDonalds, and that a high proportion of McDonalds patrons order quarter pounders. We need a case-control study, and the press release only mentions the cases.

In a perfect world, everyone who got sick ate one common thing and everyone who didn't get sick didn't. But we don't live in a perfect world. So, the result of the investigation is an educated guess.

For example, as far as I could tell, no reporter has asked the hard questions yet: they were first blaming quarter pounders, then they said it's the raw onions inside the quarter pounders... but, does this imply that McDonalds have different supplies of raw onions, and they only use Taylor Farms on quarter pounders, and use a different source of onions on other items?

Also, according to the article, Taylor Farms also supply onions to other fast-food chains, so why is the current outbreak limited to McDonalds? How come the people who ate the same onions supplied to Burger King didn't get ill?

***

Sometimes, these investigations chance upon very good leads, and it is possible to trace the contamination back to a specific batch from a specific plot of land. Sometimes, they might even be able to find the source of E.coli at the farm.

This investigation isn't that kind, though. It seems like it's so far mostly guessing based on scarce data.

***

What else do we know about food recalls from Chapter 2?

Food recalls are very expensive because everyone is throwing stuff away based on "an abundance of caution".

The benefits of food recalls are hard to nail down.

Most contamination events are temporary, and affect only certain batches of produce during a limited time window. Thus, whether or not there is a recall, these perishable goods will leave the food supply eventually, and new cases will vanish. It's very difficult to ascertain if the recall prevented any cases. Indeed, given the time it takes to investigate, and organize recalls, it's quite likely that the crisis is over before the recalls even start.

That's not to say food recalls have zero benefits. It's just not as obvious as one thinks.

 

If it's an RCT, it can be trusted

I've written often about the problems with conducting observational studies. We like to hold up randomized controlled trials (RCTs) as the "gold standard," and if a study is described as an RCT, and especially if its design is pre-registered, we don't ask many questions.

This point of view is becoming unsustainable, based on a recent Nature article.

The gist of the article is that experts now believe at least one out of every four RCTs contain so many problems that their findings should be ignored, leaving the possibility also that some of these RCTs may be entirely fake. In coming to this conclusion, these experts sought and received the underlying individual-level datasets.

These problematic or fake RCTs make their way to systematic reviews (aka meta-analyses), and throught those, they sometimes influence medical practice. Systematic reviews are summary articles that weigh evidence from all studies on a given topic, e.g. whether sugar helps reduce death after head injuries.

In a particularly egregious example, a late Japanese bone-health researcher has had over 100 studies retracted due to fraud, but his papers have appeared in 88 different systematic reviews, half of which would have arrived at a different conclusion if his studies were omitted.

Besides, there is no universal requirement for authors of systematic reviews to revise their analyses should included studies be later retracted. What a mess!

One researcher estimated 20-30 percent of the papers included in systematic reviews in women’s health are suspect. Even when the authors of these systematic reviews have been alerted to the retracted studies, many have ignored pleas to update their findings.

When editors reject studies because of data problems from one journal, the same studies almost always eventually get published in a different journal, “sometimes with different data to those submitted [to the first journal].”

Fixing the RCT problem requires a cultural shift, which does not seem to be forthcoming.

***

The Nature article catalogs a number of objections to research that exposes RCT fraud. Here is a list:

• Privacy concerns of releasing individual-level data
• Participation agreements that do not explicitly include the right to share data
• The logistics of archiving the datasets
• The effort is not worth their time
• The potential harm of false positives
• Ethical concerns related to excluding studies, e.g. wasting the participants' time

Efforts to mandate data sharing have so far been thwarted.

***

Sad to say, the proportion of bad or fake RCTs is substantially higher than 25%. How do I know?

One expert examined over 500 RCTs, and only successfully obtained the individual-level datasets from 150 (30%). In the 30% with detailed data, he found serious problems with 25% of them.

He also looked for problems in the other 70% of the studies, but could only find issues in 1 or 2 percent of them. Paradoxically, this suggests data non-disclosure is positively correlated with trust. But not really. This result points to the fact that typical data summaries in publications hide major problems, including fake data. There isn't much the expert could have done to verify the study findings when the underlying data were not made available. I encountered this issue frequently when looking at Covid-19 vaccine studies.

If we generalize the 25% fraud from studies with data disclosure to those without, then we could conclude that overall 25% of RCTs are suspect. This conclusion assumes that whether the researchers agreed to disclose data is independent of whether their studies are fraudulent. I don't like this assumption. A more sensible assumption is that fraudulent researchers are highly unlikely to disclose their data. Thus, among the studies that don't disclose their data, we should expect a higher proportion of fraud.

Observational studies that don't include everyone intended to receive the treatment

Today, I return to the Clalit booster study. I’ve written three prior posts about the study, which was the catalyst for regulatory approval of booster shots of mRNA vaccines for Covid-19 at the turn of 2022. In the first post, I showed how the methodology - common among Covid-19 vaccine studies - contains obvious biases that researchers and their peer reviewers overlooked. (link) The booster study featured a further unexplained wrinkle – some subjects are excluded after their infection status is known (link). In a third post, I developed a sports analogy to ponder what kind of effects such analytical procedures may have on the results.

This post describes further obvious biases in the study design. There are two classes of biases that affect observational studies in which we cannot randomly apply treatment to subjects. The first class of biases occurs when the two treatment groups are not statistically identical, except for the presence or absence of treatment. The second class of biases occurs when the study population does not look like the population to which the treatment if approved would be applied.

As explained in the first post, this study is unusual in that 90% of the study population appeared first in the no-booster group and later in the booster group. Thus, if we investigate covariate balance in the usual way, the two groups would look close to identical.

So the more interesting question is whether the study population represents people to whom Clalit intends to promote booster shots should they be approved. It’s tough to answer this question without access to the data. We can address a slightly different question: is the study population a random sample of Israelis 50 years old and above?

The answer is most likely no. According to OurWorldinData, the proportion of Israelis 50 year old and above who have received at least one booster dose during the study’s time frame was between 61-80%. And yet the study reported that 90% of the subjects got booster shots during the study period. So, the study population had greater inclination to take the booster shot than the general population.

In fact, fast forward to April 2022, seven months after the study ended, and the last month for which OurWorldinData compiled data, and we learn that the proportion of 50+ year old who have received at least one booster shot was stuck at 73-86 percent, still below the 90% recorded in the study.

***

Looking at the list of exclusions from these types of studies is highly informative. The key question to ask is whether people excluded from these studies will also be excluded from the rollout of the booster. Seemingly the answer is no – the booster will be recommended for essentially everyone.

This practice makes little sense to me. If the excluded people will be given boosters, why should they be excluded from the analysis?

To make this more concrete, here’s the first exclusion criterion: anyone who have ever been infected with Covid-19 prior to the study start date. I have never heard a public health official say don’t take boosters if you had been infected before; quite the opposite – they usually claim that prior infection does not confer immunity, and recommend prior infected to take the boosters.

What might be the effect of removing prior infected? If the public health claim is correct, then prior infection does not offer extra protection to this subgroup, and therefore including them doesn’t affect the effect of the booster shot. As a subgroup, those with prior infection may look different from those without, changing the baseline infection rate but that's not a reason to exclude. If the public health claim is false, that is to say, prior infection does confer some immunity, then any effect of the booster would be attenuated in this subgroup, and excluding them improves the reported result.

Another exclusion criterion involves those whose second vaccine shot (in the original two-shot Pfizer regime) was administered fewer than 5 months from the start of the study period. Once the booster is approved, is there any enforcement of this “rule”? Are people forced to wait at least 5 months before getting the booster?

Exclusions are always present in these studies, and not enough attention has been paid to the biases that exclusions create. Excluding prior infections mean only people without prior infections are in the study. This subset includes people who are health-seeking, and those who follow non-vaccine mitigation measures seriously (masks, distancing, stay home, etc.). Meanwhile, those who ignore mitigation measures and take risks are more likely to get infected, and thus get kicked out of the study.

The other exclusion also directly inserts a bias into the study – as it selects people who are less vaccine hesitant and thus take their vaccine shots earlier rather than later.

A sports analogy for data processing rules

I'm been mulling over how to explain the potential impact of strange data processing procedures used in the observational studies of Covid-19 vaccines. These are by no means standard operating procedures.

And I think I've landed on one explanation that is tantalizing.

***

Imagine a football (soccer) match that is tied 0-0 and heading into overtime. Overtime is a 30-minute affair, played through, and not sudden death. The overtime period is divded into two 15-minute halves with a short break in the middle.

That is the standard operating procedure.

***

Now, we are going to change the rulebook.

The first additional rule is that any goal scored in the first 10 minutes of overtime doesn't count. This rule is inspired the "case counting window" used in the Covid-19 vaccine studies. We're justifying it because players need time to get back to their optimal condition after the exhausting 90-minute regulation time. Therefore, it is not fair to count goals scored at the start of overtime.

The second additional rule is that any goal scored in the first 10 minutes of overtime by the visiting team doesn't count but those scored by the home team counts. How should I justify this rule? Well, to establish the need for recovery time, we take vitals of the players. The measuring machines are only installed in the home locker rooms and not the guest locker rooms, and thus, we are unable to determine if the guest players are at their optimal condition. This rule is inspired by not applying the case counting window to non-vaccinated people since we can't determine when they took the second shot (unlike the situation in clinical trials, when non-vaccinated people are given placebo shots.)

The third additional rule divides the first 10 minutes into two parts, and stipulates that if someone scores in the latter part (i.e. minutes 6-10), then they are immediately kicked out of the match, and their goal doesn't count. This rule is inspired by the special exclusion criterion disclosed in the booster study I've been discussing (link).

This third rule is complicating as it modifies the previous rules. Taken together, we have the following scenarios: for the home team, if they score in the first 5 minutes, the goal counts; if they score in minutes 6-10, the scorer is kicked out of the match, and the goal is disallowed. For the away team, if they score in the first 10 minutes, the goal is always disallowed; in addition, if the goal is scored in minutes 6-10, the scorer is ejected.

***

One can easily get lost in the details of pharmaceutical studies and lose sight of the implications of the methodologies. Transferring the context to sports helps me grasp the effects of these data processing procedures. Hope this helps you too.

PS [8-4-2023] In my haste to put down my thoughts relating to this analogy, I skipped a rule. This is a Rule #2.5. This rule says that if a goal is scored in the first 10 minutes of overtime by the visiting team, it is counted as a goal scored by the home team. In Rule #2, the away-team goal is just disallowed; in Rule #2.5, the goal is added to the home-team tally. This rule is inspired by the booster study, in which they kept people in the no-booster group for 7 days after they took the booster.

 

 

One big problem with booster studies (and observational studies of Covid19 vaccines)

I'll be writing a number of posts about observational studies of the Pfizer booster, published by the Clalit health insurer in Israel (back in December 2021), which is the primary source supporting the effectiveness of the booster shot (3rd shot). These studies provide fodder forever for classes in observational studies because (a) there are many nooks and crannies in which residual biases hid, and (b) researchers made many assertions (wrong, of course) about having corrected all biases using the simplest, most conventional methods.

This first post points out the biggest problem with such studies, which is perhaps impossible to correct.

Let's first review the basic setup of such studies. (The devil is of course in the details, and I'll dive into those in future posts.)

Each study has a well-defined start and end date. This is not a clinical trial, so no one proactively enrolls in this study. Instead, the researchers had access to databases with relevant data and extracted the study population from it.

The start date is chosen close to the launch of the booster shot in Israel, and therefore, at the start, almost everyone was in the "no booster" group. Over the course of the study, most of these same people have migrated to the "booster" group - when they took the booster. Even though the study spanned fewer than 2 months, almost everyone (90%) received the booster shot by the end date.

Here is the first thing one must recognize. This is not a typical clinical trial in which one person belongs to either the treatment group or the control group but not both. In this study, nine out of ten people featured in both the "no booster" and the "booster" groups.

So when subsequently, researchers compare the "no booster" group and the "booster" group, they talk about exposure time, rather than number of exposed. The exposure time of the "no booster" group are incurred mostly by people who eventually got the booster within the study period. This is key observation #1.

If we now hone in on an individual who got the booster shot during the study period, the entire observation time is divided into two parts: from start of study to the day of the booster ("no booster" exposure), and from the day of the booster to the end of the study ("booster" exposure). It is 100% certain that the exposure time in the "no booster" group precedes the exposure time in the "booster" group. This is key observation #2.

Moreover, observation #2 deos not depend on the date of booster, which varies by person. Since individually, each person is exposed first to "no booster" and then to "booster", collectively, the exposure time of the "no booster" group is skewed towards the start of the study period while the exposure time of the "booster" group is skewed towards the end of the study period.

What else do we know about the booster study (or, the majority of Covid vaccine studies)? These studies almost always begin during a peak infection period, which means the infection rate dropped during the study. This is key observation #3.

Together, our three observations mean that the exposure time associated with the non-booster group is skewed towards the front end of the study period, which also happens to be a peak infection period while the exposure time associated with the booster group is skewed towards the back end, when infections are much lower.

For the Israel Booster study, the authors essentially confirmed the existence of such bias with the following sentences:

"During this time [ed: study period],the incidence of Covid-19 in Israel was one of the highest in the world."

"The incidence of Covid-19 and thus exposure to SARS-CoV-2 changed during the study period."

Given that at the start of the study period, Israel's infection rate was "one of the highest in the world" and that the incidence and exposure subsequently "changed," this change represents a decrease in incidence and exposure. (The data can be found here. One of the charts shows the 60 and over age group.)

Reflecting the current state of medical "science," the authors didn't make the above statements in order to explain a source of bias. Instead, they used the first statement to explain why their study could achieve statistical significance on mortality due to Covid, which is a rare outcome, and they cited the second statement in order to make the false claim: "we assume that after adjustment for all covariates, including socioeconomic status, these changes had a similar effect in the booster group and the non-booster group."

Their "assumption" I just showed to not hold.

***

This is another example of the "background infection rate" bias that we described in our bias paper (link). In that paper, we used an example of a Danish study which also started when the infection rate peaked and then by the end of the study period, the infection rate had dropped to almost zero.

If the booster were to be completely useless, the average case rate in the no booster group would be higher than the average case rate in the booster group because of differential exposure time against a decreasing background infection rate. This leads to a high vaccine effectiveness when the effectiveness is assumed to be zero!

Adjusting for "all covariates" does not cure this bias whatsoever.

***

What can be done about such bias?

For one thing, better data disclosure. All these studies should show the distribution of exposure time, by treatment vs control group, over the study period. On top of that, we can superimpose the background infection rate. Such an analysis discloses the source of bias.

Next, attempt to do the same analysis for a different time period, especially one in which the infection rate trends in the opposite direction. In practice, this may be hard to accomplish if the public health department decided to launch vaccines around an infection peak.

It's random because we say so

Some of you are wondering why I haven't written about Covid-19 studies recently. Lamentably, the quality of such studies has not improved. It may have deteriorated even more. I wrestle with whether it's a waste of time to read them. The problems with many studies are apparent from the first page: how do they pass the peer review process?

This post concerns a half page of a recent jourmal publication called "Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data" (link). It's only half of one page because there are already enough problems with what I read that I don't see the point of continuing.

A friend sent me this paper because the authors share my concern about significant misrepresentation of vaccine effectiveness in observational studies of Covid-19 vaccines. (Here is a post about my paper published last month with two co-authors that highlighted three common biases that are not properly handled in studies.)

My comments are limited to the section with the header "Sequential Target Trial Emulation" on page 2 of the paper. In this section, the authors describe the first of two proposed analytical devices for dealing with biases in observational data.

For any observational study, there must be a well-defined starting and ending date of study enrollment. For each day in this enrollment window (and for each region), they pretend that a new target trial is being launched. This is how they put it:

on each day and in each region, eligible persons who have not yet been vaccinated are randomly assigned to receive immediate vaccination or to remain unvaccinated throughout follow-up and are then followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first

They then suggest that each daily trial can be analyzed on its own, and then the results averaged to obtain an overall effectiveness. They claim this outcome is representative of the "intention to treat" effect in a hypothetical randomized clinical trial. That's a bold claim! (Not even Pfizer, Moderna, etc. claimed to have produced ITT vaccine efficacies even in their RCTs.)

I'm not making this up. They literally claimed that "The trial-specific estimates of the effect of vaccination from each sequential trial can then be combined to estimate an observational analogue of the intention-to-treat effect of assignment to the intervention."

***

Why am I shaking my head violently?

Their statement that people are "randomly assigned to receive immediate vaccination or to remain unvaccinated" is unserious.

We are talking about observational data - data analysts poking around in a database after the fact. It doesn't matter what analytical technique is used - it remains the case that the vaccinated people self-selected the treatment. They were not randomly assigned to receive immediate vaccination.

In fact, the authors contradicted themselves in the very next sentence when they explained (my bolding):

Each calendar day is considered as time zero for a new emulated trial, with persons who are determined to be eligible assigned to the vaccination group if they were vaccinated on that day or to the no-vaccination group if they were not vaccinated,

For the vaccinated, there is no "assignment", let alone random assignment. Whoever decided to get vaccinated lands passively in the treatment group for that day. Because the vaccinated are self-selected, the unvaccinated are self-selected as well. Whatever selection biases exist in the raw data persists in this ex-post experimental setup. This step absolutely does not support presumption of random treatment assignment.

If this procedure resulted in random treatment assignment, then the original dataset contains no biases, which annuls the central premise of the entire paper.

***

Of course, they aren't taking every unvaccinated person. So the data analyst actively selects who comprise the unvaccinated group. The authors give two mechanisms for selecting unvaccinated.

The first is random selection. I had to take a deep breath before reading on. I doubled back to see if I read this correctly. "One or more persons can be selected at random from all eligible persons who were unvaccinated on that day and in that region, with exclusion of unvaccinated persons selected for a comparison made on a previous day."

If they randomly selected unvaccinated, then the random sample of unvaccinated would have the same statistical composition as the population of unvaccinated. This ensures no correction for any kind of biases. And yet at the end of this process, they claim to have found the ITT effect as if a randomized clinical trial were run!

The second mechanism for selecting unvaccinated is by matching on key "characteristics that reflect Covid-19 risk". This method is definitely more acceptable as it at least attempts to make the two treatment groups more comparable.

In practice, the matching method faces several big challenges (many of which I have discussed before on this blog):

• the matchable population is a small proportion of the total population, e.g. most studies using matching heavily undersamples older people and oversamples younger people. The method is guaranteed to undersample demographic segments in which vaccination rate is high, and oversample demographic segments in which vaccination rate is low
  
  
• VE in the matched population may be different from VE in the general population. The estimated VE in the matched population is surely not the ITT for the entire population
  
  
• most important confounders are unmeasured (e.g. taking other mitigation measures, vaccine hesitancy, exposure to virus). Matching does not infer balance on non-matching variables, a key reason why ex-post matching on observational data cannot replace randomized treatment assignment
  
  
• most countries enforce highly targeted vaccination programs which generate irreducible biases, e.g. if older people are encouraged to get vaccinated first, and have done so in large numbers (say 80%), what's the chance that the 20% unvaccinated older people are comparable to the other 80% vaccinated older people?

***

There is another aspect of the observational studies that are buried away from the summary of the methodology. In the sentences cited above, they said that the study subjects were "followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first." That surely cannot be true!

Another event that happens is vaccination. For people in the unvaccinated group, by definition, they have not been vaccinated up to the day of enrollment; however, many or most of them would get the vaccine during the follow-up period. How did the data analyst deal with this event? We are not given a hint in this methodology section.

Upon vaccination, clearly the follow-up period for the unvaccinated person must end. We simply don't know the counterfactual: what would have happened if the person did not get vaccinated? If we follow the time series of vaccine efficacy from day of vaccination forward, more and more people in the unvaccinated group are dropped from follow-up because they got themselves vaccinated. The people who stay in the unvaccinated group for a long time are the vaccine holdouts, who are clearly different from the general population, e.g. they are much less likely to take other precautions such as wearing masks, distancing, avoiding crowds, etc. How well does matching on demographics and basic health data control for these differences?

If those differences are not adequately accounted for, then the right side of the VE curve cannot be trusted.

***

Also, note that variability is artificially suppressed in this type of study. Anyone who got vaccinated during the study period appear once in the vaccinated group on the day of vaccination, but also may appear in the unvaccinated group on any day prior to vaccination. Unlike a typical RCT in which there are no overlaps in the group membership, in this type of observational studies, the majority of the people show up in both groups. So, if we subsequently run regression analysis comparing the two groups, adjusting for demographics and other factors, the variance between groups is markedly under-stated!

 

 

By how much did observational studies over-estimate Covid vaccine effectiveness

I'm happy to announce that our paper on limitations of observational (i.e. "real-world") studies of Covid-19 vaccines has been published in the Journal of Evaluation in Clinical Practice (link). I'd like to thank my co-authors (Peter and Mark) and reviewers for feedback that greatly improved the presentation, and the journal editors for the courage to print it.

In the paper, we offer three examples in which standard methods used to analyze observational data produce vaccine effectiveness (VE) estimates that are wildly off the mark. Specifically, we show that a hypothetical vaccine with 0% efficacy will be rated as 50 to 70 percent effective. The magnitude of the biases surprised us, and should surprise you too. That said, we are not claiming that Covid-19 vaccines are ineffective - as I explain below, the device of a 0% effective vaccine allows us to work around the lack of data disclosure.

The motivation for writing this paper is to encourage better science. Observational studies are hard, and the quality of Covid-19 related observational studies has been mediocre.

Two major obstacles to writing this paper were (a) the pathetic level of data disclosure for major studies that have had tremendous impact on public health policies and (b) the lack of a ground truth - no one knows the real vaccine efficacy. We found a way to work around these challenges.

For (b), we analyzed the case of the 100% ineffective vaccine. This device is very useful because if the vaccine is ineffective, it is ineffective for all subgroups of people, at all times, etc. If we assumed any other level of vaccine efficacy, say 50%, we'd have had to make a huge list of other assumptions as well. For example, there is no rationale to assume that if VE is 50% in aggregate, then it is 50% for every age group.

For (a), we opportunistically found papers that disclosed just enough data to prove the existence of specific biases. For example, to study the effect of background infection rate bias, we require data on weekly infection rates for certain subpopulations within an analysis window, which we found in a Danish study but are not typically disclosed by researchers. We didn't find a single paper that disclosed all the data required to study all three sources of bias, thus for each bias, we used data from a different reference.

In each example, we show that if the ineffective vaccine were tested in a randomized clinical trial, the typical analysis applied to RCTs would conclude correctly that VE is 0%; however, if the ineffective vaccine were analyzed using methods that were popular in early 2021, the resulting VE would erroneousy land in the 50-70% range! The magnitude of the over-estimation surprised us.

The full paper is here. Spread the word!

 

 

 

 

Strategies for validating observational studies 2/2

This post continues the previous post about strategies for validating observational studies. Both posts are inspired by my reading of the paper that claims to have discovered that people who do not take the Covid-19 vaccines are more prone to serious traffic accidents.

***

Another tactic used by the researchers to validate their findings is using a different analytical methodology to arrive at the same conclusion.

In this case, they used propensity score matching to create synthetic comparison groups. In matching studies, we accept that the vaccinated and unvaccinated groups are not directly comparable; we match pairs of people, one from each group, based on known matching variables to make sure that the matched pairs themselves are comparable.

The paper does not provide many details on how these researchers did the matching. The matching variables are the same variables used in adjusting the regression models, i.e. the demographics and prior diagnoses. These matching variables are summarized into a single number known as the "propensity score". Matched pairs have the same propensity scores (this description is slightly imprecise because propensity scores are probabilities and cannot be matched exactly. They would have matched based on ranges of values - known as calipers - but none of these details can be found even in the Supplementary Appendix.)

Nonetheless, matching cannot cure imbalance due to unobserved variables, for which there are many. As noted in the previous post, many obviously influential factors - such as exposure to driving, and driving skills - are not measured and therefore not used in the matching analysis.

Matched studies are hard to generalize. The research team reported two attempts at matching (we don't ever know how many they tried, just how many they reported trying). In the first attempt, they kept essentially the entire unvaccinated group (about 20% of the study population). Since they matched one vaccinated person to each unvaccinated, this means three out of four vaccinated persons were excluded from the study. Exclusion does not necessarily mean they are unmatchable; for each unvaccinated person, there could be multiple candidates in the vaccinated pool who match on the propensity score, all but one of whom would be thrown away.

If matching has cured most of the selection bias, the comparison between the matched pairs would be valid. But for whom? The treatment effect is estimated for people who look like those who were matched - in the first attempted matching, this means people who look like the unvaccinated.

Nevertheless, one can easily catch researchers who speak as if they can apply their finding to everyone! If such generalization is admissible, then what's the point of the matching?

The second matching attempt is more "stringent". This means the matched pairs are more similar to each other. What does it also mean? If you're following my drift, more stringent matching means the match rate is even lower, which in turns means the finding applies to an even smaller subset of the study population. As often in statistics, there is no free lunch. 

This stringent matching resulted in about 600K in each group. That means 70% of the unvaccinated, and 94% of the vaccinated persons were excluded from the matching analysis. Not useful if you ask me.

***

The problem with using propensity score matching to validate the adjusted regression model is that each method works with a different analysis population, and so the estimates are not directly comparable.

Strategies for validating observational studies 1/2

As mentioned before, the team who did the research tying Covid-19 vaccinations to car accidents convinced themselves that they have found a real effect. They attempted a variety of confirmatory analyses. In the prior post, I described how they conducted variants of the basic analysis by analyzing subgroups and by modifying the outcome metrics.

It's instructive to look at several other validation strategies and discuss their strengths and weaknesses.

***

Every observational study delivers the following table, usually called Table 1 because it's the first table in the paper:

Table 1 compares the characteristics of the treatment group and the control group. In the traffic accidents study, the two comparison groups are the vaccinated and the unvaccinated residents of Ontario (after exclusions). For a randomized clinical trial, in which vaccination status would be randomly assigned, the two columns of Table 1 should look statistically the same. For an observational study, in which vaccination status is self-selected (or influenced by biases such as public-health priorities), the differences between the two columns provide hints as to the magnitude and direction of biases.

The first thing to notice is vaccinated residents outnumbered unvaccinated about 5 times. The features are divided into three groups: demographics, prior medical conditions (e.g. diagnoses of cancer in the past year), and something labeled "general". The "general" collection includes clinical contacts, hospital admissions, and emergency room visits in the prior year. These are important variables because prior health status is a likely source of self-selection bias.

The researchers have this to say about Table 1:

The 2 groups spanned a diverse range of demographics, with comparable general health care utilization (Table 1). The largest relative differences were that those who had not received a COVID vaccine were more likely to be younger, living in a rural area, and below the middle socioeconomic quintile. Those who had not received a vaccine also were more likely to have a diagnosis of alcohol misuse or depression and less likely to have a diagnosis of sleep apnea, diabetes, cancer, or dementia. About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

This language is highly typical of how researchers opine on the likely sources of bias, i.e. those characteristics that differ between the comparison groups. Now pay attention to the last sentence: About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

The actual data from Table 1 show 4.1% of vaccinated people had a documented Covid-19 infection in the past year, compared to 3.5% of unvaccinated. That's a relative ratio of 1.17. Given the large sample size (millions of people), a 0.6% difference in infection rates is highly significant! If we randomly selected 4/5 of the population to be vaccinated, the difference in infection rates between them and the unvaccinated would typically be much smaller than 0.6%.

So the claim of no major imbalance on Covid-19 prior infection is incorrect. It is also inconsistent because in the main analysis, based on 95% confidence intervals on relative risk, they declared they have found a significant correlation between vaccination status and traffic accidents. The same method would have found the difference between 3.5% and 4.1% prior infection rate to be highly significant. Indeed, if we apply the same method to all rows of Table 1, quite a few would also be found significantly different.

The prior-year emergency visits metric is of particular interest because the primary outcome being analyzed in this study was "emergency visits for individuals injured in traffic crashes". The unvaccinated group had a much higher prior rate of 26% compared to 20% for vaccinated, a relative ratio of almost 1.3, clearly significant. The analysts did not adjust for any of the variables in the general collection in their logistic regression, perhaps because they thought the two groups have "comparable general health care utilization," which as I just explained, is incorrect.

It's ok - in fact, expected - that the unvaccinated group would differ from the vaccinated group on some metrics. It's the opposite that should raise doubt. If the two columns are effectively equivalent, then the two comparison groups resemble those found in randomized trials. If we believe that, we're claiming that obvious sources of bias due to self-selection and public-health policy have not resulted in observed bias.

The biggest weakness of Table 1 analysis is missing variables. The comparisons are made only on the measured covariates. In the case of Covid-19, the measured covariates are whatever can be found in large government databases, collected for purposes other than conducting a study of traffic accidents and Covid-19 vaccination. The researchers said: "The available databases lacked information on driver skill, functional status, personality traits, traffic infractions, political affiliation, and self-identified ethnicity." Several of these variables are clearly pivotal for our present study.

In addition, Table 1 analysis addresses only biases acting independently. Some biases act in complex ways. When I analyzed prior emergency visits, the observed data appeared to indicate a healthy user bias in the vaccinated group. But this factor interacts with the age bias, which was also present. The vaccinated group was quite markedly older than the unvaccinated group. If older people are more likely to need emergency medical service than younger, then the emergency visit bias is even more pronounced since younger, unvaccinated people have a lower baseline emergency room usage than the older, vaccinated group. If younger people are more likely to need emergency medical service older, then age could explain some of the observed bias in emergency room usage. In other words, these factors should be interpreted simultaneously. In fact, any adjustment in the regression model should contain interaction effects, but this is rarely done.

***

The researchers also used "positive controls" and "negative controls" to convince themselves they have properly removed bias from this observational dataset. The idea is to process the data in the same way but substitute other metrics instead of the traffic accidents outcome. These other metrics are chosen because the researchers are 100% sure of the direction of the effect: for positive controls, they claim they knew ex-ante that the alternate outcomes are definitely affected by Covid-19 vaccination status while for negative controls, they claim they knew ex-ante that the alternate outcomes are definitely independent of Covid-19 vaccination status.

For negative control, they presented an analysis of constipation.

There were 2,985 cases of constipation during the month of August in the study population, and the relative risk was 1.03 meaning the unvaccinated group's risk was 3% above that of the vaccinated. This difference is not significant because the confidence interval of 0.94 to 1.14 crosses the 1.00 reference level. In other words, the data provided no evidence that unvaccinated people differ from vaccinated people in terms of experiencing constipation.

This type of validation analysis has many shortcomings. First, it's a single outcome. We don't know if this was pre-specified. Even if pre-specified, it could still show the desired result by chance since there are hypothetically hundreds or thousands of possible negative control candidates. If it's not pre-specified, it's hard to know if other candidates were tried and failed the test.

Perhaps because of reviewer feedback, the research team added several other negative controls, namely, fall, water craft, appendicitis and conjunctivitis. Perhaps because these were not in the original analyses, they did not discuss these other negative controls in the article.

I excerpted the table of results below.

The last column presents the confidence interval of the relative risk, and the second last column from the right shows the point estimate of the relative risk.

The second problem is inconclusiveness. Notice the point estimates of all five negative controls are tightly clustered between 0.90 and 1.14, and every confidence interval except "falls" shows a statistically insignificant difference between the vaccinated and unvaccinated groups. How should one interpret the result of "falls"? The researchers were silent on that one. When a negative control (constipation) shows insignificance, as expected, they were quick to say this finding validated the study conclusion but when a negative control shows a significant difference, it appeared that they explained it away, probably arguing it's a chance observation.

The third problem is over-confidence. How do they know that vaccination could have zero impact on constipation? I imagine this relies on some kind of structural biological argument, such as that mRNA does not act on certain bodily systems. Remember that statistical analyses describe correlations, not causation. We have to rule out ex-ante not just a direct causal relation between vaccination and constipation but any kind of correlation. That's actually a deep assumption, not as simple as it looks.

***

Positive controls are even harder to grapple with because we have to believe ex-ante that vaccination status will certainly result in some given outcome, in this case, "covid pneumonia".

The use of "covid pneumonia" implies the absolute belief that vaccinations reduce covid pneumonia in the analysis population. A few aborted clinical trials and many observational studies do not constitute incontrovertible evidence. Imagine if the data showed no significant difference in covid pneumonia between the unvaccinated and vaccinated. What would be the conclusion? That the methodology is invalidated? That the assumption of causation was incorrect?

These validation analyses using positive or negative controls have limited utility. They produce a situation in which the only possible conclusion is supporting the study - if these analyses do not support the study, we don't know how to interpret them so we throw them out and find another metric to act as control. (As a practical matter, no one would publish the paper if the authors include "failed" controls. See my prior post on the file drawer effect.)

***

The next tactic used for validation is propensity score matching. I'll discuss that in the next post.

Prior posts about this study: 1, 2, 3

 

 

Too good to be true instinct

Good data analysts have developed an instinct of perking up when results are "too good to be true". This is a must-have attribute that I'm always looking for in a job candidate. It is a protective instinct, a response that protects the analysts from getting fooled by the data. Nassim Taleb called this type of thing "fooled by randomness". The temptation is not randomness itself but the lack thereof.

(Credit: Sonja Alves, Flickr)

Observational studies of the Covid-19 vaccines are a case in point. I have been spooked by the consistency of estimates of vaccine effectiveness. Almost all peer-reviewed studies and most working papers that get mainstream attention contain values of VE that are in the 70-90 percent range. (Since these are relative ratios, a 50-percent multiple improvement is already a huge effect.) There is a consistency across headline results of different studies, and there is also a consistency of values inside individual studies, across segments, using different outcome metrics, different methodologies, etc. etc.

Below is Table 2 from the traffic accidents study that I have been reviewing (link):

The aggregate result (derived from the inadmissible non-adjusted comparison discussed in the first post of this series) is shown in the last row and represents a value of 1.7x. The rows above come from applying the same methodology to subsets of the study population, some of them are mutually exclusive while others are overlapping. Most of the values line up close to or near that 1.7x value. There is only one segment (over 65) that has the opposite direction but barely. What does this consistency tell us?

Shown below is the top part of Table 3. Here, the researchers analyzed many variants of the outcome, applying the same analytical strategy. Whereas Figure 2 above shows any involvement in traffic accidents, Table 3 below breaks down these accidents into subtypes, like involvement as driver versus pedestrian, time of day and so on.

The consistency is once again remarkable. Whereas the aggregate number is shown as 1.7x (top row), all the other numbers ranged from 1.5x to 5.1x. Almost all the variance is on the high side of the aggregate. What does such consistency tell us?

In the Appendix, the authors rolled out even more rows of analysis:

Again, all effects are in the range 1.4x to 1.7x. What does this consistency mean? That's what I want to explore in this post.

***

In the mainstream, such consistency is portrayed as resounding evidence. Each individual study that generates another impressive relative ratio similar to the previous result is regarded as another confirmation. The authors of the traffic accidents study believe that because everywhere they looked, they saw the same consistent number, it must mean that they have found something really important - like Newton's Law of Gravity.

Unlike Newton, they aren't studying the realm of physics. The study's data concern human behavior, and social behavior, and it is actually shocking, rather than comforting, to find a law of nature in this arena. My prior - from decades looking at data about humans - is that (a) aggregate effects are typically small, so that even a 10% improvement is remarkable, and (b) there is substantial variability when looking across different segments, different metrics, etc., which is another way to say that "interaction effects" are important.

Were these researchers, mainstream scientists, and the mass media being fooled by (lack of) randomness?

***

Other than discovering some immutable law of nature, what are other possible explanations for the remarkable consistency?

File Drawer Effect

Let's say there is a medical journal editor who only publishes vaccine studies that show high vaccine efficacy - since anything else has little societal value. The result of this editorial practice is that (almost) every study that sees the light of day will definitely have a VE estimate above some lower bound (say, 70%). In this scenario, the consistency of all the peer-reviewed results may have nothing to do with the vaccines themselves, but can be explained by publication bias that is filtering out studies below that bound. This bias is sometimes called the "file drawer effect," describing researchers who stack away "failed" studies (e.g. p-value > 5%) in their file cabinets since journals would reject them.

A consequence of the file drawer effect is that the average of the published results over-estimates the true vaccine efficacy.

In the context of Covid-19 vaccines, the file drawer effect may be related to the phenomonon of cargo cult science.

 

Cargo Cult Science

Cargo cult science is a term coined by physicist Richard Feynman. Roughly speaking, he warns against the potential of bad science even when scientists follow so-called best practices.

Other than the initial clinical trials conducted in 2020 (which were scientifically aborted upon emergency use authorization when the placebo groups were destroyed), all evidence of Covid-19 vaccine efficacy have come from observational datasets. All observational data sets either lack controls, or if controls existed, treatment status was not randomly assigned, which means that unvaccinated people differ from vaccinated people in multiple dimensions besides vaccination status. Therefore, it is wrong to attribute all observed differences between the two groups to vaccination only.

Let's teleport back to the first part of 2021, weeks after the mass vaccination campaign commenced, when the first observational study appeared in print. For the sake of argument, let's say the first analysis showed a VE of 50%. How do we know whether this estimate is to be believed since the people who got vaccinated were clearly different from those who didn't? The honest answer should be no one knows because there is no ground truth in an observational study.

What circumstantial evidence do we have? The only breadcrumbs are interim analysis results from randomized clinical trials that produced VE of 90%+. We face a gnarly problem because our hypothetical observational study led to a VE estimate of 50%. One of the numbers is not like the other; they cannot both be correct. But which study is wrong?

Not surprisingly, most would give more credence to the RCT results, since RCTs are considered the gold standard of medical research. Therefore, since our observational study came in much lower, we'd go back to the drawing board, and massage the dataset some more... until the estimated VE is closer to 90%, making it more credible. The additional data massaging is justified by best practices - we are simply addressing residual confounding bias.

(It is important to realize that randomized experiments can be analyzed in a straightforward, standardized manner but observational studies rely on statistical adjustments that involve subjective judgment from researchers. Analysts try to guess what biases exist in the data, and device ways of correcting for them. It's an iterative process with limited guardrails.)

To make sure you get what's going on, think about what happens if the first analysis of an observational dataset leads to a VE value of 90%. This result would be immediately accepted. Nevertheless, it's not clear that this research team did a better job adjusting for bias than the previous team who came up with 50%.

Since the RCT result was the only piece of available evidence when the first observational study showed up, it became a filter for all subsequent observational studies. Any observational study that produces a number close to 90% feels comfortable, and any study that shows a number far from 90% feels "wrong". Such an editorial process directly triggers the file drawer effect.

***

I don't know how we landed on this stupid state of affairs. Using an observational study to "confirm" an RCT result - the gold standard - is like asking a C student to grade an A student. After the teacher has already given the A student a top grade, what additional information is gleaned by having a C student grade the A student's exam?

What's happening is even more dubious. The C student is not grading the same exam that the teacher saw - the C student is asked to grade a different exam submitted by the A student. By this, I mean: the observational studies are different from the RCTs in meaningful ways, e.g. timing of vaccinations, incidence of Covid-19, types of people eligible for vaccination, residence of patients, no placebo shots... There is scant justification for assuming that the observational studies should produce VE values in the same ballpark as the RCTs!

When another observational study comes out proclaiming 70-90% VE, it doesn't mean much. It just tells me that the research team has selected a set of adjustments so that the VE estimate lands in the zone that is considered acceptable for publication, i.e. it falls in the vicinity of the RCT result.

 

Lurking Variables

Another way in which we can get fooled by the consistency of results is via unknown or unexplored sources of bias. All observational studies can only control for factors for which there are measurements. I haven't encountered yet an observational study of Covid-19 vaccines in which the researchers collected their own data. The current practice is to pull data from pre-existing, government databases. So the variables available are typically limited to basic demographics (age, gender, etc.) or health-related data (prior health concerns, healthcare usage, etc.). Important factors such as attitudes toward freedom and government authority, risk tolerance and exposure to the virus, are not measured, and not adjusted for.

If key sources of bias are not adjusted for, they could explain why the results are so consistent among the subgroups; each such result contains residual confounding from the missing variables.

The missing variable may be a common cause of both vaccination status and infection outcome, for example, people who are generally more health conscious and healthier pre-Covid. They are both more likely to get vaccinated and less likely to get infected (regardless of vaccination status). Since the government databases have no labels for this type of people, there is no way to correct this bias. It is delusional to think that dumping age and gender variables into a regression equation properly adjusted for such a bias. There are health-conscious people in every agexgender segment! In each such demographic segment, vaccination status is still confounded with health-seeking status.

 

Multiple Comparisons

When so many analyses are performed using so many metrics, the best practice is to adjust for multiple comparisons. The general idea is that if one screens a lot of metrics, a few of them will accidentally show statistical significance, and those results are highly likely to be spurious. In other words, without adjustment, traditional statistical methods underestimate the degree of variability in the outcomes. That is to say, results appear more consistent than they are. Yet, most of these vaccine studies sidestep this issue.

I'd be shocked if elementary medical statistics courses don't cover the problem of multiple comparisons. Practitioners in this discipline seem to think that fishing for significance is fine so long as they pre-specify all their intended analyses. Actually, pre-registration does not solve the problem of multiple comparisons at all, unless the pre-registered statistical analysis explicitly outlines a strategy for dealing with it. (Unfortunately, the randomized trials also pre-registered multiple comparisons without correction.)

 

Poorly Defined Metrics

Another reason for consistent results across many analyses is a poorly chosen scale for one's outcome metric. An analogy to grade inflation in schools is telling. American schools has a grade-point average (GPA) typically defined on a scale of 0-4, 4 being grade A. Due to the grade inflation trend, most of the grades given out are As so in reality, the majority of the scale 0-3.5 is basically empty, with almost all students squeezed into the 3.5-4.0 range. Of course, all students look more or less the same.

I think something like this is at play with the relative ratio scales used to measure vaccines. I think an ineffective vaccine will show a VE substantively higher than 0% so the grading scale is compressed just like in schools.

***

In this post, I have offered five possible reasons why every research study of Covid-19 vaccines seems to offer similar effectiveness estimates, and why there appears to be little or no variability when aspects of the methodology are altered. As a statistician who have analyzed a lot of human data, I find it least likely that the reason for the consistency is that these vaccines are like a law of nature, such an overwhelming force that flattens all human variability.