Big Data, Plainly Spoken (aka Numbers Rule Your World)

Uninsightful by design

I saw the following chart in an article about the ordering of author names in research papers. This is a topic of great interest to researchers looking to become famous. Memorably, the article was written by two European researchers with the same first and last names.

 

The article investigates conventions of sequencing authors in scientific papers with more than one authors. The two most popular conventions are: order by significance of contribution ("importance"), or order alphabetically by first letter of last name.

It appears that the following analysis was performed to make the above chart: take all papers with exactly two authors, group by first character of first author's last name, then compute the probability that the last names are listed in alphabetical order, given the first character of the first author's last name.

There is a natural downward slope to this line: if the first author has a last name starting A, the proportion of papers for which the second author has a last name starting B, C, D,... or Z should be close to 100%. If the first author's last name begins with Z, there will be an overwhelming chance that the second author's last name starts with a letter before Z, which means that the pair of names would not be alphabetical.

This chart caught my attention because it's one of those charts that may be uninsightful by design.

***

Why can't it be insightful?

Let's start by finding a statistical model for the observed data (whether the pair of names are alphabetical). The order convention (by importance or by alphabet) is unobservable.

Assume we have a pair of authors. For a specific research paper, the pair is either listed alphabetically or not. If the authors are listed non-alphabetically, then we are sure that the order convention is not alphabetical - which for the purpose of this analysis, we assume to be by importance. By contrast, if the observed order is alphabetical, then either order convention could have generated it -- names being listed alphabetically does not exclude the possibility that they were ordered by importance!

In the article, the authors draw the following conclusion from the chart:

Statistics [green line] exceeds this baseline [bold black line] the most, indicating that many authors intentionally ordered their surnames alphabetically. By contrast, psychology [blue line] appears to be the discipline where the list of authors is mostly ordered by the maount of work contributed (i.e., the percentages are close to the baseline probabilities).

They see the signal in the gap between the value of a particular discipline and the "baseline" probability. If the proportion of papers with alphabetical ordering is higher, then these authors conclude that that discipline is more likely to prefer alphabetical ordering.

Must it be so?

***

Let's take an extreme case as a thought experiment. Assume that no publication uses alphabetical ordering. All authors are listed by importance of their contributions.

If we create the above chart, computing the proportion of alphabetically ordered names given the first letter of the first author's surname, it would look exactly the same - except that the horizontal axis should be relabeled "First letter of the primary author's last name" (instead of "First letter of the first author").

However, the gap between these lines gives us zero information about the preference for alphabetical ordering because all names are listed by importance, and some just happen to also be alphabetical. In this experiment, the gap is purely driven by the (chance?) pairing of authors.

Either the research doesn't accomplish what the researchers think it did, or I'm missing the point. What do you think?

NYT in story-first mode

NYT published a brain-dead article filled with statistical fallacies (link).

Since the Times has people on staff who know statistics, the real explanation for such an article is "story-first thinking". Data mined to support a story line.

***

The headline of the article provokes its least convincing statistic - that only seven black students were offered spots at Stuyvesant, one of the nation's top public high schools famous for STEM, out of 762 offers.

The position of the author is that the above statistic reveals systematic "racial and ethnic disparities". In statistics of education, "disparity" is a descriptive statistic describing a gap, which is different from "discrimination," which suggests foul play. The former is correlation while the latter is causation. This author mischievously used the term disparity while avoiding discrimination, even though the entire article presumes the disparity is unfair.

***

In fact, Stuyvesant (plus seven other similar schools in New York) only considers one factor in admissions, the student's score in the Specialized High Schools Admissions Test (SHSAT), and so the admission decision is not tainted by any judgmental criteria. All test-takers are ranked in order of test score, and then, each student is allocated a spot in their most preferred school - if all slots in the school have been filled, then the next preferred school is considered, and so on.

This allocation method guarantees "fairness" in the sense that if a student isn't allocated to a school higher on his/her preference list, it's because every offer for that school went to students who have scored higher on the SHSAT.

***

If we are interested in the question of whether the allocation of offers is fair across ethnic groups, the right question is: among students of similar ability (for example, score bands), do black or Latino students have a lower offer rate than whites and Asians?

This question of fairness pops in everywhere in education. In Chapter 3 of Numbers Rule Your World (link), I examined it in the context of designing fair SAT questions.

Surprise, surprise. Such data have not been released. Since there are no data, I'll explain conceptually how the numbers work.

***

Let's assume 30% of test-takers are Asians, and 20% each are blacks, latinos, and whites, and 10% others. Assume 50% of offers are given to Asians, 30% to whites, 5% each to blacks and latinos, and 10% to others. Overall, 15% of the test-takers receive an offer.

Given those assumptions, the offer rate for Asians is 25%, for whites it's 23%, for blacks and Latinos, it's 4%. Asians and whites are 1.5-1.7x more likely than the average to get an offer, while the offer rates of blacks and Latinos are a quarter of the average.

The NYT claims this proves that the process unfairly favors Asian and white students at the detriment of blacks and Latinos.

Not so fast. Remember all offers are completely determined by test scores, without any judgmental factors. So the proportion of students who receive offers is the same as the proportion who score above the cutoff score. Think of the cutoff score as the score of the last student taken up by a school. This cutoff score is an outcome of the offer process, not an input.

***

We can now split each subgroup into those who score above the cutoff and those below. Among Asians, the split is 25%-75%; whites, 23%-77%; blacks, 4%-96%; Latinos, 4%-96%; others, 15%-85%.

The reason why Asians and whites have more offers is because they have higher SHSAT scores.

Let's now insist that the offer rate for every ethnic group must equal the overall rate of 15%. For Asians and whites with a higher score distribution, this necessarily means that we cannot accept everyone above the cutoff score. It turns out that we can only accept about 60-67% of those who score above the cutoff score, and 0% of those who score below, in order to get to a weighted average of 15%.

For blacks and Latinos, we'd take 100% of those who score above the cutoff, but that isn't enough to get to 15%. We also must take 12% of those who score below the cutoff.

This is the unspoken math behind the NYT article. They are proposing: rejecting 40% of high-scoring Asians and whites while accepting 100% of high-scoring blacks and Latinos; and accepting 12% of low-scoring blacks and Latinos while rejecting 100% of low-scoring Asians and whites.

Shame.

 

 

It's random because we say so

Some of you are wondering why I haven't written about Covid-19 studies recently. Lamentably, the quality of such studies has not improved. It may have deteriorated even more. I wrestle with whether it's a waste of time to read them. The problems with many studies are apparent from the first page: how do they pass the peer review process?

This post concerns a half page of a recent jourmal publication called "Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data" (link). It's only half of one page because there are already enough problems with what I read that I don't see the point of continuing.

A friend sent me this paper because the authors share my concern about significant misrepresentation of vaccine effectiveness in observational studies of Covid-19 vaccines. (Here is a post about my paper published last month with two co-authors that highlighted three common biases that are not properly handled in studies.)

My comments are limited to the section with the header "Sequential Target Trial Emulation" on page 2 of the paper. In this section, the authors describe the first of two proposed analytical devices for dealing with biases in observational data.

For any observational study, there must be a well-defined starting and ending date of study enrollment. For each day in this enrollment window (and for each region), they pretend that a new target trial is being launched. This is how they put it:

on each day and in each region, eligible persons who have not yet been vaccinated are randomly assigned to receive immediate vaccination or to remain unvaccinated throughout follow-up and are then followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first

They then suggest that each daily trial can be analyzed on its own, and then the results averaged to obtain an overall effectiveness. They claim this outcome is representative of the "intention to treat" effect in a hypothetical randomized clinical trial. That's a bold claim! (Not even Pfizer, Moderna, etc. claimed to have produced ITT vaccine efficacies even in their RCTs.)

I'm not making this up. They literally claimed that "The trial-specific estimates of the effect of vaccination from each sequential trial can then be combined to estimate an observational analogue of the intention-to-treat effect of assignment to the intervention."

***

Why am I shaking my head violently?

Their statement that people are "randomly assigned to receive immediate vaccination or to remain unvaccinated" is unserious.

We are talking about observational data - data analysts poking around in a database after the fact. It doesn't matter what analytical technique is used - it remains the case that the vaccinated people self-selected the treatment. They were not randomly assigned to receive immediate vaccination.

In fact, the authors contradicted themselves in the very next sentence when they explained (my bolding):

Each calendar day is considered as time zero for a new emulated trial, with persons who are determined to be eligible assigned to the vaccination group if they were vaccinated on that day or to the no-vaccination group if they were not vaccinated,

For the vaccinated, there is no "assignment", let alone random assignment. Whoever decided to get vaccinated lands passively in the treatment group for that day. Because the vaccinated are self-selected, the unvaccinated are self-selected as well. Whatever selection biases exist in the raw data persists in this ex-post experimental setup. This step absolutely does not support presumption of random treatment assignment.

If this procedure resulted in random treatment assignment, then the original dataset contains no biases, which annuls the central premise of the entire paper.

***

Of course, they aren't taking every unvaccinated person. So the data analyst actively selects who comprise the unvaccinated group. The authors give two mechanisms for selecting unvaccinated.

The first is random selection. I had to take a deep breath before reading on. I doubled back to see if I read this correctly. "One or more persons can be selected at random from all eligible persons who were unvaccinated on that day and in that region, with exclusion of unvaccinated persons selected for a comparison made on a previous day."

If they randomly selected unvaccinated, then the random sample of unvaccinated would have the same statistical composition as the population of unvaccinated. This ensures no correction for any kind of biases. And yet at the end of this process, they claim to have found the ITT effect as if a randomized clinical trial were run!

The second mechanism for selecting unvaccinated is by matching on key "characteristics that reflect Covid-19 risk". This method is definitely more acceptable as it at least attempts to make the two treatment groups more comparable.

In practice, the matching method faces several big challenges (many of which I have discussed before on this blog):

• the matchable population is a small proportion of the total population, e.g. most studies using matching heavily undersamples older people and oversamples younger people. The method is guaranteed to undersample demographic segments in which vaccination rate is high, and oversample demographic segments in which vaccination rate is low
  
  
• VE in the matched population may be different from VE in the general population. The estimated VE in the matched population is surely not the ITT for the entire population
  
  
• most important confounders are unmeasured (e.g. taking other mitigation measures, vaccine hesitancy, exposure to virus). Matching does not infer balance on non-matching variables, a key reason why ex-post matching on observational data cannot replace randomized treatment assignment
  
  
• most countries enforce highly targeted vaccination programs which generate irreducible biases, e.g. if older people are encouraged to get vaccinated first, and have done so in large numbers (say 80%), what's the chance that the 20% unvaccinated older people are comparable to the other 80% vaccinated older people?

***

There is another aspect of the observational studies that are buried away from the summary of the methodology. In the sentences cited above, they said that the study subjects were "followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first." That surely cannot be true!

Another event that happens is vaccination. For people in the unvaccinated group, by definition, they have not been vaccinated up to the day of enrollment; however, many or most of them would get the vaccine during the follow-up period. How did the data analyst deal with this event? We are not given a hint in this methodology section.

Upon vaccination, clearly the follow-up period for the unvaccinated person must end. We simply don't know the counterfactual: what would have happened if the person did not get vaccinated? If we follow the time series of vaccine efficacy from day of vaccination forward, more and more people in the unvaccinated group are dropped from follow-up because they got themselves vaccinated. The people who stay in the unvaccinated group for a long time are the vaccine holdouts, who are clearly different from the general population, e.g. they are much less likely to take other precautions such as wearing masks, distancing, avoiding crowds, etc. How well does matching on demographics and basic health data control for these differences?

If those differences are not adequately accounted for, then the right side of the VE curve cannot be trusted.

***

Also, note that variability is artificially suppressed in this type of study. Anyone who got vaccinated during the study period appear once in the vaccinated group on the day of vaccination, but also may appear in the unvaccinated group on any day prior to vaccination. Unlike a typical RCT in which there are no overlaps in the group membership, in this type of observational studies, the majority of the people show up in both groups. So, if we subsequently run regression analysis comparing the two groups, adjusting for demographics and other factors, the variance between groups is markedly under-stated!

 

 

By how much did observational studies over-estimate Covid vaccine effectiveness

I'm happy to announce that our paper on limitations of observational (i.e. "real-world") studies of Covid-19 vaccines has been published in the Journal of Evaluation in Clinical Practice (link). I'd like to thank my co-authors (Peter and Mark) and reviewers for feedback that greatly improved the presentation, and the journal editors for the courage to print it.

In the paper, we offer three examples in which standard methods used to analyze observational data produce vaccine effectiveness (VE) estimates that are wildly off the mark. Specifically, we show that a hypothetical vaccine with 0% efficacy will be rated as 50 to 70 percent effective. The magnitude of the biases surprised us, and should surprise you too. That said, we are not claiming that Covid-19 vaccines are ineffective - as I explain below, the device of a 0% effective vaccine allows us to work around the lack of data disclosure.

The motivation for writing this paper is to encourage better science. Observational studies are hard, and the quality of Covid-19 related observational studies has been mediocre.

Two major obstacles to writing this paper were (a) the pathetic level of data disclosure for major studies that have had tremendous impact on public health policies and (b) the lack of a ground truth - no one knows the real vaccine efficacy. We found a way to work around these challenges.

For (b), we analyzed the case of the 100% ineffective vaccine. This device is very useful because if the vaccine is ineffective, it is ineffective for all subgroups of people, at all times, etc. If we assumed any other level of vaccine efficacy, say 50%, we'd have had to make a huge list of other assumptions as well. For example, there is no rationale to assume that if VE is 50% in aggregate, then it is 50% for every age group.

For (a), we opportunistically found papers that disclosed just enough data to prove the existence of specific biases. For example, to study the effect of background infection rate bias, we require data on weekly infection rates for certain subpopulations within an analysis window, which we found in a Danish study but are not typically disclosed by researchers. We didn't find a single paper that disclosed all the data required to study all three sources of bias, thus for each bias, we used data from a different reference.

In each example, we show that if the ineffective vaccine were tested in a randomized clinical trial, the typical analysis applied to RCTs would conclude correctly that VE is 0%; however, if the ineffective vaccine were analyzed using methods that were popular in early 2021, the resulting VE would erroneousy land in the 50-70% range! The magnitude of the over-estimation surprised us.

The full paper is here. Spread the word!

 

 

 

 

Practical issues in observational studies

When I read Covid-19 medical research during the pandemic, one recurring practice bugs me a lot. Researchers seem to believe that a simple adjustment term inserted into a regression model eliminates bias in analyzing observational studies. To wit, if I suspect that age is a confounder, all I have to do is to insert an age term into my regression equation, and somehow that magically resolves the age bias fully.

That's just odd. Such a simple adjustment scheme works for simple biases. Specificially, one must get the age strata defined correctly so that within each age stratum, the treatment status is as if random (conditional on any other confounders used). In most real-world scenarios, this conditional independence assumption is hard to believe.

The age bias example is the tip of the iceberg. Even more bewildering is calendar time adjustment. This is used as a cure-all for all time-varying confounders. If one just inserts a calendar-time term into the regression model, all time-varying biases disappear! Shhh, don't even ask how they determine the appropriate time strata (days, weeks, months, etc.). [Whispering: you don't say the passage of time does not cause confounding.]

Then, I came across Andrew's recent paper - coauthored with Lauren Kennedy - called "causal quartets" (link to PDF), which nicely lays out some of the problems I've tossed around in my head.

Let me summarize the key points of this paper.

***

The methodology being discussed is "a causal model using linear regressions without interactions". This is the dominant type of causal models used in "real-world" studies of Covid-19 vaccines (should nicely generalize to logistic regressions, Cox regressions, etc.). The researchers identify a set of potential confounders - almost always, these are convenient variables that are pulled out of large-scale government databases, such as age, gender, prior hospital visits, etc. - then throw these confounders as main effects in a regression model (typically, a Cox regression). Then, the practice is to take "the coefficient of the treatment variable" as the "causal effect" of the treatment. As Andrew and Lauren stated, this causal effect is really an "average" treatment effect.

It's an average effect in the sense that the treatment under study has an effect for each individual, which may vary across individuals, and the single number that summarizes this distribution is the average of individual effects.

The existence of an average effect does not imply that the effect is identical to all individuals. Indeed, the causal quartet presents different distributions of individual effects that lead to the same average effect.

[Readers will be reminded of the so-called Anscombe's quartets, a favorite of statistics professors to warn against running "trendlines" on any and all datasets.]

Frequently, individuals can be organized into groups (e.g. age groups). Each age group has an average treatment effect among its individuals, and the average treatment effect is an average of the group averages. The value of the average clearly then depends on age group distribution within the population of individuals.

For more, see the Prologue of Numbersense (link), where I discuss how this plays into the Simpson's paradox.

***

To nail down the ideas, let's look at a simplest case in which age is the only confounder. The age-adjusted linear regression model is:

Outcome (given Treatment and AgeGroup) = a + b1*Treatment + b2(j)*AgeGroup(j)

where AgeGroup is one of j=3 ("young", "mid", "old"), and "Not Treated" and "old" are considered the reference levels, thus b2("old") = 0 and b1 is set to 0 for Not Treated.

Under this model, we have the following calculations:

Outcome (given Treated, "old") = a + b1

Outcome (given Not Treated, "old") = a

Treatment Effect (given "old") = a + b1 - a = b1

Similarly,

Outcome (given Treated, "young") = a + b1 + b2("young")

Outcome (given Not Treated, "young") = a + b2("young")

Treatment Effect (given "young") = a + b1 + b2("young") - a - b2("young") = b1

In sum, for all age groups, the model gives the same treatment effect of b1. Thus, b1 is the average treatment effect for the entire population. In this case, there is no difference between weighted and unweighted average; the age distribution does not matter.

The madness begins when people take the above and conclude that "the treatment effect is the same for all age groups".

This gets things backwards. This conclusion is the starting point, not the ending point, of the regression model; it is a necessary consequence of its structure.

Let's look at the regression equation again:

Outcome (given Treatment and AgeGroup) = a + b1*Treatment + b2(j)*AgeGroup(j)

Three terms together constitute the outcome. The first term ("a") applies to every individual regardless of treatment status or age. Its value is set to be the outcome of the Not Treated in the "old" group (not the only possibility but what I'm using in this discussion). Think of selecting the Not Treated, "old" group as the reference group, and measuring other groups against it.

Then, for each Treated person, regardless of age, we add the second term "b1".

Finally, for each person in AgeGroup j, we add the third term "b2(j)", except the "old" group (for which we add zero). 

Now, let's consider an alternative hypothesis: that the treatment effect decreases by age group; the older the individual, the lower the effect. Under the above regression model, however, the treatment effect is always b1 regardless of age. Therefore, it is impossible to represent the alt hypothesis with this model!

We need b1 to vary by age group. What we need is an "interaction" or "cross" term in the regression model, something like this:

Outcome (given Treatment and AgeGroup) = a + b1*Treatment + b2(j)AgeGroup(j) + b3(j)*Treatment*AgeGroup(j)

where, for simplicity of presentation, I set b3(j) = 0 for Not Treated regardless of Age Group so that I don't have to make b3(t, j)

Under this model, we can compute:

Outcome (given Treated, "old") = a + b1 + b3(Treated, "old")

Outcome (given Not Treated, "old") = a

Treatment Effect (given "old") = b1 + b3(Treated, "old")

Similarly,

Outcome (given Treated, "young") = a + b1 + b2("young") + b3(Treated, "young")

Outcome (given Not Treated, "young") = a + b2("young")

Treatment Effect (given "young") = b1 + b3(Treated, "young")

Now, the treatment effect consists of two parts, b1 which is independent of age, and b3 which varies with age group, and so in total, the treatment effect varies with age group.

***

By using the first regression model, the modeler makes the assumption that the treatment effect is the same for all age groups. If the modeler concludes, after fitting that model, that the treatment effect is the same for all age groups, s/he has set up a tautology, and has proven nothing.

Unfortunately, that's the kind of thing that is common in Covid-19 observational studies. They don't even publish the regression equations, coefficients, measures of fit, etc. They should have also tried the model with interactions, and explain why that is not preferred.

In addition, it appears that most of these same studies take b1 from the first model, and call it the treatment effect. SInce this first model assumes that the effect is constant for all subgroups, we can conveniently ignore the relative sizes of these subgroups.

If we have to compute the average treatment effect under the second model (the one with interactions), it would be:

Average treatment effect = Proportion "young" * (b1 + b3(Treated, "young")) + Proportion "mid" * (b1 + b3(Treated, "mid")) + Proportion "old" * (b1 + b3(Treated, "old")) = b1 + weighted average of b3(Treated, j)

The only scenario when we can ignore the age distribution is when the treatment effect does not depend on age. In this case, the weighted average equals the unweighted average so that the second model "converges" to the first model. As already noted, that is an assumption of the researchers, not a conclusion of the fitted model.

If the true treatment effect varies by age, then the form of the first regression model is inappropriate, and the average treatment effect estimated by this model is not the required weighted average effect. That's one of the key points of the Andrew and Lauren paper.

(Scroll to the end of the post to see a visualization of the two models.)

***

Andrew and Lauren suggest that the practice of using regression models without interactions is due to lack of data. It takes "16 times the sample size" to estimate one interaction effect relative to the main effect.

I get it when the observational data came from the professor's undergraduate class of 100 students but in Covid-19 studies, they have easily hundreds of thousands, if not millions, of subjects.

Nevertheless, the point of the causal quartets paper is to advocate including varying treatment effects in modeling observational data. In Section 3.1, they put some numbers behind the concepts I covered above. The study had a treatment effect of 25% higher survival rate (55% survived in the treatment group vs 30% in the control group).

If the regression model does not contain interactions, and the treatment effect (b1) from that model is interpreted as the average treatment effect, one expects each individual who takes the drug should have a 25% higher survival rate. That is only one possibility. Another possibility is that only 25% of the patients benefit from the drug, and for these, the chance of longer survival is 100%. The average treatment effect is 25%*100% + 0%*0% = 25%. Gelman and Lauren then concludes that "once we accept the idea that the drug works on some people and not others,..., we realize that 'the treatment effect' of any given study will depend entirely on the patient mix." Yep, bingo, 100%, +1.

The second possibility is more plausible than the first, and it is excluded by the structure of the regression model without interactions.

In later sections, Gelman and Lauren explore the implication of the fact that the intervention in most observational studies has no impact on a good portion of the subjects. In particular, they note that even if there is a huge impact on a small proportion of the subjects, the average treatment effect may still be small because of averaging. This calls into questions studies that (a) show huge average treatment effects and (b) for which most treated subjects would have no benefit.

Next, they point out that if we knew who might benefit from treatment, we'd have a better measurement if we restricted the test population to that subgroup. This effectively removes the majority of the subjects expected not to benefit, and is widely practiced in all Covid-19 studies (long lists of exclusions). All well and good... until the researchers then pretend that they didn't exclude anyone, and apply the treatment effect from the restricted study to the entire population. Instead, "when generalizing to a larger population, some modeling is necessary conditional on any information used in patient selection." (Gelman & Lauren) Unfortunately, I have not seen this done even once in any of the Covid-19 studies I read.

***
Here is an illustration of the differences between the two regression models discussed above, for those visually inclined:

The second model allows the three green dots for Treated to be placed anywhere: the profile of the Treated line (green) along the age dimension can be anything, does not have to be linear or parallel to the Not Treated line (blue). In this illustration, I arbitrarily decided that the treatment effect is below average for "mid" and "old" age groups; that it is the same for "mid" as for "old"; and that the treatment effect for "young" is higher. By assumption for simplicity, the blue lines (Not Treated) are equivalent on both panels but in general, two of the three blue dots on the right panel can also vary.

P.S. [3-1-23] Fixed the title of the right panel chart.

 

Strategies for validating observational studies 2/2

This post continues the previous post about strategies for validating observational studies. Both posts are inspired by my reading of the paper that claims to have discovered that people who do not take the Covid-19 vaccines are more prone to serious traffic accidents.

***

Another tactic used by the researchers to validate their findings is using a different analytical methodology to arrive at the same conclusion.

In this case, they used propensity score matching to create synthetic comparison groups. In matching studies, we accept that the vaccinated and unvaccinated groups are not directly comparable; we match pairs of people, one from each group, based on known matching variables to make sure that the matched pairs themselves are comparable.

The paper does not provide many details on how these researchers did the matching. The matching variables are the same variables used in adjusting the regression models, i.e. the demographics and prior diagnoses. These matching variables are summarized into a single number known as the "propensity score". Matched pairs have the same propensity scores (this description is slightly imprecise because propensity scores are probabilities and cannot be matched exactly. They would have matched based on ranges of values - known as calipers - but none of these details can be found even in the Supplementary Appendix.)

Nonetheless, matching cannot cure imbalance due to unobserved variables, for which there are many. As noted in the previous post, many obviously influential factors - such as exposure to driving, and driving skills - are not measured and therefore not used in the matching analysis.

Matched studies are hard to generalize. The research team reported two attempts at matching (we don't ever know how many they tried, just how many they reported trying). In the first attempt, they kept essentially the entire unvaccinated group (about 20% of the study population). Since they matched one vaccinated person to each unvaccinated, this means three out of four vaccinated persons were excluded from the study. Exclusion does not necessarily mean they are unmatchable; for each unvaccinated person, there could be multiple candidates in the vaccinated pool who match on the propensity score, all but one of whom would be thrown away.

If matching has cured most of the selection bias, the comparison between the matched pairs would be valid. But for whom? The treatment effect is estimated for people who look like those who were matched - in the first attempted matching, this means people who look like the unvaccinated.

Nevertheless, one can easily catch researchers who speak as if they can apply their finding to everyone! If such generalization is admissible, then what's the point of the matching?

The second matching attempt is more "stringent". This means the matched pairs are more similar to each other. What does it also mean? If you're following my drift, more stringent matching means the match rate is even lower, which in turns means the finding applies to an even smaller subset of the study population. As often in statistics, there is no free lunch. 

This stringent matching resulted in about 600K in each group. That means 70% of the unvaccinated, and 94% of the vaccinated persons were excluded from the matching analysis. Not useful if you ask me.

***

The problem with using propensity score matching to validate the adjusted regression model is that each method works with a different analysis population, and so the estimates are not directly comparable.

Strategies for validating observational studies 1/2

As mentioned before, the team who did the research tying Covid-19 vaccinations to car accidents convinced themselves that they have found a real effect. They attempted a variety of confirmatory analyses. In the prior post, I described how they conducted variants of the basic analysis by analyzing subgroups and by modifying the outcome metrics.

It's instructive to look at several other validation strategies and discuss their strengths and weaknesses.

***

Every observational study delivers the following table, usually called Table 1 because it's the first table in the paper:

Table 1 compares the characteristics of the treatment group and the control group. In the traffic accidents study, the two comparison groups are the vaccinated and the unvaccinated residents of Ontario (after exclusions). For a randomized clinical trial, in which vaccination status would be randomly assigned, the two columns of Table 1 should look statistically the same. For an observational study, in which vaccination status is self-selected (or influenced by biases such as public-health priorities), the differences between the two columns provide hints as to the magnitude and direction of biases.

The first thing to notice is vaccinated residents outnumbered unvaccinated about 5 times. The features are divided into three groups: demographics, prior medical conditions (e.g. diagnoses of cancer in the past year), and something labeled "general". The "general" collection includes clinical contacts, hospital admissions, and emergency room visits in the prior year. These are important variables because prior health status is a likely source of self-selection bias.

The researchers have this to say about Table 1:

The 2 groups spanned a diverse range of demographics, with comparable general health care utilization (Table 1). The largest relative differences were that those who had not received a COVID vaccine were more likely to be younger, living in a rural area, and below the middle socioeconomic quintile. Those who had not received a vaccine also were more likely to have a diagnosis of alcohol misuse or depression and less likely to have a diagnosis of sleep apnea, diabetes, cancer, or dementia. About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

This language is highly typical of how researchers opine on the likely sources of bias, i.e. those characteristics that differ between the comparison groups. Now pay attention to the last sentence: About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

The actual data from Table 1 show 4.1% of vaccinated people had a documented Covid-19 infection in the past year, compared to 3.5% of unvaccinated. That's a relative ratio of 1.17. Given the large sample size (millions of people), a 0.6% difference in infection rates is highly significant! If we randomly selected 4/5 of the population to be vaccinated, the difference in infection rates between them and the unvaccinated would typically be much smaller than 0.6%.

So the claim of no major imbalance on Covid-19 prior infection is incorrect. It is also inconsistent because in the main analysis, based on 95% confidence intervals on relative risk, they declared they have found a significant correlation between vaccination status and traffic accidents. The same method would have found the difference between 3.5% and 4.1% prior infection rate to be highly significant. Indeed, if we apply the same method to all rows of Table 1, quite a few would also be found significantly different.

The prior-year emergency visits metric is of particular interest because the primary outcome being analyzed in this study was "emergency visits for individuals injured in traffic crashes". The unvaccinated group had a much higher prior rate of 26% compared to 20% for vaccinated, a relative ratio of almost 1.3, clearly significant. The analysts did not adjust for any of the variables in the general collection in their logistic regression, perhaps because they thought the two groups have "comparable general health care utilization," which as I just explained, is incorrect.

It's ok - in fact, expected - that the unvaccinated group would differ from the vaccinated group on some metrics. It's the opposite that should raise doubt. If the two columns are effectively equivalent, then the two comparison groups resemble those found in randomized trials. If we believe that, we're claiming that obvious sources of bias due to self-selection and public-health policy have not resulted in observed bias.

The biggest weakness of Table 1 analysis is missing variables. The comparisons are made only on the measured covariates. In the case of Covid-19, the measured covariates are whatever can be found in large government databases, collected for purposes other than conducting a study of traffic accidents and Covid-19 vaccination. The researchers said: "The available databases lacked information on driver skill, functional status, personality traits, traffic infractions, political affiliation, and self-identified ethnicity." Several of these variables are clearly pivotal for our present study.

In addition, Table 1 analysis addresses only biases acting independently. Some biases act in complex ways. When I analyzed prior emergency visits, the observed data appeared to indicate a healthy user bias in the vaccinated group. But this factor interacts with the age bias, which was also present. The vaccinated group was quite markedly older than the unvaccinated group. If older people are more likely to need emergency medical service than younger, then the emergency visit bias is even more pronounced since younger, unvaccinated people have a lower baseline emergency room usage than the older, vaccinated group. If younger people are more likely to need emergency medical service older, then age could explain some of the observed bias in emergency room usage. In other words, these factors should be interpreted simultaneously. In fact, any adjustment in the regression model should contain interaction effects, but this is rarely done.

***

The researchers also used "positive controls" and "negative controls" to convince themselves they have properly removed bias from this observational dataset. The idea is to process the data in the same way but substitute other metrics instead of the traffic accidents outcome. These other metrics are chosen because the researchers are 100% sure of the direction of the effect: for positive controls, they claim they knew ex-ante that the alternate outcomes are definitely affected by Covid-19 vaccination status while for negative controls, they claim they knew ex-ante that the alternate outcomes are definitely independent of Covid-19 vaccination status.

For negative control, they presented an analysis of constipation.

There were 2,985 cases of constipation during the month of August in the study population, and the relative risk was 1.03 meaning the unvaccinated group's risk was 3% above that of the vaccinated. This difference is not significant because the confidence interval of 0.94 to 1.14 crosses the 1.00 reference level. In other words, the data provided no evidence that unvaccinated people differ from vaccinated people in terms of experiencing constipation.

This type of validation analysis has many shortcomings. First, it's a single outcome. We don't know if this was pre-specified. Even if pre-specified, it could still show the desired result by chance since there are hypothetically hundreds or thousands of possible negative control candidates. If it's not pre-specified, it's hard to know if other candidates were tried and failed the test.

Perhaps because of reviewer feedback, the research team added several other negative controls, namely, fall, water craft, appendicitis and conjunctivitis. Perhaps because these were not in the original analyses, they did not discuss these other negative controls in the article.

I excerpted the table of results below.

The last column presents the confidence interval of the relative risk, and the second last column from the right shows the point estimate of the relative risk.

The second problem is inconclusiveness. Notice the point estimates of all five negative controls are tightly clustered between 0.90 and 1.14, and every confidence interval except "falls" shows a statistically insignificant difference between the vaccinated and unvaccinated groups. How should one interpret the result of "falls"? The researchers were silent on that one. When a negative control (constipation) shows insignificance, as expected, they were quick to say this finding validated the study conclusion but when a negative control shows a significant difference, it appeared that they explained it away, probably arguing it's a chance observation.

The third problem is over-confidence. How do they know that vaccination could have zero impact on constipation? I imagine this relies on some kind of structural biological argument, such as that mRNA does not act on certain bodily systems. Remember that statistical analyses describe correlations, not causation. We have to rule out ex-ante not just a direct causal relation between vaccination and constipation but any kind of correlation. That's actually a deep assumption, not as simple as it looks.

***

Positive controls are even harder to grapple with because we have to believe ex-ante that vaccination status will certainly result in some given outcome, in this case, "covid pneumonia".

The use of "covid pneumonia" implies the absolute belief that vaccinations reduce covid pneumonia in the analysis population. A few aborted clinical trials and many observational studies do not constitute incontrovertible evidence. Imagine if the data showed no significant difference in covid pneumonia between the unvaccinated and vaccinated. What would be the conclusion? That the methodology is invalidated? That the assumption of causation was incorrect?

These validation analyses using positive or negative controls have limited utility. They produce a situation in which the only possible conclusion is supporting the study - if these analyses do not support the study, we don't know how to interpret them so we throw them out and find another metric to act as control. (As a practical matter, no one would publish the paper if the authors include "failed" controls. See my prior post on the file drawer effect.)

***

The next tactic used for validation is propensity score matching. I'll discuss that in the next post.

Prior posts about this study: 1, 2, 3

 

 

Too good to be true instinct

Good data analysts have developed an instinct of perking up when results are "too good to be true". This is a must-have attribute that I'm always looking for in a job candidate. It is a protective instinct, a response that protects the analysts from getting fooled by the data. Nassim Taleb called this type of thing "fooled by randomness". The temptation is not randomness itself but the lack thereof.

(Credit: Sonja Alves, Flickr)

Observational studies of the Covid-19 vaccines are a case in point. I have been spooked by the consistency of estimates of vaccine effectiveness. Almost all peer-reviewed studies and most working papers that get mainstream attention contain values of VE that are in the 70-90 percent range. (Since these are relative ratios, a 50-percent multiple improvement is already a huge effect.) There is a consistency across headline results of different studies, and there is also a consistency of values inside individual studies, across segments, using different outcome metrics, different methodologies, etc. etc.

Below is Table 2 from the traffic accidents study that I have been reviewing (link):

The aggregate result (derived from the inadmissible non-adjusted comparison discussed in the first post of this series) is shown in the last row and represents a value of 1.7x. The rows above come from applying the same methodology to subsets of the study population, some of them are mutually exclusive while others are overlapping. Most of the values line up close to or near that 1.7x value. There is only one segment (over 65) that has the opposite direction but barely. What does this consistency tell us?

Shown below is the top part of Table 3. Here, the researchers analyzed many variants of the outcome, applying the same analytical strategy. Whereas Figure 2 above shows any involvement in traffic accidents, Table 3 below breaks down these accidents into subtypes, like involvement as driver versus pedestrian, time of day and so on.

The consistency is once again remarkable. Whereas the aggregate number is shown as 1.7x (top row), all the other numbers ranged from 1.5x to 5.1x. Almost all the variance is on the high side of the aggregate. What does such consistency tell us?

In the Appendix, the authors rolled out even more rows of analysis:

Again, all effects are in the range 1.4x to 1.7x. What does this consistency mean? That's what I want to explore in this post.

***

In the mainstream, such consistency is portrayed as resounding evidence. Each individual study that generates another impressive relative ratio similar to the previous result is regarded as another confirmation. The authors of the traffic accidents study believe that because everywhere they looked, they saw the same consistent number, it must mean that they have found something really important - like Newton's Law of Gravity.

Unlike Newton, they aren't studying the realm of physics. The study's data concern human behavior, and social behavior, and it is actually shocking, rather than comforting, to find a law of nature in this arena. My prior - from decades looking at data about humans - is that (a) aggregate effects are typically small, so that even a 10% improvement is remarkable, and (b) there is substantial variability when looking across different segments, different metrics, etc., which is another way to say that "interaction effects" are important.

Were these researchers, mainstream scientists, and the mass media being fooled by (lack of) randomness?

***

Other than discovering some immutable law of nature, what are other possible explanations for the remarkable consistency?

File Drawer Effect

Let's say there is a medical journal editor who only publishes vaccine studies that show high vaccine efficacy - since anything else has little societal value. The result of this editorial practice is that (almost) every study that sees the light of day will definitely have a VE estimate above some lower bound (say, 70%). In this scenario, the consistency of all the peer-reviewed results may have nothing to do with the vaccines themselves, but can be explained by publication bias that is filtering out studies below that bound. This bias is sometimes called the "file drawer effect," describing researchers who stack away "failed" studies (e.g. p-value > 5%) in their file cabinets since journals would reject them.

A consequence of the file drawer effect is that the average of the published results over-estimates the true vaccine efficacy.

In the context of Covid-19 vaccines, the file drawer effect may be related to the phenomonon of cargo cult science.

 

Cargo Cult Science

Cargo cult science is a term coined by physicist Richard Feynman. Roughly speaking, he warns against the potential of bad science even when scientists follow so-called best practices.

Other than the initial clinical trials conducted in 2020 (which were scientifically aborted upon emergency use authorization when the placebo groups were destroyed), all evidence of Covid-19 vaccine efficacy have come from observational datasets. All observational data sets either lack controls, or if controls existed, treatment status was not randomly assigned, which means that unvaccinated people differ from vaccinated people in multiple dimensions besides vaccination status. Therefore, it is wrong to attribute all observed differences between the two groups to vaccination only.

Let's teleport back to the first part of 2021, weeks after the mass vaccination campaign commenced, when the first observational study appeared in print. For the sake of argument, let's say the first analysis showed a VE of 50%. How do we know whether this estimate is to be believed since the people who got vaccinated were clearly different from those who didn't? The honest answer should be no one knows because there is no ground truth in an observational study.

What circumstantial evidence do we have? The only breadcrumbs are interim analysis results from randomized clinical trials that produced VE of 90%+. We face a gnarly problem because our hypothetical observational study led to a VE estimate of 50%. One of the numbers is not like the other; they cannot both be correct. But which study is wrong?

Not surprisingly, most would give more credence to the RCT results, since RCTs are considered the gold standard of medical research. Therefore, since our observational study came in much lower, we'd go back to the drawing board, and massage the dataset some more... until the estimated VE is closer to 90%, making it more credible. The additional data massaging is justified by best practices - we are simply addressing residual confounding bias.

(It is important to realize that randomized experiments can be analyzed in a straightforward, standardized manner but observational studies rely on statistical adjustments that involve subjective judgment from researchers. Analysts try to guess what biases exist in the data, and device ways of correcting for them. It's an iterative process with limited guardrails.)

To make sure you get what's going on, think about what happens if the first analysis of an observational dataset leads to a VE value of 90%. This result would be immediately accepted. Nevertheless, it's not clear that this research team did a better job adjusting for bias than the previous team who came up with 50%.

Since the RCT result was the only piece of available evidence when the first observational study showed up, it became a filter for all subsequent observational studies. Any observational study that produces a number close to 90% feels comfortable, and any study that shows a number far from 90% feels "wrong". Such an editorial process directly triggers the file drawer effect.

***

I don't know how we landed on this stupid state of affairs. Using an observational study to "confirm" an RCT result - the gold standard - is like asking a C student to grade an A student. After the teacher has already given the A student a top grade, what additional information is gleaned by having a C student grade the A student's exam?

What's happening is even more dubious. The C student is not grading the same exam that the teacher saw - the C student is asked to grade a different exam submitted by the A student. By this, I mean: the observational studies are different from the RCTs in meaningful ways, e.g. timing of vaccinations, incidence of Covid-19, types of people eligible for vaccination, residence of patients, no placebo shots... There is scant justification for assuming that the observational studies should produce VE values in the same ballpark as the RCTs!

When another observational study comes out proclaiming 70-90% VE, it doesn't mean much. It just tells me that the research team has selected a set of adjustments so that the VE estimate lands in the zone that is considered acceptable for publication, i.e. it falls in the vicinity of the RCT result.

 

Lurking Variables

Another way in which we can get fooled by the consistency of results is via unknown or unexplored sources of bias. All observational studies can only control for factors for which there are measurements. I haven't encountered yet an observational study of Covid-19 vaccines in which the researchers collected their own data. The current practice is to pull data from pre-existing, government databases. So the variables available are typically limited to basic demographics (age, gender, etc.) or health-related data (prior health concerns, healthcare usage, etc.). Important factors such as attitudes toward freedom and government authority, risk tolerance and exposure to the virus, are not measured, and not adjusted for.

If key sources of bias are not adjusted for, they could explain why the results are so consistent among the subgroups; each such result contains residual confounding from the missing variables.

The missing variable may be a common cause of both vaccination status and infection outcome, for example, people who are generally more health conscious and healthier pre-Covid. They are both more likely to get vaccinated and less likely to get infected (regardless of vaccination status). Since the government databases have no labels for this type of people, there is no way to correct this bias. It is delusional to think that dumping age and gender variables into a regression equation properly adjusted for such a bias. There are health-conscious people in every agexgender segment! In each such demographic segment, vaccination status is still confounded with health-seeking status.

 

Multiple Comparisons

When so many analyses are performed using so many metrics, the best practice is to adjust for multiple comparisons. The general idea is that if one screens a lot of metrics, a few of them will accidentally show statistical significance, and those results are highly likely to be spurious. In other words, without adjustment, traditional statistical methods underestimate the degree of variability in the outcomes. That is to say, results appear more consistent than they are. Yet, most of these vaccine studies sidestep this issue.

I'd be shocked if elementary medical statistics courses don't cover the problem of multiple comparisons. Practitioners in this discipline seem to think that fishing for significance is fine so long as they pre-specify all their intended analyses. Actually, pre-registration does not solve the problem of multiple comparisons at all, unless the pre-registered statistical analysis explicitly outlines a strategy for dealing with it. (Unfortunately, the randomized trials also pre-registered multiple comparisons without correction.)

 

Poorly Defined Metrics

Another reason for consistent results across many analyses is a poorly chosen scale for one's outcome metric. An analogy to grade inflation in schools is telling. American schools has a grade-point average (GPA) typically defined on a scale of 0-4, 4 being grade A. Due to the grade inflation trend, most of the grades given out are As so in reality, the majority of the scale 0-3.5 is basically empty, with almost all students squeezed into the 3.5-4.0 range. Of course, all students look more or less the same.

I think something like this is at play with the relative ratio scales used to measure vaccines. I think an ineffective vaccine will show a VE substantively higher than 0% so the grading scale is compressed just like in schools.

***

In this post, I have offered five possible reasons why every research study of Covid-19 vaccines seems to offer similar effectiveness estimates, and why there appears to be little or no variability when aspects of the methodology are altered. As a statistician who have analyzed a lot of human data, I find it least likely that the reason for the consistency is that these vaccines are like a law of nature, such an overwhelming force that flattens all human variability.

So what if rule-breakers are unvaccinated

In the last post on the Covid-19 vaccine and traffic accidents study (link), I reviewed the analysis methodology: I found it hard to believe the vaccinated and unvaccinated groups differ only by their vaccination status, even after “adjustments”, and thus I am not convinced by their causal interpretation.

In this post, I examine a few issues.

Right at the start, the research team proclaimed:

The proximate causes of most crashes are human behaviors including speeding, inattention, tailgating, impairment, improper passing, disobeying a signal, failing to yield right-of-way, or other infractions.

This assertion is pivotal to the research conclusion, which they exploited by using involvement in traffic accidents as a proxy for rule-breaking behavior on the road. The researchers speculated that an underlying trait of rule-breaking is the common cause of (a) a higher risk of traffic accidents and (b) more vaccine hesitancy which results in a lower chance of being vaccinated.

The causal diagram – even if true – can have few public policy implications. It explains a plausible self-selection bias in the unvaccinated group (inherent rule-breaking personality). It may suggest that if governments successfully “cure” this bias, then we’d simultaneously observe not just higher vaccination rates but also a reduction in traffic accidents, other traffic violations, other crimes, absenteeism in schools, fare evasion in public transport, tax evasion, etc. But solving general unruliness appears to me a much harder problem than solving vaccine hesitancy!

The claim that most traffic accidents are driver-caused is supposedly supported by a third-party reference but that study tallies only traffic accidents involving fatalities (which comprised fewer than 0.2% of the outcomes in this study), and even so, about half of the documented fatal accidents occurred under “normal” conditions, without any rule-breaking.

The causal proposal is further weakened by linking individuals to traffic accidents even if they were pedestrians or passengers, rather than drivers. These people – who could not have caused crashes through undesirable behavior – accounted for almost 60% of all outcomes.

Imagine a drugged out but unvaccinated driver carrying one passenger in the car ran a red light, and hit and killed a pedestrian. All three individuals are counted as involved in traffic accidents. The study authors would have concluded that the unvaccinated driver would not have crashed the vehicle if s/he had been vaccinated. By contrast, if the pedestrian was unvaccinated, then the study authors would have us believe that s/he would not be dead if vaccinated, regardless of whether the driver was vaccinated!

It’s a mess. It’s easier to avoid such a mistake if one collected the data through shoe leather.

***

One factor that drives vaccination behavior has received scant attention – risk tolerance. It’s well known, and not just among economists or psychologists, that humans have varying levels of risk-seeking or risk-averse behavior. Much of the insurance and gambling/gaming industry is built upon such heterogeneity. We also know that the measured risk of Covid-19 infection or case severity depends substantially on key demographic factors such as age, gender, occupation, and prior comorbidities. So it should be obvious that propensity to want the Covid-19 vaccine should vary with (a) predictable risk of infection and/or case severity (b) individual level of risk aversion and (c) individual’s assessment of probability of exposure, including the interaction between these factors.

The public-health narrative tends to adopt an axiom that everyone is alike, and that may be its biggest failure. As noted in the prior blog, the peer reviewers and the study’s researchers failed, by neglecting the most obvious proxies for the above factors – prior traffic accidents, and prior healthcare usage. They have no excuse for excluding these factors because the former was computed for the follow-up period while the latter was used in an exploratory matching analysis. The research finidng will be seen in a new light if it turns out that the unvaccinated group had been involved in more traffic accidents prior to the study period, for example.

***

All over the paper, risk is given in the form x per million chance of getting into traffic accidents. For example, the risk for unvaccinated was described as 912 per million, and for vaccinated, 530 per million. These are tiny numbers made to look huge. Since there were 1.8 million unvaccinated, the ~900 per million means about 900*1.8 = 1,620 unvaccinated people were involved in accidents, while 500*9.2 = 4,600 vaccinated people were involved in accidents. In what period of time?

Glad you asked :) The study has a follow-up period of just one month (30 days)! Literally, the research only covered the month of August, 2021. They did not look at any other month. (It’s quite likely there would be some reversion to the mean over time.)

912 per million is another way of saying 0.0912%. Roughly 0.1% of unvaccinated people were involved in a traffic accident during August 2021, and roughly 0.05% of vaccinated people were involved in a traffic accident. Only 16% of the population were unvaccinated. Even if we assumed correlation is causation, and if we forcibly vaccinated everyone, the impact of such a policy is 1.8 million * (0.05%) = 900 people. It’s 900 people out of 11 million, and remember, some of these are pedestrians or passengers, not drivers. Most of these accidents have zero fatalities.

This number represents the risk in a single month, not annualized. There is no explanation in the paper as to why such a short follow-up time was applied. Perhaps it has to do with the continuing vaccination campaign, so that the number of unvaccinated – and thus potential beneficiaries – keeps declining. (For those interested, none of the main results in the study censor or remove unvaccinated people who subsequently got vaccinated during or after August 2021.)

***

As with most Covid-19 observational studies, researchers selected a start date for counting outcomes, and didn’t bother to demonstrate that their findings are invariant to the choice of such a date. In this study, the follow-up start date was set as July 31, 2021, so the follow-up period comprises the single month of August 2021.

This is a strange date on which to settle. Take a look at the case curve in Ontario:

At the end of July 2021, cases in Ontario were at a trough, and during August, Covid-19 cases were steadily growing. This entire month missed the next peak of infection, which was toward the end of the year. In each Covid-19 wave, hospitalizations and deaths lag cases. The following curve shows Covid-19 deaths in Ontario:

Thus, many of the deaths associated with cases reported in August showed up after the 30-day follow-up window (in September). We should ignore any commentary on hospitalizations and deaths - specifically, the claim that this study “validate[d] that vaccination is associated with large reductions in subsequent COVID pneumonia.”

The authors pointed to Table 3 to support the COVID pneumonia claim. There, we find a total of 5,358 cases. This number is five times higher than the hospitalization count shown in the above chart for August 2021 (31 days x < 30 per day), implying less than 20% of patients with COVID pneumonia are hospitalized. This contradicts the description by Cleveland Clinic that says "If you’re diagnosed with COVID pneumonia, it’s likely that you’ll be admitted to the hospital." (link).

What else happened during August 2021? The Ontario government started to differentiate testing policies between vaccinated and unvaccinated mid-month (link). In some settings, only unvaccinated people were subject to routine testing. The more tests, the more cases found.

What else also happened during August 2021? According to this press release, the provincial government issued a vaccine mandate for "high risk" settings, made third doses more widely available, and expanded Pfizer vaccine to younger children. These policies create selection bias in the analysis groups.

The timing issue is an endemic problem with all Covid-19 observational studies. Researchers should repeat the same analysis methodology applied to different time periods and show us the results. I have a hunch that many of these results are transient.

 

P.S. [1/11/2023] Added link to the original study

The traffic accidents study exposes the lamentable state of Covid-19 science

tldr; The recent study tying lack of vaccinations to traffic accidents illustrates everything that is wrong with the state of our peer-reviewed scientific process. The study uses data that do not capture any of the important behavioral factors driving one's vaccination status and propensity to traffic accidents, and yet draws conclusions as if the biases in the observational data have been sufficiently cured. The researchers ignores some findings that contradict their headline and which reveal shortfalls of their methodology. These inconvenient results suggest that their regression model did not adequately correct the biases. The model structure does not align with the hypothesized causal structure, which in any case is far too complex to be handled by a basic regression. Worse of all, the study foregrounds an entirely inappropriate analysis, on to which reporters latched. Peer reviewers somehow looked the other way.

Late last year, several readers prodded me to blog about a sensational study urging people to take Covid-19 vaccines in order to prevent traffic accidents (link). Indeed, I had already noticed the breathless headlines, and flagged the study as suspected “science for emergency use” (i.e., not the usual science).

Now that I’ve read the paper, I confirm the initial suspicion. The study is not frivolous, as these researchers made efforts to strengthen the evidence on a finding that is predictably controversial. Nevertheless, they fell too much in love with a headline grabber, and missed some red flags.

As with most Covid-19 observational studies, these researchers gained access to a set of government databases – vaccinations, Covid-19 cases, demographics, traffic accidents, etc., linked them together, selected a “test group” and a “control group”, and performed several statistical analyses, ranging from simple tests of proportions to regressions with simple adjustments. Reflecting the preference during pandemic times, this type of research is conducted purely remotely, in an arms-length manner, without any engagement with the persons or objects under study. It’s quintessential ivory-tower data mining, no use for shoe leather.

For this study, the researchers obtained “N=All” data from Ontario, Canada. They selected about 11 million people to include in their analyses, divided into those who have taken zero vaccine shots (“unvaccinated”) and those who have taken one or more vaccine shots (“vaccinated”). The unvaccinated group comprised only 16% of the analysis population at the time of the analysis.

The researchers chose to highlight their least convincing result, perhaps because it carries the biggest “effect”. This is the claim that “unvaccinated individuals [experienced] a 72% increased relative risk [of traffic crashes] compared with those vaccinated”. This claim orginates from a simple, unadjusted comparison of the rate of traffic accidents between the two groups.

This simple, unadjusted comparison is completely useless! It is valid only under the obviously false assumption that the only difference between the “test group” and the “control group” is their vaccination status. Nevertheless, reporters latched onto it like sticky glue. I like to use the term “XYopia” (i.e. X-Y myopia) to describe the fallacy of pretending that the observed outcome Y (traffic accidents) is affected only by the offered explainer X (vaccinations) and nothing else.

In fact, the observed difference in traffic accident rates could be partially or wholly due to any number of other factors. After all, this type of research study is an exercise in data mining, not a carefully designed clinical trial. In every province and country, the unvaccinated group is expected to vary from the vaccinated group, not just in their vaccination status but due to self-selection and local, discriminatory vaccination policies. Thus, any unadjusted analysis ignoring other factors is pure garbage.

Why did the researchers let this happen? It’s not like they can’t smell the stench from a mile away. This is elementary stuff taught in the first lecture on observational studies. The paper actually included several adjusted analyses. Maybe they will argue that the simple analysis was “pre-registered”, and thus according to the norms of scientific publishing, they are “obligated” to headline it. The protocol is not published so we don't know. If true, this situation exposes the Achille’s heel of using “pre-registration” to preempt bad science – since patently absurd statistical methods could be pre-approved. (Besides, in many other Covid-19 studies, including trials, the protocols were continuously "revised" up to the time of analysis, rendering pre-registration toothless.)

After adjustment, the relative risk increase dropped to about 50% (the adjustment is insufficient, as indicated below).

***

The researchers are well aware that their finding would be controversial, obviously a prime example of correlation, not necessarily causation. This is also where "causation creep" shows up. Just like the majority of such studies, the authors clearly state that "correlation is not causation", but their conclusion - "Together, the findings suggest that unvaccinated adults need to be careful indoors with other people and outside with surrounding traffic" - makes sense only under a causal interpretation of the observed correlation.

Think about the situation faced by all research on observational data. If the finding is merely an interesting correlation, there is no reason to publish it, as the correlation could be spurious. The only studies that could get published are the ones in which the researchers somehow turn that observed correlation into causation. What's strange and wrong in my opinion is the declaration that they have found only a correlation when they have convinced themselves (and peer reviewers) that they have offered sufficient evidence of causality. If that's what they believe, they should own it and say it clearly.

To strengthen their own conviction of causality, the researchers sliced and diced the data some more, and found a similar signal almost everywhere. Instead of ringing alarm bells, such unlikely consistency is interpreted as confirming the aggregate result. But the signal isn’t “everywhere”, only everywhere they looked. If an overlooked factor was driving the aggregate result, it may well drive all the subgroup analyses, with the neglected factor cutting across all the known subgroups.

For example, in Table 2, the analysts replicated the simple unadjusted comparison of rates for a variety of subgroups, such as males, younger or older age, people with cancer, etc. We learn that among those older than 65 years old, those unvaccinated have a 30% lower risk of traffic accidents than those who are vaccinated. (This finding, just like the other findings, assumes all else is equal, which is obviously false.) Applying the same reasoning these authors used to interpret the 50% overall percentage increase in risk, we’d have to advise Ontario residents above 65 years old against getting the Covid-19 vaccine as doing so may make them more vulnerable to traffic accidents! This effect remains even after adjustment.

Next, in Table 3, the researchers performed the simple comparison of vaccinated vs. unvaccinated on subsidiary outcomes. One such outcome was involvement as pedestrian (rather than driver or passenger). Amusingly, being unvaccinated increased the risk of getting hit by vehicles by 40% (p = 0.00). Moreover, not getting the vaccine more than doubled the risk of traffic accidents *at night*.

Any attempt to explain away these inconvenient findings will involve invoking some other factors e.g. maybe unvaccinated people are more likely to have a night life – such theorizing does not rescue the analytical model but merely proves that it failed to account for all relevant factors.

The wayward results were disclosed in the paper – the research team chose to look the other way, focusing on the outcomes that support their headline-grabbing conclusion. But these results demonstrate that the "adjustments" applied have failed to cure the selection biases in the observational data.

***

It comes as little surprise that the simple adjustments do not cure the selection biases. (And when such biases are not cured, they could infect any analysis.) This problem is endemic in the Covid-19 literature. Most studies - including this one - rely on basic regression models that include just main effects of various demographic variables (age, gender, location, etc.). Very few of these studies publish the actual models - almost never do we see any tables of coefficient estimates or goodness of fit statistics. Almost no one discusses the causal structure associated with the structure of their regressions. Nevertheless, they all proceed as if their adjusted models have completely cured any selection biases.

The regression model in the traffic accidents study accounted for "baseline demographic and diagnostic predictors", which are not explicitly listed. Presumably they contain the items in Table 2: age (<40, 40-64, >65), gender, urban/rural, socioeconomic status (high, medium, low), prior alcohol misuse, prior sleep apnea, prior diabetes, prior depression, prior dementia, prior hypertension, prior cancer, prior Covid-19 infection.

Missing from this list is the most obvious one: prior traffic accidents. Another obvious one is prior vaccinations. I'd also include prior traffic violations. Yet another relevant factor is prior hospitalizations or hospital contacts. I'm not sure how peer reviewers missed all these.

This last factor is clearly important - but most of the Covid-19 researchers make a ridiculous decision - that is, they excluded people with recent hospital contacts, which improperly injects a healthy person bias into all such studies. This factor should not be used as an exclusion criterion but should be an adjustment variable.

The study's appendix includes a very telling causal graph:

I'm not going to adjudicate as to whether this diagram properly captures the complex relationships between these factors, nor whether it includes all relevant factors. Just focus on the white nodes - these are the only factors/outcomes that have real-world measurements, and are represented in the regression model. Everything else is not directly observable.

The regression model ignores all the unmeasured quantities so it is inconsistent with the above diagram. And yet, they assert that all biases have been cured. Alright - they didn't really claim all biases have been cured. In fact, they compiled a long list of factors that they knowingly ignored. In describing the "limitations" of the study, they listed the following factors: distrust of government, belief in freedom, misconception of everyday risks, faith in natural protection, antipathy toward regulation, chronic poverty, exposure to misinformation, insufficient resources, other personal beliefs, political identity, negative past experiences, limited health literacy, social networks, amount of driving, vehicle factors (speed, spacing, configuration, location, weather, distance driven), access to care, risk compensation, non-reported crashes, etc.

The researchers' stance is that since they don't have data for these factors, they are left as homework exercises for future research. That regrettably is normal research practice: even though many important factors are ignored, the study is still publishable since the researchers acknowledge the "limitations". It's like an investment banker disclosing all the reasons why a tech startup would tank, while still selling the IPO stock fervently. Here is a sample from Twitter's 2013 IPO that is good for giggles:

• "Our products and services may contain undetected software errors, which could harm our business and operating results."
• "We rely on assumptions and estimates to calculate certain of our key metrics, and real or perceived inaccuracies in such metrics may harm our reputation and negatively affect our business."
• "We have incurred significant operating losses in the past, and we may not be able to achieve or subsequently maintain profitability."
• "If we fail to grow our user base, or if user engagement or ad engagement on our platform decline, our revenue, business and operating results may be harmed."
• "If we fail to effectively manage our growth, our business and operating results could be harmed."

Maybe it's just me - if I believed those risk factors (plus many others), I wouldn't buy this stock. But of course, I wouldn't buy any tech stocks since they all use similar language.

If had read the limitations of the traffic accidents study first, I'd have saved a lot of time by not reading the study.

 

P.S. This post has run too long, and I'll post the second part next week.