Big Data, Plainly Spoken (aka Numbers Rule Your World)

Strategies for validating observational studies 1/2

As mentioned before, the team who did the research tying Covid-19 vaccinations to car accidents convinced themselves that they have found a real effect. They attempted a variety of confirmatory analyses. In the prior post, I described how they conducted variants of the basic analysis by analyzing subgroups and by modifying the outcome metrics.

It's instructive to look at several other validation strategies and discuss their strengths and weaknesses.

***

Every observational study delivers the following table, usually called Table 1 because it's the first table in the paper:

Table 1 compares the characteristics of the treatment group and the control group. In the traffic accidents study, the two comparison groups are the vaccinated and the unvaccinated residents of Ontario (after exclusions). For a randomized clinical trial, in which vaccination status would be randomly assigned, the two columns of Table 1 should look statistically the same. For an observational study, in which vaccination status is self-selected (or influenced by biases such as public-health priorities), the differences between the two columns provide hints as to the magnitude and direction of biases.

The first thing to notice is vaccinated residents outnumbered unvaccinated about 5 times. The features are divided into three groups: demographics, prior medical conditions (e.g. diagnoses of cancer in the past year), and something labeled "general". The "general" collection includes clinical contacts, hospital admissions, and emergency room visits in the prior year. These are important variables because prior health status is a likely source of self-selection bias.

The researchers have this to say about Table 1:

The 2 groups spanned a diverse range of demographics, with comparable general health care utilization (Table 1). The largest relative differences were that those who had not received a COVID vaccine were more likely to be younger, living in a rural area, and below the middle socioeconomic quintile. Those who had not received a vaccine also were more likely to have a diagnosis of alcohol misuse or depression and less likely to have a diagnosis of sleep apnea, diabetes, cancer, or dementia. About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

This language is highly typical of how researchers opine on the likely sources of bias, i.e. those characteristics that differ between the comparison groups. Now pay attention to the last sentence: About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

The actual data from Table 1 show 4.1% of vaccinated people had a documented Covid-19 infection in the past year, compared to 3.5% of unvaccinated. That's a relative ratio of 1.17. Given the large sample size (millions of people), a 0.6% difference in infection rates is highly significant! If we randomly selected 4/5 of the population to be vaccinated, the difference in infection rates between them and the unvaccinated would typically be much smaller than 0.6%.

So the claim of no major imbalance on Covid-19 prior infection is incorrect. It is also inconsistent because in the main analysis, based on 95% confidence intervals on relative risk, they declared they have found a significant correlation between vaccination status and traffic accidents. The same method would have found the difference between 3.5% and 4.1% prior infection rate to be highly significant. Indeed, if we apply the same method to all rows of Table 1, quite a few would also be found significantly different.

The prior-year emergency visits metric is of particular interest because the primary outcome being analyzed in this study was "emergency visits for individuals injured in traffic crashes". The unvaccinated group had a much higher prior rate of 26% compared to 20% for vaccinated, a relative ratio of almost 1.3, clearly significant. The analysts did not adjust for any of the variables in the general collection in their logistic regression, perhaps because they thought the two groups have "comparable general health care utilization," which as I just explained, is incorrect.

It's ok - in fact, expected - that the unvaccinated group would differ from the vaccinated group on some metrics. It's the opposite that should raise doubt. If the two columns are effectively equivalent, then the two comparison groups resemble those found in randomized trials. If we believe that, we're claiming that obvious sources of bias due to self-selection and public-health policy have not resulted in observed bias.

The biggest weakness of Table 1 analysis is missing variables. The comparisons are made only on the measured covariates. In the case of Covid-19, the measured covariates are whatever can be found in large government databases, collected for purposes other than conducting a study of traffic accidents and Covid-19 vaccination. The researchers said: "The available databases lacked information on driver skill, functional status, personality traits, traffic infractions, political affiliation, and self-identified ethnicity." Several of these variables are clearly pivotal for our present study.

In addition, Table 1 analysis addresses only biases acting independently. Some biases act in complex ways. When I analyzed prior emergency visits, the observed data appeared to indicate a healthy user bias in the vaccinated group. But this factor interacts with the age bias, which was also present. The vaccinated group was quite markedly older than the unvaccinated group. If older people are more likely to need emergency medical service than younger, then the emergency visit bias is even more pronounced since younger, unvaccinated people have a lower baseline emergency room usage than the older, vaccinated group. If younger people are more likely to need emergency medical service older, then age could explain some of the observed bias in emergency room usage. In other words, these factors should be interpreted simultaneously. In fact, any adjustment in the regression model should contain interaction effects, but this is rarely done.

***

The researchers also used "positive controls" and "negative controls" to convince themselves they have properly removed bias from this observational dataset. The idea is to process the data in the same way but substitute other metrics instead of the traffic accidents outcome. These other metrics are chosen because the researchers are 100% sure of the direction of the effect: for positive controls, they claim they knew ex-ante that the alternate outcomes are definitely affected by Covid-19 vaccination status while for negative controls, they claim they knew ex-ante that the alternate outcomes are definitely independent of Covid-19 vaccination status.

For negative control, they presented an analysis of constipation.

There were 2,985 cases of constipation during the month of August in the study population, and the relative risk was 1.03 meaning the unvaccinated group's risk was 3% above that of the vaccinated. This difference is not significant because the confidence interval of 0.94 to 1.14 crosses the 1.00 reference level. In other words, the data provided no evidence that unvaccinated people differ from vaccinated people in terms of experiencing constipation.

This type of validation analysis has many shortcomings. First, it's a single outcome. We don't know if this was pre-specified. Even if pre-specified, it could still show the desired result by chance since there are hypothetically hundreds or thousands of possible negative control candidates. If it's not pre-specified, it's hard to know if other candidates were tried and failed the test.

Perhaps because of reviewer feedback, the research team added several other negative controls, namely, fall, water craft, appendicitis and conjunctivitis. Perhaps because these were not in the original analyses, they did not discuss these other negative controls in the article.

I excerpted the table of results below.

The last column presents the confidence interval of the relative risk, and the second last column from the right shows the point estimate of the relative risk.

The second problem is inconclusiveness. Notice the point estimates of all five negative controls are tightly clustered between 0.90 and 1.14, and every confidence interval except "falls" shows a statistically insignificant difference between the vaccinated and unvaccinated groups. How should one interpret the result of "falls"? The researchers were silent on that one. When a negative control (constipation) shows insignificance, as expected, they were quick to say this finding validated the study conclusion but when a negative control shows a significant difference, it appeared that they explained it away, probably arguing it's a chance observation.

The third problem is over-confidence. How do they know that vaccination could have zero impact on constipation? I imagine this relies on some kind of structural biological argument, such as that mRNA does not act on certain bodily systems. Remember that statistical analyses describe correlations, not causation. We have to rule out ex-ante not just a direct causal relation between vaccination and constipation but any kind of correlation. That's actually a deep assumption, not as simple as it looks.

***

Positive controls are even harder to grapple with because we have to believe ex-ante that vaccination status will certainly result in some given outcome, in this case, "covid pneumonia".

The use of "covid pneumonia" implies the absolute belief that vaccinations reduce covid pneumonia in the analysis population. A few aborted clinical trials and many observational studies do not constitute incontrovertible evidence. Imagine if the data showed no significant difference in covid pneumonia between the unvaccinated and vaccinated. What would be the conclusion? That the methodology is invalidated? That the assumption of causation was incorrect?

These validation analyses using positive or negative controls have limited utility. They produce a situation in which the only possible conclusion is supporting the study - if these analyses do not support the study, we don't know how to interpret them so we throw them out and find another metric to act as control. (As a practical matter, no one would publish the paper if the authors include "failed" controls. See my prior post on the file drawer effect.)

***

The next tactic used for validation is propensity score matching. I'll discuss that in the next post.

Prior posts about this study: 1, 2, 3

 

 

Know your data 34: we reserve the right to refuse service to anyone

At the dawn of the age of surveillance tech, Google's then CEO Eric Schmidt famously quipped that privacy is not a thing, and one should not worry if one isn't doing anything wrong. The fallacy of that statement has always been there: "wrong" is in the eye of the beholder!

The future has arrived. The news came out that the management of Madison Square Garden, the venue for many a grand sports event in NYC, has been using facial recognition technology to pick out people on their "enemies list" and expel them from MSG events.

Who are these enemies? We know at least one group - lawyers who work for law firms that are on the wrong side of lawsuits involving MSG. Without a doubt, there are other types of enemies on the list, we just don't know.

What wrong have these lawyers done? Their offense is to be employed by law firms that have taken on certain cases.

Take this to its absurd conclusion: one cannot work for law firms anymore. There are always opponents on every case so any number of establishments can ban you.

Not just lawyers either.

Amazon can ban you because you dare shop at Walmart. Delta can ban you because you dare fly on United. Tesla can ban you because you made fun of Elon Musk on Twitter. James Dolan, MSG's CEO, already threatened journalists who dared object to his use of facial recognition tech.

Don't forget six degrees.

Facebook (I mean, Meta) bans you from Instagram because you work for Google. Wait, Facebook has the people graph, so it can ban you because your spouse works for Google, or your best friend from high school works for Google.

***

With the number of surveillance cameras mounted everywhere, it'd be a surprise that we don't already have an app that tells us where any given person is currently located. Want to know if your boyfriend or girlfriend is cheating? There's an app for it. Want to know if your kid is skipping classes or out partying late? There's an app for it. Want to know if your employee is working from home or slacking off? There's an app for it. Want to know what time your boss left the office? There's an app for it.

There is no natural law to confine surveillance technologies to use cases that benefit users. Indeed, such technology has acknowledged benefits, such as helping law enforcement catch burglars or murderers. Some people may find it "convenient" not having to face another human when checking out their groceries while consenting to having their faces captured and stored in databases.

It's the same technology: if you desire the good, you have to accept the consequences of the bad.

***

I read that Amazon is just now testing pay by your palmprint. I know, that article cites senators who are "concerned" about the palmprint technology. It's amusing that they are not concerned about government usage of surveillance tech (unless it's a foreign government), or so-called public-private partnerships that lead to surveillance data being passed from companies to governments. We seem to be comforting ourselves that surveillance tech is fine because we have a benevolent government. History also tells us that autocracies are fine so long as the rulers are benevolent.

 

 

Too good to be true instinct

Good data analysts have developed an instinct of perking up when results are "too good to be true". This is a must-have attribute that I'm always looking for in a job candidate. It is a protective instinct, a response that protects the analysts from getting fooled by the data. Nassim Taleb called this type of thing "fooled by randomness". The temptation is not randomness itself but the lack thereof.

(Credit: Sonja Alves, Flickr)

Observational studies of the Covid-19 vaccines are a case in point. I have been spooked by the consistency of estimates of vaccine effectiveness. Almost all peer-reviewed studies and most working papers that get mainstream attention contain values of VE that are in the 70-90 percent range. (Since these are relative ratios, a 50-percent multiple improvement is already a huge effect.) There is a consistency across headline results of different studies, and there is also a consistency of values inside individual studies, across segments, using different outcome metrics, different methodologies, etc. etc.

Below is Table 2 from the traffic accidents study that I have been reviewing (link):

The aggregate result (derived from the inadmissible non-adjusted comparison discussed in the first post of this series) is shown in the last row and represents a value of 1.7x. The rows above come from applying the same methodology to subsets of the study population, some of them are mutually exclusive while others are overlapping. Most of the values line up close to or near that 1.7x value. There is only one segment (over 65) that has the opposite direction but barely. What does this consistency tell us?

Shown below is the top part of Table 3. Here, the researchers analyzed many variants of the outcome, applying the same analytical strategy. Whereas Figure 2 above shows any involvement in traffic accidents, Table 3 below breaks down these accidents into subtypes, like involvement as driver versus pedestrian, time of day and so on.

The consistency is once again remarkable. Whereas the aggregate number is shown as 1.7x (top row), all the other numbers ranged from 1.5x to 5.1x. Almost all the variance is on the high side of the aggregate. What does such consistency tell us?

In the Appendix, the authors rolled out even more rows of analysis:

Again, all effects are in the range 1.4x to 1.7x. What does this consistency mean? That's what I want to explore in this post.

***

In the mainstream, such consistency is portrayed as resounding evidence. Each individual study that generates another impressive relative ratio similar to the previous result is regarded as another confirmation. The authors of the traffic accidents study believe that because everywhere they looked, they saw the same consistent number, it must mean that they have found something really important - like Newton's Law of Gravity.

Unlike Newton, they aren't studying the realm of physics. The study's data concern human behavior, and social behavior, and it is actually shocking, rather than comforting, to find a law of nature in this arena. My prior - from decades looking at data about humans - is that (a) aggregate effects are typically small, so that even a 10% improvement is remarkable, and (b) there is substantial variability when looking across different segments, different metrics, etc., which is another way to say that "interaction effects" are important.

Were these researchers, mainstream scientists, and the mass media being fooled by (lack of) randomness?

***

Other than discovering some immutable law of nature, what are other possible explanations for the remarkable consistency?

File Drawer Effect

Let's say there is a medical journal editor who only publishes vaccine studies that show high vaccine efficacy - since anything else has little societal value. The result of this editorial practice is that (almost) every study that sees the light of day will definitely have a VE estimate above some lower bound (say, 70%). In this scenario, the consistency of all the peer-reviewed results may have nothing to do with the vaccines themselves, but can be explained by publication bias that is filtering out studies below that bound. This bias is sometimes called the "file drawer effect," describing researchers who stack away "failed" studies (e.g. p-value > 5%) in their file cabinets since journals would reject them.

A consequence of the file drawer effect is that the average of the published results over-estimates the true vaccine efficacy.

In the context of Covid-19 vaccines, the file drawer effect may be related to the phenomonon of cargo cult science.

 

Cargo Cult Science

Cargo cult science is a term coined by physicist Richard Feynman. Roughly speaking, he warns against the potential of bad science even when scientists follow so-called best practices.

Other than the initial clinical trials conducted in 2020 (which were scientifically aborted upon emergency use authorization when the placebo groups were destroyed), all evidence of Covid-19 vaccine efficacy have come from observational datasets. All observational data sets either lack controls, or if controls existed, treatment status was not randomly assigned, which means that unvaccinated people differ from vaccinated people in multiple dimensions besides vaccination status. Therefore, it is wrong to attribute all observed differences between the two groups to vaccination only.

Let's teleport back to the first part of 2021, weeks after the mass vaccination campaign commenced, when the first observational study appeared in print. For the sake of argument, let's say the first analysis showed a VE of 50%. How do we know whether this estimate is to be believed since the people who got vaccinated were clearly different from those who didn't? The honest answer should be no one knows because there is no ground truth in an observational study.

What circumstantial evidence do we have? The only breadcrumbs are interim analysis results from randomized clinical trials that produced VE of 90%+. We face a gnarly problem because our hypothetical observational study led to a VE estimate of 50%. One of the numbers is not like the other; they cannot both be correct. But which study is wrong?

Not surprisingly, most would give more credence to the RCT results, since RCTs are considered the gold standard of medical research. Therefore, since our observational study came in much lower, we'd go back to the drawing board, and massage the dataset some more... until the estimated VE is closer to 90%, making it more credible. The additional data massaging is justified by best practices - we are simply addressing residual confounding bias.

(It is important to realize that randomized experiments can be analyzed in a straightforward, standardized manner but observational studies rely on statistical adjustments that involve subjective judgment from researchers. Analysts try to guess what biases exist in the data, and device ways of correcting for them. It's an iterative process with limited guardrails.)

To make sure you get what's going on, think about what happens if the first analysis of an observational dataset leads to a VE value of 90%. This result would be immediately accepted. Nevertheless, it's not clear that this research team did a better job adjusting for bias than the previous team who came up with 50%.

Since the RCT result was the only piece of available evidence when the first observational study showed up, it became a filter for all subsequent observational studies. Any observational study that produces a number close to 90% feels comfortable, and any study that shows a number far from 90% feels "wrong". Such an editorial process directly triggers the file drawer effect.

***

I don't know how we landed on this stupid state of affairs. Using an observational study to "confirm" an RCT result - the gold standard - is like asking a C student to grade an A student. After the teacher has already given the A student a top grade, what additional information is gleaned by having a C student grade the A student's exam?

What's happening is even more dubious. The C student is not grading the same exam that the teacher saw - the C student is asked to grade a different exam submitted by the A student. By this, I mean: the observational studies are different from the RCTs in meaningful ways, e.g. timing of vaccinations, incidence of Covid-19, types of people eligible for vaccination, residence of patients, no placebo shots... There is scant justification for assuming that the observational studies should produce VE values in the same ballpark as the RCTs!

When another observational study comes out proclaiming 70-90% VE, it doesn't mean much. It just tells me that the research team has selected a set of adjustments so that the VE estimate lands in the zone that is considered acceptable for publication, i.e. it falls in the vicinity of the RCT result.

 

Lurking Variables

Another way in which we can get fooled by the consistency of results is via unknown or unexplored sources of bias. All observational studies can only control for factors for which there are measurements. I haven't encountered yet an observational study of Covid-19 vaccines in which the researchers collected their own data. The current practice is to pull data from pre-existing, government databases. So the variables available are typically limited to basic demographics (age, gender, etc.) or health-related data (prior health concerns, healthcare usage, etc.). Important factors such as attitudes toward freedom and government authority, risk tolerance and exposure to the virus, are not measured, and not adjusted for.

If key sources of bias are not adjusted for, they could explain why the results are so consistent among the subgroups; each such result contains residual confounding from the missing variables.

The missing variable may be a common cause of both vaccination status and infection outcome, for example, people who are generally more health conscious and healthier pre-Covid. They are both more likely to get vaccinated and less likely to get infected (regardless of vaccination status). Since the government databases have no labels for this type of people, there is no way to correct this bias. It is delusional to think that dumping age and gender variables into a regression equation properly adjusted for such a bias. There are health-conscious people in every agexgender segment! In each such demographic segment, vaccination status is still confounded with health-seeking status.

 

Multiple Comparisons

When so many analyses are performed using so many metrics, the best practice is to adjust for multiple comparisons. The general idea is that if one screens a lot of metrics, a few of them will accidentally show statistical significance, and those results are highly likely to be spurious. In other words, without adjustment, traditional statistical methods underestimate the degree of variability in the outcomes. That is to say, results appear more consistent than they are. Yet, most of these vaccine studies sidestep this issue.

I'd be shocked if elementary medical statistics courses don't cover the problem of multiple comparisons. Practitioners in this discipline seem to think that fishing for significance is fine so long as they pre-specify all their intended analyses. Actually, pre-registration does not solve the problem of multiple comparisons at all, unless the pre-registered statistical analysis explicitly outlines a strategy for dealing with it. (Unfortunately, the randomized trials also pre-registered multiple comparisons without correction.)

 

Poorly Defined Metrics

Another reason for consistent results across many analyses is a poorly chosen scale for one's outcome metric. An analogy to grade inflation in schools is telling. American schools has a grade-point average (GPA) typically defined on a scale of 0-4, 4 being grade A. Due to the grade inflation trend, most of the grades given out are As so in reality, the majority of the scale 0-3.5 is basically empty, with almost all students squeezed into the 3.5-4.0 range. Of course, all students look more or less the same.

I think something like this is at play with the relative ratio scales used to measure vaccines. I think an ineffective vaccine will show a VE substantively higher than 0% so the grading scale is compressed just like in schools.

***

In this post, I have offered five possible reasons why every research study of Covid-19 vaccines seems to offer similar effectiveness estimates, and why there appears to be little or no variability when aspects of the methodology are altered. As a statistician who have analyzed a lot of human data, I find it least likely that the reason for the consistency is that these vaccines are like a law of nature, such an overwhelming force that flattens all human variability.

Is this data science professor a victim or a villain?

Apparently, some secretive entity is trying to maximally embarrass Forbes magazine as much as possible. One by one, they are exposing people on Forbes' 30 under 30 or 40 under 40 list of young entrepreneurs as fraudsters. Elizabeth Holmes has already been found guilty of fraud in her medical tech startup. Sam Bankman-Fried is being sued for running a type of Ponzi scheme in his crypto startup. And now, Charlie Javice is being sued by JPMorgan for faking customer lists in her college financial aid app startup (named Frank).

According to the Wall Street Journal (link with paywall), an unnamed data science professor played a role in this fraud, and that definitely caught my attention.

The unnamed professor, who was paid $18,000, used computer-generated information to show Frank’s purported user base, including details like names, birthdays and schools they attended, the suit alleged...“Our plan was to sample first name and last name independently and then ensure none of the sampled names are real,” the suit alleges the professor wrote to Ms. Javice.

This sounds very bad. Apparently, JPMorgan alleged that about 90% of the data were faked.

However, I don't think we can read from this excerpt whether the professor has also been duped, or the professor was part of the con. It could be the case that Javice contracted the professor to generate simulated data, supposedly for some legitimate purpose.

The other reason this case strikes a chord with me is that many of the emerging data science/AI technologies can easily be exploited to deceive and hurt people, and there does not appear to be much serious discussion of this problem.

 

P.S. [1/19/2023] My friend Mark P. sent me a note saying that some people on Twitter who has read the lawsuit learned that Excel is featured in it. Apparently, JP Morgan got suspicious when they received an Excel spreadsheet with the customer data, which cut off at ~1 million rows, which is the row limit for Excel files.

I have to say I don't really understand this point because I'd have thought that their IT people didn't realize that 4 million names don't fit into an Excel spreadsheet; I wouldn't have immediately assumed they had fewer customers than they claimed.