Big Data, Plainly Spoken (aka Numbers Rule Your World)

When you don't need data

I bet you can't believe I'm going to say I don't need/want data on a blog about intelligent thinking about data.

I'm going to argue that intelligent thinking about data includes recognizing when you don't need data - and by extension, you don't need more data.

This post is motivated by opinions that are circulating about how certain public health measures against infectious diseases such as masks and lockdowns are 100% useless - although if you start thinking about other domains, you can find pertinent examples.

Let's start with lockdowns first as that's more black and white. I don't need any data to believe that lockdowns reduce infections. I'd need data if I want to establish the magnitude of protection but for the direction, I don't need data. Why? Because I have theory.

What theory? Infectious diseases spread by infections, which requires contacts. Turning this around, one can say no infection can occur unless there is contact. During the early part of the pandemic, and especially before vaccines were available, in key hotspots, schools and businesses were closed; people were asked to stay home; employees were allowed to work from home. As a result, the frequency of contacts each of us had with others was drastically reduced, and for some, reduced to near zero. By the theory, the dramatic reduction of contacts resulted in a reduction of infections.

Do I need data to prove that statement? No. [The word "theory" is weird. By theory, I mean an immutable phenomenon, such as incorporating the law of gravity in a physical model. There's probably a better word for this. Structural model?]  

***

Would I like to have data? Sure. But it's not a must-have. If I had data, I hope to answer more questions, like how much reduction? does it affect all demographic segments equally?

Would I not want to have data? There is even an argument for this (although this sentiment won't be universally shared). I suspect strongly that any data that could be made available to me would be almost impossible to interpret - because we cannot run randomized experiments on lockdowns. Thus, it would take a lot of effort to clean the data, and to adjust the data, and none of this can be accomplished without making lots of subjective assumptions, sure to enrage some and delight others.

For example, all such data confound the effects of lockdowns and vaccinations because most places that had lockdowns also pushed vaccines simultaneously. What's more, masking, social distancing, and other measures were also simultaneously put to work. So, if someone had the data, it is likely to confuse rather than illuminate. Let's not forget about enforcement of lockdowns, and compliance to lockdowns. Contacts might not have been reduced if the lockdown was in name only!

So if I had data, it could show infections going up, down or sideways, and the effect of lockdowns would be masked by all the other different factors. More data could be useless.

***

More data, however, could not overturn my theory... otherwise, my theory is just conjecture. If the data happened to show that, after adjusting for all kinds of other factors, lockdowns have zero or negative impact on infections, I'd be much more likely to reject some of the subjective assumptions or the indirect data proxies, rather than to question the theory, which in this case, means I have discovered evidence that infections occur without contacts.

Even if surveillance data show that compliance to lockdowns was nonexistent, the proper way to interpret the data is to say that lockdowns by themselves should reduce infections but its effect is masked by compliance or lack thereof, so that data that confound both factors show that infections did not drop. Embedded in there is still the theory that reduced contacts reduces infections.

The theory - the structural model - is an immutable part of the larger statistical model. We expect data to conform to this structure. We don't use data to modify the theory. (This structural model is not the same thing as a prior in Bayesian models. Bayesian priors are subject to updating on observing data so it does not represent anything immutable.)

***
There are other examples in other domains as well. Here is Andrew's model of golf putting (link). The section called "modeling from first principles" is an example of a statistical model that embeds a (geometric) theory. He also later uses the term "geometry-based" model.

The simple geometry is not enough, and further iterations of the model add other factors but the original geomtry is still in there.

***
This post records my first thoughts on this topic. I'll get to masks next.

Strategies for validating observational studies 2/2

This post continues the previous post about strategies for validating observational studies. Both posts are inspired by my reading of the paper that claims to have discovered that people who do not take the Covid-19 vaccines are more prone to serious traffic accidents.

***

Another tactic used by the researchers to validate their findings is using a different analytical methodology to arrive at the same conclusion.

In this case, they used propensity score matching to create synthetic comparison groups. In matching studies, we accept that the vaccinated and unvaccinated groups are not directly comparable; we match pairs of people, one from each group, based on known matching variables to make sure that the matched pairs themselves are comparable.

The paper does not provide many details on how these researchers did the matching. The matching variables are the same variables used in adjusting the regression models, i.e. the demographics and prior diagnoses. These matching variables are summarized into a single number known as the "propensity score". Matched pairs have the same propensity scores (this description is slightly imprecise because propensity scores are probabilities and cannot be matched exactly. They would have matched based on ranges of values - known as calipers - but none of these details can be found even in the Supplementary Appendix.)

Nonetheless, matching cannot cure imbalance due to unobserved variables, for which there are many. As noted in the previous post, many obviously influential factors - such as exposure to driving, and driving skills - are not measured and therefore not used in the matching analysis.

Matched studies are hard to generalize. The research team reported two attempts at matching (we don't ever know how many they tried, just how many they reported trying). In the first attempt, they kept essentially the entire unvaccinated group (about 20% of the study population). Since they matched one vaccinated person to each unvaccinated, this means three out of four vaccinated persons were excluded from the study. Exclusion does not necessarily mean they are unmatchable; for each unvaccinated person, there could be multiple candidates in the vaccinated pool who match on the propensity score, all but one of whom would be thrown away.

If matching has cured most of the selection bias, the comparison between the matched pairs would be valid. But for whom? The treatment effect is estimated for people who look like those who were matched - in the first attempted matching, this means people who look like the unvaccinated.

Nevertheless, one can easily catch researchers who speak as if they can apply their finding to everyone! If such generalization is admissible, then what's the point of the matching?

The second matching attempt is more "stringent". This means the matched pairs are more similar to each other. What does it also mean? If you're following my drift, more stringent matching means the match rate is even lower, which in turns means the finding applies to an even smaller subset of the study population. As often in statistics, there is no free lunch. 

This stringent matching resulted in about 600K in each group. That means 70% of the unvaccinated, and 94% of the vaccinated persons were excluded from the matching analysis. Not useful if you ask me.

***

The problem with using propensity score matching to validate the adjusted regression model is that each method works with a different analysis population, and so the estimates are not directly comparable.

Strategies for validating observational studies 1/2

As mentioned before, the team who did the research tying Covid-19 vaccinations to car accidents convinced themselves that they have found a real effect. They attempted a variety of confirmatory analyses. In the prior post, I described how they conducted variants of the basic analysis by analyzing subgroups and by modifying the outcome metrics.

It's instructive to look at several other validation strategies and discuss their strengths and weaknesses.

***

Every observational study delivers the following table, usually called Table 1 because it's the first table in the paper:

Table 1 compares the characteristics of the treatment group and the control group. In the traffic accidents study, the two comparison groups are the vaccinated and the unvaccinated residents of Ontario (after exclusions). For a randomized clinical trial, in which vaccination status would be randomly assigned, the two columns of Table 1 should look statistically the same. For an observational study, in which vaccination status is self-selected (or influenced by biases such as public-health priorities), the differences between the two columns provide hints as to the magnitude and direction of biases.

The first thing to notice is vaccinated residents outnumbered unvaccinated about 5 times. The features are divided into three groups: demographics, prior medical conditions (e.g. diagnoses of cancer in the past year), and something labeled "general". The "general" collection includes clinical contacts, hospital admissions, and emergency room visits in the prior year. These are important variables because prior health status is a likely source of self-selection bias.

The researchers have this to say about Table 1:

The 2 groups spanned a diverse range of demographics, with comparable general health care utilization (Table 1). The largest relative differences were that those who had not received a COVID vaccine were more likely to be younger, living in a rural area, and below the middle socioeconomic quintile. Those who had not received a vaccine also were more likely to have a diagnosis of alcohol misuse or depression and less likely to have a diagnosis of sleep apnea, diabetes, cancer, or dementia. About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

This language is highly typical of how researchers opine on the likely sources of bias, i.e. those characteristics that differ between the comparison groups. Now pay attention to the last sentence: About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

The actual data from Table 1 show 4.1% of vaccinated people had a documented Covid-19 infection in the past year, compared to 3.5% of unvaccinated. That's a relative ratio of 1.17. Given the large sample size (millions of people), a 0.6% difference in infection rates is highly significant! If we randomly selected 4/5 of the population to be vaccinated, the difference in infection rates between them and the unvaccinated would typically be much smaller than 0.6%.

So the claim of no major imbalance on Covid-19 prior infection is incorrect. It is also inconsistent because in the main analysis, based on 95% confidence intervals on relative risk, they declared they have found a significant correlation between vaccination status and traffic accidents. The same method would have found the difference between 3.5% and 4.1% prior infection rate to be highly significant. Indeed, if we apply the same method to all rows of Table 1, quite a few would also be found significantly different.

The prior-year emergency visits metric is of particular interest because the primary outcome being analyzed in this study was "emergency visits for individuals injured in traffic crashes". The unvaccinated group had a much higher prior rate of 26% compared to 20% for vaccinated, a relative ratio of almost 1.3, clearly significant. The analysts did not adjust for any of the variables in the general collection in their logistic regression, perhaps because they thought the two groups have "comparable general health care utilization," which as I just explained, is incorrect.

It's ok - in fact, expected - that the unvaccinated group would differ from the vaccinated group on some metrics. It's the opposite that should raise doubt. If the two columns are effectively equivalent, then the two comparison groups resemble those found in randomized trials. If we believe that, we're claiming that obvious sources of bias due to self-selection and public-health policy have not resulted in observed bias.

The biggest weakness of Table 1 analysis is missing variables. The comparisons are made only on the measured covariates. In the case of Covid-19, the measured covariates are whatever can be found in large government databases, collected for purposes other than conducting a study of traffic accidents and Covid-19 vaccination. The researchers said: "The available databases lacked information on driver skill, functional status, personality traits, traffic infractions, political affiliation, and self-identified ethnicity." Several of these variables are clearly pivotal for our present study.

In addition, Table 1 analysis addresses only biases acting independently. Some biases act in complex ways. When I analyzed prior emergency visits, the observed data appeared to indicate a healthy user bias in the vaccinated group. But this factor interacts with the age bias, which was also present. The vaccinated group was quite markedly older than the unvaccinated group. If older people are more likely to need emergency medical service than younger, then the emergency visit bias is even more pronounced since younger, unvaccinated people have a lower baseline emergency room usage than the older, vaccinated group. If younger people are more likely to need emergency medical service older, then age could explain some of the observed bias in emergency room usage. In other words, these factors should be interpreted simultaneously. In fact, any adjustment in the regression model should contain interaction effects, but this is rarely done.

***

The researchers also used "positive controls" and "negative controls" to convince themselves they have properly removed bias from this observational dataset. The idea is to process the data in the same way but substitute other metrics instead of the traffic accidents outcome. These other metrics are chosen because the researchers are 100% sure of the direction of the effect: for positive controls, they claim they knew ex-ante that the alternate outcomes are definitely affected by Covid-19 vaccination status while for negative controls, they claim they knew ex-ante that the alternate outcomes are definitely independent of Covid-19 vaccination status.

For negative control, they presented an analysis of constipation.

There were 2,985 cases of constipation during the month of August in the study population, and the relative risk was 1.03 meaning the unvaccinated group's risk was 3% above that of the vaccinated. This difference is not significant because the confidence interval of 0.94 to 1.14 crosses the 1.00 reference level. In other words, the data provided no evidence that unvaccinated people differ from vaccinated people in terms of experiencing constipation.

This type of validation analysis has many shortcomings. First, it's a single outcome. We don't know if this was pre-specified. Even if pre-specified, it could still show the desired result by chance since there are hypothetically hundreds or thousands of possible negative control candidates. If it's not pre-specified, it's hard to know if other candidates were tried and failed the test.

Perhaps because of reviewer feedback, the research team added several other negative controls, namely, fall, water craft, appendicitis and conjunctivitis. Perhaps because these were not in the original analyses, they did not discuss these other negative controls in the article.

I excerpted the table of results below.

The last column presents the confidence interval of the relative risk, and the second last column from the right shows the point estimate of the relative risk.

The second problem is inconclusiveness. Notice the point estimates of all five negative controls are tightly clustered between 0.90 and 1.14, and every confidence interval except "falls" shows a statistically insignificant difference between the vaccinated and unvaccinated groups. How should one interpret the result of "falls"? The researchers were silent on that one. When a negative control (constipation) shows insignificance, as expected, they were quick to say this finding validated the study conclusion but when a negative control shows a significant difference, it appeared that they explained it away, probably arguing it's a chance observation.

The third problem is over-confidence. How do they know that vaccination could have zero impact on constipation? I imagine this relies on some kind of structural biological argument, such as that mRNA does not act on certain bodily systems. Remember that statistical analyses describe correlations, not causation. We have to rule out ex-ante not just a direct causal relation between vaccination and constipation but any kind of correlation. That's actually a deep assumption, not as simple as it looks.

***

Positive controls are even harder to grapple with because we have to believe ex-ante that vaccination status will certainly result in some given outcome, in this case, "covid pneumonia".

The use of "covid pneumonia" implies the absolute belief that vaccinations reduce covid pneumonia in the analysis population. A few aborted clinical trials and many observational studies do not constitute incontrovertible evidence. Imagine if the data showed no significant difference in covid pneumonia between the unvaccinated and vaccinated. What would be the conclusion? That the methodology is invalidated? That the assumption of causation was incorrect?

These validation analyses using positive or negative controls have limited utility. They produce a situation in which the only possible conclusion is supporting the study - if these analyses do not support the study, we don't know how to interpret them so we throw them out and find another metric to act as control. (As a practical matter, no one would publish the paper if the authors include "failed" controls. See my prior post on the file drawer effect.)

***

The next tactic used for validation is propensity score matching. I'll discuss that in the next post.

Prior posts about this study: 1, 2, 3

 

 

Know your data 34: we reserve the right to refuse service to anyone

At the dawn of the age of surveillance tech, Google's then CEO Eric Schmidt famously quipped that privacy is not a thing, and one should not worry if one isn't doing anything wrong. The fallacy of that statement has always been there: "wrong" is in the eye of the beholder!

The future has arrived. The news came out that the management of Madison Square Garden, the venue for many a grand sports event in NYC, has been using facial recognition technology to pick out people on their "enemies list" and expel them from MSG events.

Who are these enemies? We know at least one group - lawyers who work for law firms that are on the wrong side of lawsuits involving MSG. Without a doubt, there are other types of enemies on the list, we just don't know.

What wrong have these lawyers done? Their offense is to be employed by law firms that have taken on certain cases.

Take this to its absurd conclusion: one cannot work for law firms anymore. There are always opponents on every case so any number of establishments can ban you.

Not just lawyers either.

Amazon can ban you because you dare shop at Walmart. Delta can ban you because you dare fly on United. Tesla can ban you because you made fun of Elon Musk on Twitter. James Dolan, MSG's CEO, already threatened journalists who dared object to his use of facial recognition tech.

Don't forget six degrees.

Facebook (I mean, Meta) bans you from Instagram because you work for Google. Wait, Facebook has the people graph, so it can ban you because your spouse works for Google, or your best friend from high school works for Google.

***

With the number of surveillance cameras mounted everywhere, it'd be a surprise that we don't already have an app that tells us where any given person is currently located. Want to know if your boyfriend or girlfriend is cheating? There's an app for it. Want to know if your kid is skipping classes or out partying late? There's an app for it. Want to know if your employee is working from home or slacking off? There's an app for it. Want to know what time your boss left the office? There's an app for it.

There is no natural law to confine surveillance technologies to use cases that benefit users. Indeed, such technology has acknowledged benefits, such as helping law enforcement catch burglars or murderers. Some people may find it "convenient" not having to face another human when checking out their groceries while consenting to having their faces captured and stored in databases.

It's the same technology: if you desire the good, you have to accept the consequences of the bad.

***

I read that Amazon is just now testing pay by your palmprint. I know, that article cites senators who are "concerned" about the palmprint technology. It's amusing that they are not concerned about government usage of surveillance tech (unless it's a foreign government), or so-called public-private partnerships that lead to surveillance data being passed from companies to governments. We seem to be comforting ourselves that surveillance tech is fine because we have a benevolent government. History also tells us that autocracies are fine so long as the rulers are benevolent.