Big Data, Plainly Spoken (aka Numbers Rule Your World)

Who's the richest person in the world right now?

Business Insider is running out of breath from having to write and rewrite headlines recently announcing that Elon Musk is now the richest man in the world, no Musk has lost his #1 spot, no he has regained it, etc.

This is happening because much of the wealth of really rich people is tied up in stock, whose value fluctuate with the blowing wind.

I have a modest proposal.

***

We don't have to use daily stock prices to measure the wealth of someone. We should use the trailing 6-month average prices to smooth out the volatile component of their wealth. (3-month, 6-month, or 12-month, depending on how volatile these prices really are).

 

Practical issues in observational studies

When I read Covid-19 medical research during the pandemic, one recurring practice bugs me a lot. Researchers seem to believe that a simple adjustment term inserted into a regression model eliminates bias in analyzing observational studies. To wit, if I suspect that age is a confounder, all I have to do is to insert an age term into my regression equation, and somehow that magically resolves the age bias fully.

That's just odd. Such a simple adjustment scheme works for simple biases. Specificially, one must get the age strata defined correctly so that within each age stratum, the treatment status is as if random (conditional on any other confounders used). In most real-world scenarios, this conditional independence assumption is hard to believe.

The age bias example is the tip of the iceberg. Even more bewildering is calendar time adjustment. This is used as a cure-all for all time-varying confounders. If one just inserts a calendar-time term into the regression model, all time-varying biases disappear! Shhh, don't even ask how they determine the appropriate time strata (days, weeks, months, etc.). [Whispering: you don't say the passage of time does not cause confounding.]

Then, I came across Andrew's recent paper - coauthored with Lauren Kennedy - called "causal quartets" (link to PDF), which nicely lays out some of the problems I've tossed around in my head.

Let me summarize the key points of this paper.

***

The methodology being discussed is "a causal model using linear regressions without interactions". This is the dominant type of causal models used in "real-world" studies of Covid-19 vaccines (should nicely generalize to logistic regressions, Cox regressions, etc.). The researchers identify a set of potential confounders - almost always, these are convenient variables that are pulled out of large-scale government databases, such as age, gender, prior hospital visits, etc. - then throw these confounders as main effects in a regression model (typically, a Cox regression). Then, the practice is to take "the coefficient of the treatment variable" as the "causal effect" of the treatment. As Andrew and Lauren stated, this causal effect is really an "average" treatment effect.

It's an average effect in the sense that the treatment under study has an effect for each individual, which may vary across individuals, and the single number that summarizes this distribution is the average of individual effects.

The existence of an average effect does not imply that the effect is identical to all individuals. Indeed, the causal quartet presents different distributions of individual effects that lead to the same average effect.

[Readers will be reminded of the so-called Anscombe's quartets, a favorite of statistics professors to warn against running "trendlines" on any and all datasets.]

Frequently, individuals can be organized into groups (e.g. age groups). Each age group has an average treatment effect among its individuals, and the average treatment effect is an average of the group averages. The value of the average clearly then depends on age group distribution within the population of individuals.

For more, see the Prologue of Numbersense (link), where I discuss how this plays into the Simpson's paradox.

***

To nail down the ideas, let's look at a simplest case in which age is the only confounder. The age-adjusted linear regression model is:

Outcome (given Treatment and AgeGroup) = a + b1*Treatment + b2(j)*AgeGroup(j)

where AgeGroup is one of j=3 ("young", "mid", "old"), and "Not Treated" and "old" are considered the reference levels, thus b2("old") = 0 and b1 is set to 0 for Not Treated.

Under this model, we have the following calculations:

Outcome (given Treated, "old") = a + b1

Outcome (given Not Treated, "old") = a

Treatment Effect (given "old") = a + b1 - a = b1

Similarly,

Outcome (given Treated, "young") = a + b1 + b2("young")

Outcome (given Not Treated, "young") = a + b2("young")

Treatment Effect (given "young") = a + b1 + b2("young") - a - b2("young") = b1

In sum, for all age groups, the model gives the same treatment effect of b1. Thus, b1 is the average treatment effect for the entire population. In this case, there is no difference between weighted and unweighted average; the age distribution does not matter.

The madness begins when people take the above and conclude that "the treatment effect is the same for all age groups".

This gets things backwards. This conclusion is the starting point, not the ending point, of the regression model; it is a necessary consequence of its structure.

Let's look at the regression equation again:

Outcome (given Treatment and AgeGroup) = a + b1*Treatment + b2(j)*AgeGroup(j)

Three terms together constitute the outcome. The first term ("a") applies to every individual regardless of treatment status or age. Its value is set to be the outcome of the Not Treated in the "old" group (not the only possibility but what I'm using in this discussion). Think of selecting the Not Treated, "old" group as the reference group, and measuring other groups against it.

Then, for each Treated person, regardless of age, we add the second term "b1".

Finally, for each person in AgeGroup j, we add the third term "b2(j)", except the "old" group (for which we add zero). 

Now, let's consider an alternative hypothesis: that the treatment effect decreases by age group; the older the individual, the lower the effect. Under the above regression model, however, the treatment effect is always b1 regardless of age. Therefore, it is impossible to represent the alt hypothesis with this model!

We need b1 to vary by age group. What we need is an "interaction" or "cross" term in the regression model, something like this:

Outcome (given Treatment and AgeGroup) = a + b1*Treatment + b2(j)AgeGroup(j) + b3(j)*Treatment*AgeGroup(j)

where, for simplicity of presentation, I set b3(j) = 0 for Not Treated regardless of Age Group so that I don't have to make b3(t, j)

Under this model, we can compute:

Outcome (given Treated, "old") = a + b1 + b3(Treated, "old")

Outcome (given Not Treated, "old") = a

Treatment Effect (given "old") = b1 + b3(Treated, "old")

Similarly,

Outcome (given Treated, "young") = a + b1 + b2("young") + b3(Treated, "young")

Outcome (given Not Treated, "young") = a + b2("young")

Treatment Effect (given "young") = b1 + b3(Treated, "young")

Now, the treatment effect consists of two parts, b1 which is independent of age, and b3 which varies with age group, and so in total, the treatment effect varies with age group.

***

By using the first regression model, the modeler makes the assumption that the treatment effect is the same for all age groups. If the modeler concludes, after fitting that model, that the treatment effect is the same for all age groups, s/he has set up a tautology, and has proven nothing.

Unfortunately, that's the kind of thing that is common in Covid-19 observational studies. They don't even publish the regression equations, coefficients, measures of fit, etc. They should have also tried the model with interactions, and explain why that is not preferred.

In addition, it appears that most of these same studies take b1 from the first model, and call it the treatment effect. SInce this first model assumes that the effect is constant for all subgroups, we can conveniently ignore the relative sizes of these subgroups.

If we have to compute the average treatment effect under the second model (the one with interactions), it would be:

Average treatment effect = Proportion "young" * (b1 + b3(Treated, "young")) + Proportion "mid" * (b1 + b3(Treated, "mid")) + Proportion "old" * (b1 + b3(Treated, "old")) = b1 + weighted average of b3(Treated, j)

The only scenario when we can ignore the age distribution is when the treatment effect does not depend on age. In this case, the weighted average equals the unweighted average so that the second model "converges" to the first model. As already noted, that is an assumption of the researchers, not a conclusion of the fitted model.

If the true treatment effect varies by age, then the form of the first regression model is inappropriate, and the average treatment effect estimated by this model is not the required weighted average effect. That's one of the key points of the Andrew and Lauren paper.

(Scroll to the end of the post to see a visualization of the two models.)

***

Andrew and Lauren suggest that the practice of using regression models without interactions is due to lack of data. It takes "16 times the sample size" to estimate one interaction effect relative to the main effect.

I get it when the observational data came from the professor's undergraduate class of 100 students but in Covid-19 studies, they have easily hundreds of thousands, if not millions, of subjects.

Nevertheless, the point of the causal quartets paper is to advocate including varying treatment effects in modeling observational data. In Section 3.1, they put some numbers behind the concepts I covered above. The study had a treatment effect of 25% higher survival rate (55% survived in the treatment group vs 30% in the control group).

If the regression model does not contain interactions, and the treatment effect (b1) from that model is interpreted as the average treatment effect, one expects each individual who takes the drug should have a 25% higher survival rate. That is only one possibility. Another possibility is that only 25% of the patients benefit from the drug, and for these, the chance of longer survival is 100%. The average treatment effect is 25%*100% + 0%*0% = 25%. Gelman and Lauren then concludes that "once we accept the idea that the drug works on some people and not others,..., we realize that 'the treatment effect' of any given study will depend entirely on the patient mix." Yep, bingo, 100%, +1.

The second possibility is more plausible than the first, and it is excluded by the structure of the regression model without interactions.

In later sections, Gelman and Lauren explore the implication of the fact that the intervention in most observational studies has no impact on a good portion of the subjects. In particular, they note that even if there is a huge impact on a small proportion of the subjects, the average treatment effect may still be small because of averaging. This calls into questions studies that (a) show huge average treatment effects and (b) for which most treated subjects would have no benefit.

Next, they point out that if we knew who might benefit from treatment, we'd have a better measurement if we restricted the test population to that subgroup. This effectively removes the majority of the subjects expected not to benefit, and is widely practiced in all Covid-19 studies (long lists of exclusions). All well and good... until the researchers then pretend that they didn't exclude anyone, and apply the treatment effect from the restricted study to the entire population. Instead, "when generalizing to a larger population, some modeling is necessary conditional on any information used in patient selection." (Gelman & Lauren) Unfortunately, I have not seen this done even once in any of the Covid-19 studies I read.

***
Here is an illustration of the differences between the two regression models discussed above, for those visually inclined:

The second model allows the three green dots for Treated to be placed anywhere: the profile of the Treated line (green) along the age dimension can be anything, does not have to be linear or parallel to the Not Treated line (blue). In this illustration, I arbitrarily decided that the treatment effect is below average for "mid" and "old" age groups; that it is the same for "mid" as for "old"; and that the treatment effect for "young" is higher. By assumption for simplicity, the blue lines (Not Treated) are equivalent on both panels but in general, two of the three blue dots on the right panel can also vary.

P.S. [3-1-23] Fixed the title of the right panel chart.

 

When you don't need data

I bet you can't believe I'm going to say I don't need/want data on a blog about intelligent thinking about data.

I'm going to argue that intelligent thinking about data includes recognizing when you don't need data - and by extension, you don't need more data.

This post is motivated by opinions that are circulating about how certain public health measures against infectious diseases such as masks and lockdowns are 100% useless - although if you start thinking about other domains, you can find pertinent examples.

Let's start with lockdowns first as that's more black and white. I don't need any data to believe that lockdowns reduce infections. I'd need data if I want to establish the magnitude of protection but for the direction, I don't need data. Why? Because I have theory.

What theory? Infectious diseases spread by infections, which requires contacts. Turning this around, one can say no infection can occur unless there is contact. During the early part of the pandemic, and especially before vaccines were available, in key hotspots, schools and businesses were closed; people were asked to stay home; employees were allowed to work from home. As a result, the frequency of contacts each of us had with others was drastically reduced, and for some, reduced to near zero. By the theory, the dramatic reduction of contacts resulted in a reduction of infections.

Do I need data to prove that statement? No. [The word "theory" is weird. By theory, I mean an immutable phenomenon, such as incorporating the law of gravity in a physical model. There's probably a better word for this. Structural model?]  

***

Would I like to have data? Sure. But it's not a must-have. If I had data, I hope to answer more questions, like how much reduction? does it affect all demographic segments equally?

Would I not want to have data? There is even an argument for this (although this sentiment won't be universally shared). I suspect strongly that any data that could be made available to me would be almost impossible to interpret - because we cannot run randomized experiments on lockdowns. Thus, it would take a lot of effort to clean the data, and to adjust the data, and none of this can be accomplished without making lots of subjective assumptions, sure to enrage some and delight others.

For example, all such data confound the effects of lockdowns and vaccinations because most places that had lockdowns also pushed vaccines simultaneously. What's more, masking, social distancing, and other measures were also simultaneously put to work. So, if someone had the data, it is likely to confuse rather than illuminate. Let's not forget about enforcement of lockdowns, and compliance to lockdowns. Contacts might not have been reduced if the lockdown was in name only!

So if I had data, it could show infections going up, down or sideways, and the effect of lockdowns would be masked by all the other different factors. More data could be useless.

***

More data, however, could not overturn my theory... otherwise, my theory is just conjecture. If the data happened to show that, after adjusting for all kinds of other factors, lockdowns have zero or negative impact on infections, I'd be much more likely to reject some of the subjective assumptions or the indirect data proxies, rather than to question the theory, which in this case, means I have discovered evidence that infections occur without contacts.

Even if surveillance data show that compliance to lockdowns was nonexistent, the proper way to interpret the data is to say that lockdowns by themselves should reduce infections but its effect is masked by compliance or lack thereof, so that data that confound both factors show that infections did not drop. Embedded in there is still the theory that reduced contacts reduces infections.

The theory - the structural model - is an immutable part of the larger statistical model. We expect data to conform to this structure. We don't use data to modify the theory. (This structural model is not the same thing as a prior in Bayesian models. Bayesian priors are subject to updating on observing data so it does not represent anything immutable.)

***
There are other examples in other domains as well. Here is Andrew's model of golf putting (link). The section called "modeling from first principles" is an example of a statistical model that embeds a (geometric) theory. He also later uses the term "geometry-based" model.

The simple geometry is not enough, and further iterations of the model add other factors but the original geomtry is still in there.

***
This post records my first thoughts on this topic. I'll get to masks next.

Strategies for validating observational studies 2/2

This post continues the previous post about strategies for validating observational studies. Both posts are inspired by my reading of the paper that claims to have discovered that people who do not take the Covid-19 vaccines are more prone to serious traffic accidents.

***

Another tactic used by the researchers to validate their findings is using a different analytical methodology to arrive at the same conclusion.

In this case, they used propensity score matching to create synthetic comparison groups. In matching studies, we accept that the vaccinated and unvaccinated groups are not directly comparable; we match pairs of people, one from each group, based on known matching variables to make sure that the matched pairs themselves are comparable.

The paper does not provide many details on how these researchers did the matching. The matching variables are the same variables used in adjusting the regression models, i.e. the demographics and prior diagnoses. These matching variables are summarized into a single number known as the "propensity score". Matched pairs have the same propensity scores (this description is slightly imprecise because propensity scores are probabilities and cannot be matched exactly. They would have matched based on ranges of values - known as calipers - but none of these details can be found even in the Supplementary Appendix.)

Nonetheless, matching cannot cure imbalance due to unobserved variables, for which there are many. As noted in the previous post, many obviously influential factors - such as exposure to driving, and driving skills - are not measured and therefore not used in the matching analysis.

Matched studies are hard to generalize. The research team reported two attempts at matching (we don't ever know how many they tried, just how many they reported trying). In the first attempt, they kept essentially the entire unvaccinated group (about 20% of the study population). Since they matched one vaccinated person to each unvaccinated, this means three out of four vaccinated persons were excluded from the study. Exclusion does not necessarily mean they are unmatchable; for each unvaccinated person, there could be multiple candidates in the vaccinated pool who match on the propensity score, all but one of whom would be thrown away.

If matching has cured most of the selection bias, the comparison between the matched pairs would be valid. But for whom? The treatment effect is estimated for people who look like those who were matched - in the first attempted matching, this means people who look like the unvaccinated.

Nevertheless, one can easily catch researchers who speak as if they can apply their finding to everyone! If such generalization is admissible, then what's the point of the matching?

The second matching attempt is more "stringent". This means the matched pairs are more similar to each other. What does it also mean? If you're following my drift, more stringent matching means the match rate is even lower, which in turns means the finding applies to an even smaller subset of the study population. As often in statistics, there is no free lunch. 

This stringent matching resulted in about 600K in each group. That means 70% of the unvaccinated, and 94% of the vaccinated persons were excluded from the matching analysis. Not useful if you ask me.

***

The problem with using propensity score matching to validate the adjusted regression model is that each method works with a different analysis population, and so the estimates are not directly comparable.