This post continues the previous post about strategies for validating observational studies. Both posts are inspired by my reading of the paper that claims to have discovered that people who do not take the Covid-19 vaccines are more prone to serious traffic accidents.
***
Another tactic used by the researchers to validate their findings is using a different analytical methodology to arrive at the same conclusion.
In this case, they used propensity score matching to create synthetic comparison groups. In matching studies, we accept that the vaccinated and unvaccinated groups are not directly comparable; we match pairs of people, one from each group, based on known matching variables to make sure that the matched pairs themselves are comparable.
The paper does not provide many details on how these researchers did the matching. The matching variables are the same variables used in adjusting the regression models, i.e. the demographics and prior diagnoses. These matching variables are summarized into a single number known as the "propensity score". Matched pairs have the same propensity scores (this description is slightly imprecise because propensity scores are probabilities and cannot be matched exactly. They would have matched based on ranges of values - known as calipers - but none of these details can be found even in the Supplementary Appendix.)
Nonetheless, matching cannot cure imbalance due to unobserved variables, for which there are many. As noted in the previous post, many obviously influential factors - such as exposure to driving, and driving skills - are not measured and therefore not used in the matching analysis.
Matched studies are hard to generalize. The research team reported two attempts at matching (we don't ever know how many they tried, just how many they reported trying). In the first attempt, they kept essentially the entire unvaccinated group (about 20% of the study population). Since they matched one vaccinated person to each unvaccinated, this means three out of four vaccinated persons were excluded from the study. Exclusion does not necessarily mean they are unmatchable; for each unvaccinated person, there could be multiple candidates in the vaccinated pool who match on the propensity score, all but one of whom would be thrown away.
If matching has cured most of the selection bias, the comparison between the matched pairs would be valid. But for whom? The treatment effect is estimated for people who look like those who were matched - in the first attempted matching, this means people who look like the unvaccinated.
Nevertheless, one can easily catch researchers who speak as if they can apply their finding to everyone! If such generalization is admissible, then what's the point of the matching?
The second matching attempt is more "stringent". This means the matched pairs are more similar to each other. What does it also mean? If you're following my drift, more stringent matching means the match rate is even lower, which in turns means the finding applies to an even smaller subset of the study population. As often in statistics, there is no free lunch.
This stringent matching resulted in about 600K in each group. That means 70% of the unvaccinated, and 94% of the vaccinated persons were excluded from the matching analysis. Not useful if you ask me.
***
The problem with using propensity score matching to validate the adjusted regression model is that each method works with a different analysis population, and so the estimates are not directly comparable.
Comments