I'm happy to announce that our paper on limitations of observational (i.e. "real-world") studies of Covid-19 vaccines has been published in the Journal of Evaluation in Clinical Practice (link). I'd like to thank my co-authors (Peter and Mark) and reviewers for feedback that greatly improved the presentation, and the journal editors for the courage to print it.
In the paper, we offer three examples in which standard methods used to analyze observational data produce vaccine effectiveness (VE) estimates that are wildly off the mark. Specifically, we show that a hypothetical vaccine with 0% efficacy will be rated as 50 to 70 percent effective. The magnitude of the biases surprised us, and should surprise you too. That said, we are not claiming that Covid-19 vaccines are ineffective - as I explain below, the device of a 0% effective vaccine allows us to work around the lack of data disclosure.
The motivation for writing this paper is to encourage better science. Observational studies are hard, and the quality of Covid-19 related observational studies has been mediocre.
Two major obstacles to writing this paper were (a) the pathetic level of data disclosure for major studies that have had tremendous impact on public health policies and (b) the lack of a ground truth - no one knows the real vaccine efficacy. We found a way to work around these challenges.
For (b), we analyzed the case of the 100% ineffective vaccine. This device is very useful because if the vaccine is ineffective, it is ineffective for all subgroups of people, at all times, etc. If we assumed any other level of vaccine efficacy, say 50%, we'd have had to make a huge list of other assumptions as well. For example, there is no rationale to assume that if VE is 50% in aggregate, then it is 50% for every age group.
For (a), we opportunistically found papers that disclosed just enough data to prove the existence of specific biases. For example, to study the effect of background infection rate bias, we require data on weekly infection rates for certain subpopulations within an analysis window, which we found in a Danish study but are not typically disclosed by researchers. We didn't find a single paper that disclosed all the data required to study all three sources of bias, thus for each bias, we used data from a different reference.
In each example, we show that if the ineffective vaccine were tested in a randomized clinical trial, the typical analysis applied to RCTs would conclude correctly that VE is 0%; however, if the ineffective vaccine were analyzed using methods that were popular in early 2021, the resulting VE would erroneousy land in the 50-70% range! The magnitude of the over-estimation surprised us.
The full paper is here. Spread the word!
Comments