Back in March, I reviewed several early "real-world studies" of vaccine effectiveness (here, here, here, here), including the Dagan et. al. study from Israel that eventually became one of the most frequently cited studies on the unreasonable effectiveness of mRNA vaccines.
The researchers of these early studies acknowledged the difference between observational datasets, and data coming from randomized clinical trials; specifically, the problem of selection bias inherent in the former but obviated by the design of the latter. Matching methods were deployed to reduce the effect of selection before generating a vaccine effectiveness number that mirrors vaccine efficacy in an RCT.
I also later reviewed several studies (Denmark, CDC, CDC 2) that utilized regression adjustments. Regression adjustment is an exercise in wishful thinking: for example, an age-adjusted estimate of vaccine effectiveness is the value of VE after removing the effect of age. Mathematically, the regression adjustment imposes a common age distribution on the vaccinated and unvaccinated subgroups, so age becomes a non-factor.
However, the age-adjusted regression estimate is relevant only if
(a) the estimate of VE for any given age range is reliable, and
(b) the specific age distribution imposed on both subgroups is realistic.
In the present situation, it is hard to believe either of those assumptions. For vaccination campaigns everywhere prioritize high-risk subgroups, particularly older people with or without comorbidities.
The headlines coming out of those early real-world studies were loud and clear: real-world evidence confirmed that the vaccines performed as well as and maybe even better than the clinical trials indicated. The media highlighted VE estimates of over 90% on all reported cases of Covid-19.
As a reminder, this was a CNN headline from February 2021.
The first sentence of the article cites the 90% number: "Pfizer-BioNTech's Covid-19 vaccine appears to reduce symptomatic coronavirus infections by more than 90% in the real world, Israeli researchers said Sunday."
If you want the other view, Fox News said the same thing.
***
By end of May, amidst reports of breakthrough infections, the story started to evolve. The new headlines now focused on VE estimates of close to 100% on reported severe cases, hospitalizations, or deaths. Readers were admonished for ever believing that vaccines should prevent disease, even though the primary endpoints for all clinical trials and early real-world studies were prevention of symptomatic, PCR-positive Covid-19 cases in the selected case-counting window.
Since June, the U.K., despite having one of the world's fastest vaccination programmes, began to suffer another surge in cases. The new headlines now drew attention to hospitalizations rather than cases.
The report cites a Public Health England study, saying "although the Delta variant reduces the effectiveness of vaccines against symptomatic infection, two doses of COVID-19 vaccine still protect against severe disease."
Israel, the early star in vaccinations, faced a similar scenario, and their reporters followed the same path, delivering headlines such as this:
It's not about infections anymore; only hospitalizations matter.
It's not just the headlines that changed. This new batch of studies employs different methods of analysis. Matching and regression adjustments were both sent to the scrap pile. The new analyses now wantonly ignore selection and other kinds of biases, and treat the observational data as if they were collected from randomized clinical trials. They assume the vaccinated and unvaccinated groups are balanced on all variables, an assumption not remotely plausible.
Here is an example of such a calculation, lifted from Matan Holzer's twitter account (see title of bottom chart):
In these type of studies, vaccine "efficiency" for hospitalizations is based on the ratio of the odds of hospitalizations for vaccinated vs unvaccinated to the odds of vaccination. The same ratio is obtained by dividing the hospitalization rate of the vaccinated by the hospitalization rate of the unvaccinated. This is the analysis that would have been performed to evaluate VE in a clinical trial - although those trials did not contain enough participants to attain statistical significance on the hospitalization metric.
Such analyses gave rise to headlines that proclaim that only unvaccinated people are sick enough to go to hospitals.
What are the data supporting the above headline (link)? It's the share of hospitalizations that are attributed to the unvaccinated people.
In the meantime, the following chart from BBC shows how the new wave of infections is growing even faster than in the last surge, which happened before vaccinations.
***
By mid July, hospitals in the U.K. and Israel are admitting a rising number of fully vaccinated patients. So, the story is shifting again.
The previous metric comparing vaccinated and unvaccinated groups as if they were randomly assigned is abandoned. Experts now pretend that they did not introduce this analysis to the public, and criticize those looking at this number as a probability illiterate - failing to grasp Bayes' Rule. (here) Nate Silver chimed in:
I agree that the efficiency metric is poorly conceived, a fact that was as obvious in May as it is in July. I don't like the "story-first" mentality adopted by many scientists, which led to all those headlines about almost no vaccinated people getting hospitalized.
As the hospitalization story falls apart, the media is lining up behind the VE metric on deaths. We are being told that the vaccines have broken the link between infections and deaths, which reminds me of when they said the vaccines severed the link between infections and hospitalizations.
The math has been clear on this from the start: hospitalizations lag cases, and deaths lag both. We are going to find out in the coming weeks whether this latest narrative sticks
We need a post-mortem on the real-world studies, a honest evaluation on why these findings have low predictive value.
P.S. [7/23/2021] The same story is unfolding in Israel. Weeks ago, they happily cited naive estimates of vaccine effectiveness - first on infections, later on severe disease. Now that those metrics don't show what they used to show, suddenly those metrics are worthless. This Times of Israel article is highly informative.
Comments