Curve watching during the pandemic

The pandemic has revealed the importance of a most exciting area of statistics - causal inference from observational data. We've been inundated with all kinds of reports about the effect of [vaccines, masking, social distancing, school closures, outdoor dinings...] on the pandemic. Most of these "studies" focus on either one factor or one region or one demographic segment. Then, the headlines and PR coming out of these studies apply the specific results to the general public. Vaccines work! Masks don't work! Closing schools doesn't make sense!

The quality of the causal inference is tested when we apply the findings of these studies to other settings. So in today's post, I explore the claim that vaccination campaigns are responsible for the decline in cases seen in many countries.

In order to shut down our biases, I am presenting the graphs first without country labels. In each chart, the time period is from November 2020, before any vaccinations, to April 2021. I used data from OurWorldinData.org. The red line shows the new cases per million while the blue line shows the proportion of the population who have been vaccinated (defined as those who received at least one shot if it's a two-shot vaccine).

If we only look at individual countries, here are the stories one can tell:

Panel A presents a clean picture. Cases have tumbled from a peak in mid January to a level below that of November by end of April. The decline begun soon after initiating the vaccination campaign, which has reached about half the population. So vaccines worked!

In Panel B, this country's highly successful vaccination campaign has covered over 60 percent of the population by end of April. That appeared to be a critical mass, at which point cases dropped below the level of November. One can build a story about herd immunity.

The herd immunity story is shaky though. The 60-percent threshold does not apply to Panel A. Using the same logic, one would argue that the country reached critical mass even at 30% vaccinated. But at 30% vaccinated, the country in Panel B was approaching peak cases!

In Panel C, the sharp drop in cases happened as the proportion vaccinated rose from 20 to 40 percent while cases grew during the first two months of the campaign.

In Panel D, cases plunged within weeks of the start of the vaccination campaign, and reached a low plateau with fewer than 10% of the population vaccinated.

Then, you have Panels E, F and G. If one only analyzes those countries, then one comes to the opposite conclusion. In these places, the cases apparently have not been slowed by vaccinations. Over 60% of the people in Panel E have taken a shot but no sharp decline is evident yet. For Panel F, there have been a spurt in cases in April with the vaccination proportion nears 60%.

The picture is very confusing. That's because the vaccine is not the only factor affecting the rise and fall of cases. Any model will fit some of the data some of the time; a good causal model must work across multiple countries.

Now, I reveal the countries. These are the top 11 countries ranked by the proportion vaccinated by end of April. I excluded countries with low overall Covid-19 incidence. Seven of these countries have experienced a nice drop in cases per million.

***

It's also a good idea to take a peek at the other end of the spectrum. What's going on in countries that have barely begun their vaccination campaigns.

Here are 12 countries with very low vaccination rates but with case incidence in a similar range to the 11 countries shown above.

None of these countries have seen spikes in cases, almost all of them are seeing downward trends in April, many of them exhibit up-and-down cycles over these months, and a few have marked plunges such as Georgia in December and Montenegro since March.

***

Simple one-factor models aren't going to work to explain trends across countries and time. A good causal model should include a baseline trendline, a vaccine factor, plus lockdowns and other mitigation measures.

Another challenge for data analysts: the data post-vaccination will be hard to compare to past data. There will be fewer Covid tests among the fully vaccinated. Should cause of death be assigned differently if the patient is known to have been vaccinated? Will we see more deaths from co-morbidities? These are open questions - when I learn more, I'll report back.

 

 

Comments

[Michael Droy]

Good stuff.
It would seem natural to assume that there is a seasonal effect and/or an inherent rush to peak and collapse curve that would explain much of the drop off without reference to vaccines.
Also many countries now seem to be suffering extended long waves over winter 20/21, while others has a much shorter wave. This seems to be linked with having a smaller winter/spring 2020 wave (which can only reasonably be interpolated from deaths as infections were not measured well then).

And at heart super-spreader effects make any modelling complicated.

[Kaiser]

MD: The effects may also differ by country or country characteristics. So the vaccine effect might be higher in some places and lower in others. Same with lockdowns e.g. depending on compliance and other factors.