A beautiful curve and its deadly misinterpretation

When the preliminary analyses of their Phase 3 trials came out , vaccine developers pleased their audience of scientists with the following data graphic:

The above was lifted out of the FDA briefing document for the Pfizer / Biontech vaccine.

Some commentators have honed in on the blue line for the vaccinated arm of the Pfizer trial.

Since the vertical axis shows cumulative number of cases, it is noted that the vaccine reached peak efficacy after 14 days following the first dose. The second dose was administered around Day 21. At this point, the vaccine curve appeared almost flat. Thus, these commentators argued, we should make a big bet on the first dose.

***

The chart is indeed very beautiful. It's rare to see such a huge gap between the test group and the control group. Notice that I just described the gap between test and control. That's what a statistician is looking at in that chart - not the blue line, but the gap between the red and blue lines.

Imagine: if the curve for the placebo group looked the same as that for the vaccinated group, then the chart would lose all its luster. Screams of victory would be replaced by tears of sadness.

Here I bring back both lines, and you should focus on the gaps between the lines:

Does the action stop around day 14? The answer is a resounding No! In fact, the red line keeps rising so over time, the vaccine's efficacy improves (since VE is a ratio between the two groups).

The following shows the vaccine efficacy curve:

Right before the second dose, VE is just below 50%. VE keeps rising and reaches 70% by day 50, which is about a month after the second dose.

If the FDA briefing document has shown the VE curve, instead of the cumulative-cases curve, few would argue that you don't need the second dose!

***

What went wrong here? How come the beautiful chart may turn out to be lethal? (See this post on my book blog for reasons why I think foregoing or delaying the second dose will exacerbate the pandemic.)

It's a bit of bait and switch. The original chart plots cumulative case counts, separately for each treatment group. Cumulative case counts are inputs to computing vaccine efficacy. It is true that as the blue line for the vaccine flattens, VE would likely rise. But the case count for the vaccine group is an imperfect proxy for VE. As I showed above, the VE continues to gain strength long after the vaccine case count has levelled.

The important lesson for data visualization designers is: plot the metric that matters to decision-makers; avoid imperfect proxies.

 

P.S. [1/19/2021: For those who wants to get behind the math of all this, the following several posts on my book blog will help.

One-dose Pfizer is not happening, and here's why

The case for one-dose vaccines is lacking key details

One-dose vaccine strategy elevates PR over science

]

[1/21/2021: The Guardian chimes in with "Single Covid vaccine dose in Israel 'less effective than we thought'" (link). "In remarks reported by Army Radio, Nachman Ash said a single dose appeared “less effective than we had thought”, and also lower than Pfizer had suggested." To their credit, Pfizer has never publicly recommended a one-dose treatment.]

[1/21/2021: For people in marketing or business, I wrote up a new post that expresses the one-dose vs two-dose problem in terms of optimizing an email drip campaign. It boils down to: do you accept that argument that you should get rid of your latter touches because the first email did all the work? Or do you want to run an experiment with just one email before you decide? You can read this on the book blog here.]

Comments

[Mike Liveright]

I'm confused. If I look at Figure 2, and the following figures,I think they could have been produced by a one-dose or two-dose situation since the treated people seem to be close to fully resistant by day 14 and there seems to be little observable curve bending after that time, e.g. after day 21, the second dose.

Since there was not, as far as I read it, a test of one and two-dose groups, this curve might be the effect that the first dose was sufficient or that the second dose would have led to a stronger or longer resistivity after 4 months.

It seems to me that "logically" the second dose might lead to a longer/stronger resistance and that since we do see some anti-immune reactions in the infected people, it is possible that the second dose also might lead to some bad, delayed side effects.

Regardless, I realize that: 1) I probably don't fully understand your logic, 2) In a while we will have some semi-random results as there may be people who do not take the second dose, 3) I'm guessing that the second dose will act as a booster and thus it is more probable and is better me and for the society to have longer, stronger resistance than that I may suffer an anti-immune side effect, So I'll probably be taking the second dose.

-------------------------------------- To Author ------------------------------------
1) As I have said I may have missed the logic of the article, so ignore, or re-write, what I have written.

2) I also don't think that Figure 2 strongly indicates a decrease rate at which the control group who have symptoms and get test positive after day 70, It may, but I'm not convinced, and I don't see a reason that the control group would start becoming a little more resistant after a few months.

3) Also thanks for the blog.

[Kaiser]

ML: Thanks for the comment. It's always a pleasure to see where the confusion might lie. The logic you described is the starting point of this post - it is indeed possible to look at that blue line and draw the conclusion that almost everything is achieved by day 14. The key to seeing why that interpretation doesn't work is exactly the reason why in a clinical trial, we run a placebo group. The placebo group establishes the baseline to which we can compare the blue line. If the red line landed on top of the blue line, then we'd say the vaccine was worthless - even though the blue line looked exactly the same! This shows that the blue line cannot be evaluated by itself - it must be evaluated relative to the red line; more precisely, we look at the gap between the two lines.

The so-called vaccine efficacy (VE) is a measure of the gap - it's a ratio rather than a difference. In this other post, I get into the details behind this.

[Brad Mohr]

I, too, feel like I'm missing something in your logic. Efficacy continues to increase over time only because the placebo-group case counts continue to rise. But, by definition, the 2nd dose has no effect on the placebo-group case counts.

I think that in applying your final point ("plot the metric that matters to decision-makers"), you need to be cognizant of which decision-makers are your target audience. If you're targeting the FDA or vaccine-company leaders, that metric is indeed efficacy. But if you're targeting presidents and prime ministers, that metric is almost certainly case count. After all, the purpose of vaccination is not to maximize VE, it is to minimize case count.

[Kaiser]

BM: I added some links to the bottom of this blog post. The first link in particular goes into more details about this. Let me know if you still have questions after reading that post.
Quick summary: If you think the blue line will remain the same without the second dose, then you are making the assumption that the second dose is useless, which is a tautology.

[Hilary]

I disagree with your analysis. Current effectiveness over time, rather than cumulative effectiveness, is the important measure. This could be plotted as the ratio between cases in vaccinated and placebo groups during each consecutive (say) 7 day period (doubtless some smoothing would be needed).

Looking at cumulative effectiveness over the initial 21 days is highly misleading because it combines data from days 0-7 (when, as expected, the vaccine has essentially no effect) with that from days 8 to 21 (during which effectiveness increases dramatically). The levelling off of the cumulative effectiveness curve from around 100 days or so also appears to be an artefact driven by non-pharmaceutical control measures (social distancing etc) causing stabilisation of the placebo group case rate.

The absence of change in the shape of the vaccination group curve following the booster dose at 21 days is highly reassuring. It indicates that immunity from a single dose is strong and endures over 112 days. There are good reasons to believe that a second dose significantly improves longer term durability of immunity, so it remains clinically essential. However there is nothing in this data to support the 21 day interval between prime and booster doses. This was the interval used in the phase 3 effectiveness trial but my understanding is that it was chosen as the SHORTEST possible interval at which a booster dose might be expected to have any effect, in order to accelerate the trial and licensing of the vaccine. It was not chosen as the optimal interval, and based on immunological responses to other vaccines it is quite likely that a longer interval improves the booster effect.

This does leave a difficult ethical conundrum - the licensed interval has to be the one used in the phase 3 trial. Usually there is a clear case for sticking to licensed use of medications. But in this case, trials were accelerated, and many experts think that modified use would possibly enhance vaccine effectiveness for individuals as well as enabling faster public health control of the pandemic. I live in the UK so I expect my second dose (of either the Pfizer or Astra-Zeneca vaccine, not sure which as I haven't yet had the first) at an interval of about 10-12 weeks. I'm glad of that.

[derek]

You keep saying the curve cannot be interpreted as indicating that a second shot is unnecessary, which is true. However, it also cannot be interpreted as indicating that a second shot is necessary. The information was simply never gathered: these two curves are for two shots versus two placebos.

There needed to be a third curve, representing a group that was assigned to be given a shot followed up by a placebo. This would have resolved the ambiguity.

[Kaiser]

Hilary: If these experts would lay out their assumptions, as you have started to do here, then we can have a debate. In my second link above, I put out the key additional assumptions necessary to make the case.

Derek: it's not a both-sides situation here. Everyone agrees a third curve is necessary - in fact, in the first supplementary link above, I drew in a third curve so people can see why drawing one requires making lots of assumptions. It's not two-sides because we do know what the effect of the two-dose regimen is from the trial but we do NOT know what the one-dose curve looks like. The bar for a clinical trial has never been whether the treatment is the best of all possible alternatives; it is to prove one alternative is definitively (in a statistical sense) better than the placebo (or standard of care).

Also, I'd like someone to argue the case that double the people at half protection will lead to lower spread than half the people at full protection. I think that is a reasonable point of debate. (It's useless to debate double the people at full protection versus half the people at full protection. If you believe that, I have a bridge to sell you.)