Vaccine researchers discard the start-at-zero rule

I struggled to decide on which blog to put this post. The reality is it bridges the graphical and analytical sides of me. But I ultimately placed it on the dataviz blog because that's where today's story starts.

Data visualization has few set-in-stone rules. If pressed for one, I'd likely cite the "start-at-zero" rule, which has featured regularly on Junk Charts (here, here, and here, for example). This rule only applies to a bar chart, where the heights (and thus, areas) of the bars should encode the data.

Here is a stacked column chart that earns boos from us:

I made it so I'm downvoting myself. What's wrong with this chart? The vertical axis starts at 42 instead of zero. I've cropped out exactly 42 units from each column. Therefore, the column areas are no longer proportional to the ratio of the data. Forty-two is 84% of the column A while it is 19% of column B. By shifting the x-axis, I've made column B dwarf column A. For comparison, I added a second chart that has the x-axis start at zero.

On the right side, Column B is 22 times the height of column A. On the left side, it is 4 times as high. Both are really the same chart, except one has its legs chopped off.

***

Now, let me reveal the data behind the above chart. It is a re-imagination of the famous cumulative case curve from the Pfizer vaccine trial.

I transferred the data to a stacked column chart. Each column block shows the incremental cases observed in a given week of the trial. All the blocks stacked together rise to the total number of cases observed by the time the interim analysis was presented to the FDA.

Observe that in the cumulative cases chart, the count starts at zero on Day 0 (first dose). This means the chart corresponds to the good stacked column chart, with the x-axis starting from zero on Day 0.

The Pfizer chart above is, however, disconnected from the oft-chanted 95% vaccine efficacy number. You can't find this number on there. Yes, everyone has been lying to you. In a previous post, I did the math, and if you trace the vaccine efficacy throughout the trial, you end up at about 80% toward the right, not 95%.

How can they conclude VE is 95% but show a chart that never reaches that level? The chart was created for a "secondary" analysis included in the report for completeness. The FDA and researchers have long ago decided, before the trials started enrolling people, that they don't care about the cumulative case curve starting on Day 0. The "primary" analysis counts cases starting 7 days after the second shot, which means Day 29.

The first week that concerns the FDA is Days 29-35 (for Pfizer's vaccine). The vaccine arm saw 41 cases in the first 28 days of the trial. In effect, the experts chop the knees off the column chart. When they talk about 95% VE, they are looking at the column chart with the axis starting at 42.

Yes, that deserves a boo.

***

It's actually even worse than that, if you could believe it.

The most commonly cited excuse for the knee-chop is that any vaccine is expected to be useless in the first X days (X being determined after the trial ends when they analyze the data). A recently published "real world" analysis of the situation in Israel contains a lengthy defense of this tactic, in which they state:

Strictly speaking, the vaccine effectiveness based on this risk ratio overestimates the overall vaccine effectiveness in our study because it does not include the early follow-up period during which the vaccine has no detectable effect (and thus during which the ratio is 1). [Appendix, Supplement 4]

Assuming VE = 0 prior to day X is equivalent to stipulating that the number of cases found in the vaccine arm is the same (within margin of error) as the number of cases in the placebo arm during the first X days.

That assumption is refuted by the Pfizer trial (and every other trial that has results so far.)

The Pfizer/Biontech vaccine was not useless during the first week. It's not 95% efficacious, more like 16%. In the second week, it improves to 33%, and so on. (See the VE curve I plotted above for the Pfizer trial.)

What happened was all the weeks before which the VE has not plateaued were dropped.

***

So I was simplifying the picture by chopping same-size blocks from both columns in the stacked column chart. Contrary to the no-effect assumption, the blocks at the bottom of each column are of different sizes. Much more was chopped from the placebo arm than from the vaccine arm.

You'd think that would unjustifiably favor the placebo. Not true! As almost all the cases on the vaccine arm were removed, the remaining cases on the placebo arm are now many multiples of those on the vaccine arm.

The following shows what the VE would have been reported if they had started counting cases from day X. The first chart counts all cases from first shot. The second chart removes the first two weeks of cases, corresponding to the analysis that other pharmas have done, namely, evaluate efficacy from 14 days after the first dose. The third chart removes even more cases, and represents what happens if the analysis is conducted from second dose. The fourth chart is the official Pfizer analysis, which began days after the second shot. Finally, the fifth chart shows analysis begining from 14 days after the second shot, the window selected by Moderna and Astrazeneca.

The premise that any vaccine is completely useless for a period after administration is refuted by the actual data. By starting analysis windows at some arbitrary time, the researchers make it unnecessarily difficult to compare trials. Selecting the time of analysis based on the results of a single trial is the kind of post-hoc analysis that statisticians have long warned leads to over-estimation. It's equivalent to making the vertical axis of a column chart start above zero in order to exaggerate the relative heights of the columns.

 

P.S. [3/1/2021] See comment below. I'm not suggesting vaccines are useless. They are still a miracle of science. I believe the desire to report a 90% VE number is counterproductive. I don't understand why a 70% or 80% effective vaccine is shameful. I really don't.

Comments

[Richard Krablin]

While your analysis is correct and most interesting, the important number is VE after two shots, which is 95%. That's the protection that i have received from the vaccination. What happened before 29 days was only of importance then, as the FDA has concluded. Of course, what the VE is as time goes on will be crucial, but the data are not yet available.

[Kaiser]

RK: Pfizer second shot is at Day 21. So in a per-protocol analysis, they should have started counting at Day 22 not Day 29. The case counts during Day 22-27 was 2 vs 18 and there's no way to argue that the vaccine was "useless" during that week. If one wants to do an intent-to-treat analysis, you'd got to count from Day 1 because the PP analysis removes anyone who got infected and did not receive the second shot. (This is the origin of the cumulative cases plot anyway.) Like you said, we don't really know the duration of protection. Further, at least half the cohort did not contribute to the right side of the curve so there is uncertainty there too.

[Charles]

This is such a bad take.

I think you're looking at this far too analytically and "data viz principles"-y. It's immediately obvious to me, at least, that any vaccine (or heck, most treatments) will have a ramp-up period in which the average treatment effect is far lower than the plateau.

This is also the reason why the arbitrary cutoff time is decided before the trial is run in the first place.

There's no way of doing this analysis fairly while allowing the vaccines to act if we use your methodology. Yes, the technical VE over the entire period is lower, but it also hasn't given the treatment time to do its magic.

Separately - I'm not sure why your blogs have gone in this direction but both this blog and your other one have been on a real bender against vaccine trials and against most ideas concerning the vaccines or covid, generally asking for ideal situations which aren't feasible or of any interest for people in positions of power. Is there a reason for this?

[Kaiser]

Charles: If you've been following the literature, the cutoff time has not been decided before the trial. The protocols outline a whole series of options, and in addition, each team conducting post-hoc analysis chooses their own case counting window. Most of the "findings" that get publicized are post-hoc analyses. My viewpoint is not something strange - it is an intent-to-treat analysis. That is how the Pfizer cumulative case count curve came into being in the first place. If there is a shift in my views, it's reacting to policymakers veering away from the science. As I have stated recently on the other blog, I believe the desire to get to a 90% VE number is counterproductive. I don't understand why having a 70% or 80% vaccine is so shameful. I really don't.

[Cris]

I think they should have started counting in December 2019, when the virus was first identified. That would be the only valid zero point for a vaccine trial. After all, all those people that got sick before the vaccine was developed can not just be ignored!

[Joshua]

The vaccine companies have all decided to optimize a single figure, the percent efficacy, with a target of over 90%. This is a MARKETING decision. And this variable was chosen for optimization because, as Kaiser has highlighted in this series of posts, it is one that is fairly trivial to game. Simply run the trial, pick the point at which the case count plateaus, and then develop a post hoc justification for rejecting all data before that time point. If gerrymandering is Politicians picking their voters, than what we are seeing here is scientists picking their data.

If the option to cherry pick the data were removed by the FDA forcing a Day 0 start point for all data analysis, then what Kaiser has illustrated above could still be turned to a marketing advantage. The data clearly shows that about a week after the first dose the vaccine starts to provide at least some protection. Total cases in the vaccine arm tops out below the day 0-14 count in the placebo arm. There may be differences between vaccines in the rate of protection this soon into the regimen, that one or the other could use to emphasize the benefit of their vaccine over the competition (which is what the manipulations to achieve 90, to 95% efficacy is really all about at the end of the day, since the FDA said 50% would be good enough).

[Kimberly Schult]

Can you do J&J too?

[Kaiser]

KS: Yes, I am working on it. Probably later this week.