In the previous post, I started to review the newly released study about the "6 month" efficacy and safety of the Pfizer-Biontech vaccine.
This post examines the central efficacy claim of the paper. Its interpretation turns on a few words.
The paper starts with the finding that "VE against COVID-19 was 91% ... through up to 6 months of follow-up among evaluable participants..." and ends with the claim that "the data in this report demonstrate that BNT162b2 [Ed: the name of the Pfizer vaccine] prevents COVID-19 effectively for up to 6 months post dose 2..."
On the surface, this sounds like a straightforward extension of the interim analysis published in December 2020 that led to the emergency use approval of the Pfizer-Biontech vaccine. That study generated the now-famous VE of 96%. At the time of the study, roughly (but not quite) 50% of the participants have been tracked for two months after their second doses. Fast forward four months. Using exactly the same methodology, pre-specified in the trial protocol, the scientists updated the VE number "up to 6 months of follow-up", resulting in a revised VE of 91%, so there have been negligible attenuation of the vaccine's power.
That's not quite 2what happened, though. Strictly speaking, the quoted statements describe the collected data. However, the meaning of these statements turn on a few words. I'll first focus on why they said "up to 6 months" rather than "6 months".
(In a related post, I explain why when I read journal papers, I don't make assumptions of what the researchers may or may not have done. I focus on what they commit to paper.)
***
This headline result is on line 4 of the following table:
The famous VE of 96% was based on a case counting window that runs from 7 days after second dose (2D+7) to mid-November, when just under half of the study population have reached two months post second dose. Recall the train metaphor from the last post: the participants are seated in different carriages based on the dates of enrollment (embarkment).
So what's new in this study? The head of the train has moved from 2 months to 6 months of follow-up while the rear of the train also moved 4 months forward so almost all participants should have reached the 4-month mark, meaning their case history contributes to the case rate from 2D+7 up to 2D+4 months.
Given the focus on the "6 month" point by the investigators, I was expecting to find the words six months in this table. They are not there. The closest thing we can see is >= 4 months, which is the last line of the above table.
The reason why six months can't be found is that a proper six-month efficacy analysis has become impossible when the FDA allowed the vaccine developers to offer the vaccine to everyone in the placebo group. In the paper, this is known as the "vaccine transition option". Between December 2020 and March 2021, all but ~2,000 participants jumped off the train... before they arrived at the six-month marker. Once they are off the train, there is no re-boarding so no further updates on "long term efficacy" is possible.
***
The paper glosses over how the vaccine transition option totally changed the game. I'll now fill in the details. The following chart shows how the transition option caused follow-up times to be cut short across the board.
Without the option, as the Pfizer trial was originally designed, we should have 55 percent of the participants reaching 6 months or longer after second dose by March 2021, with all but two thousand odd people exceeding 4 months of follow-up. We should have had solid evidence on efficacy "up to 6 months".
With the transition option, the train was stopped, and almost all of the 44,000 passengers exited. What's more, the conductor instructed passengers to use exits at the back of the train as the doors near the front would not open. As a result, the proportion with at least 6 months of follow-up crashed from 55% to 7% while the number with fewer than 4 months of follow-up doubled. The vast majority of participants (about 65%) left the train with >=4 and <5 months of follow-up.
What the investigators meant by "up to 6 months" is really "four to seven months," with 80% of "evaluable participants" with fewer than 5 months of follow-up time. The last row in that VE table labeled ">= 4 months" is really better described as "4 to <5 months".
***
Those who decided to stay on the train for 6 months or longer are not your regular passengers - they consist solely of people who do not want to know if they were given the vaccine or the placebo, and so for some reason, they don't care if they are inoculated with a 96%-efficacious vaccine during a deepening pandemic. We don't know the psychology of those individuals but we are pretty sure they are not the average person.
This is another reason why the numbers on the right side of the case curve (>= 5 months) are neither reliable nor valid. If they show the error bars around those rates, they will be huge. If they conduct a survey, they will possibly learn the reasons why most exited the train but a few remained on it.
***
Elsewhere in the paper, the case-counting window was described as "from 4 months to the data cut-off".
If this were an ordinary clinical trial, these words suggest that the data were censored at the time of the interim analysis. Censoring is a technical term that signals the curtailment of the follow-up period. Common reasons for censoring include the participant reaching an endpoint (once someone gets infected, the outcome is known, removing the need to keep testing the person for infection), or the participant has dropped out or gone missing.
A different type of censoring is triggered by performing an interim analysis. At the full analysis, each participant has amassed the required follow-up time and every participant can be analyzed for the identical amount of follow-up. At interim analysis, the train is stopped at the half way point, so that the early enrollees have been observed for more time than the later enrollees. Every participant who has not previously been censored is now censored through an act of the investigators'.
Interim analyses have become common in clinical trials and so, censoring by "data cut-off" follows. But what happened in the Pfizer trial is not censoring by data cut-off. It is censoring by the vaccine transition option. Study participants censored themselves by exiting the train. This is a type of "informative censoring," which requires bias correction (not discussed in the paper).
March 13, 2021 was the data cut-off date for this study. In a standard interim analysis, censoring by data cut-off means anyone who hasn't been censored for some other reason will have their follow-up period stopped on March 13. When the censoring is due to the vaccine transition option, each individual has a different end date - the date on which they accepted the transition option. Almost everyone who accepted did so before March 13. Thousands exited in December 2020 alone.
The use of "4 months to data cut-off" creates the impression that the censoring occurs just like any interim analysis that stops follow-up on data cut-off. In reality, almost nobody was censored on data cut-off date; they were censored before data cut-off date, when they were told whether they received the vaccine or placebo.
***
In this post, I covered two examples of statistical statements that turn on a few words. "Up to 6 months" and "to data cut-off" don't mean what we might think they mean. That's because the vaccine transition option was not anticipated or pre-specified.
In summary, the six-month study is primarily about the 4th month after 2nd dose. There is some decline in efficacy but not large. The current estimate is less reliable than the previous analysis because the vaccine option curtailed follow-up time across all participants. Besides, we are unable to learn about efficacy beyond four months - that is the cost of giving the placebo participants the vaccine. I also have doubt that the people who stayed on board are representative of the average participant.
P.S. [8-9-2021] Here are my other posts about this study: post 1, post 3
Comments