We don't live in normal times and so the vaccine trials have been accelerated in order to deliver the promise of stemming the pandemic.

What does a normal clinical trial look like? In the most fundamental setting, you recruit participants, screen them for eligibility, randomly split them into two equal groups, give one group the medicine being tested, and the other group a placebo, wait a pre-determined follow-up time, and when the trial ends, you tally the outcomes in both groups and compare them.

In designing and running randomized controlled trials (RCTs), every effort is taken to make sure that the two groups of participants are identical and identically treated so that any difference in outcomes can be attributed solely to the medicine (vs placebo) and nothing else.

***

One way to accelerate trial results is to conduct an interim analysis. In the simplest setting, one conducts an analysis half way through the trial. This procedure assumes that the first half of the trial is predictive of the second half. It also assumes that the sequence of enrollment is random.

Think of participants as boarding a train from the front to the rear. During a normal follow-up period, the train moves a fixed distance forward. Passengers can only alight at a fixed disembark point on the platform, so at the end of the journey, the rear of the train has stopped there. In other words, every participant is tracked for the same amount of time, say one year.

For an interim analysis, the train is stopped midway to its destination. An inspector boards the train to check everyone's documents. The people at the front of the train has traversed a longer distance than the people at the back of the train. In statistical jargon, we say that the outcomes are "censored".

At interim analysis of the vaccine trials, someone may have just received their first shot a week before the date of analysis, and so the investigators have followed them for one week only - and they can only contribute to the case counts up to 1D+7 days (7 days after first dose).

If you look at a cumulative case curve like this at interim analysis, imagine the front of the train running from left to right from Day 0 to Day 238. The interim analysis stops the train. The passengers at the front of the train have moved a further distance to the right than those who boarded last. The number of participants who can contribute to the case counts decreases as you go along the time axis from left to right.

As I have pointed out before (link 1, link 2), most vaccine trials have some form of staged enrollment, usually recruiting lower-risk participants first, and so the assumption of random sequencing of enrollment fails. These are the types of biases that appear when one deviates from the standard design. Investigators should correct for this and other biases.

At final analysis, all passengers have exited the train at the same disembark point. That means all participants have experienced the entire timeline. So the bias correction is only necessary at interim analysis but not at final analysis.

A different adjustment - unrelated to bias - is that the bar for statistical significance is raised at interim analysis to account for additional uncertainty due to shorter follow-up. (Nevertheless, statistical significance has become collateral damage of the pandemic.)

***

That is a long setup for the topic of today's post, which is the "6-month" analysis of the Pfizer vaccine trial. A preprint of the scientific results has finally been published last week (here). The report is notable not just for what it says but what it doesn't say. My goal is to help fill out the details so you can digest the analysis with full context.

The first thing I noticed on the summary page of the report is that the investigators carefully inserted the words "up to", which includes the possibility that only a small proportion of people have reached 6 months of follow-up after the second dose.

Enrollment in the Pfizer trial took place between end of July and end of October of 2020, and this interim "6 month" analysis froze the data on March 13, 2021. At this point, the head of the train is at about 7.5 months out while the rear of the train is at 4 months so we know for sure not everyone has reached 6 months of follow-up.

A key sentence in the paper discloses that about 55% of the participants have reached 6 months following their second doses: "During blinded and open-label periods combined, 55% of BNT162b2 [Ed: name of the Pfizer vaccine] recipients had ≥6 months follow-up post-dose 2."

It's important to realize that the above sentence did not state simply that "55% of trial participants had ≥6 months follow-up post-dose 2." The wording makes all the difference!

They didn't say the simple sentence because it's not true. The statement would have applied under normal circumstances but we live in extraordinary times. Since December, when Pfizer obtained EUA for the vaccine, a steady stream of placebo participants have been given the vaccine - they have effectively disembarked from the train before reaching the destination (link). By Pfizer's own publicity (link), they aimed to offer all placebo participants the vaccine by March 1, two weeks before the analysis date.

As a result of the early-than-planned exits, only a fraction of the original placebo arm had ≥6 months follow-up post-dose 2. How many remained? How about 6%? This fact was disclosed in the adjoining sentence: "6% of placebo recipients had ≥6 months follow-up post-dose 2".

This is the data challenge that the analysts faced due to the decision to dismantle the placebo group. By design, 55% of the placebo group should have had 6 months of follow-up by March 13 but because of the "vaccine transition option", the number reaching that goalpost shrank from 55% to 6%... from about ~10,000 to ~1,000. To use the terminology from above, 9,000 or so persons have been censored at earlier dates so they have zero impact on the later part of the case curve. The later part of the curve is where we find "long-term efficacy".

For the interim analysis published in December, 2020, the FDA required a median follow-up period of two months following the second dose. Fast forward four months, and the middle of this train should have reached six months, triggering the six-month update. And yet, this is just theory that ignores the vaccine transition plan for the placebo group. In reality, the six-month mark is at the 6th percentile of follow-up times, not the 50th percentile. Nevertheless, passengers who left (and received the vaccine) are not allowed to re-board the train, and thus conducting the analysis later will make no difference.

Tellingly, in the VE table (Figure 2), the last line computes VE >= 4 months after dose 2. That's strange when the study is used to confirm "six month safety and efficacy."

***

As an analyst, the crisis you're facing is that the vaccine transition option effectively destroys the careful RCT design, turning the data into observational ("real-world") data. This means the key to success is bias identification and correction.

Unfortunately, that's not the path taken. The paper makes no mention of how many placebo participants chose to receive the vaccine, and when they did so. Timing matters a lot. It does not provide background statistics comparing those who transitioned, and those who didn't, nor any differences between those who transitioned earlier relative to those who transitioned later.

What the investigators did instead is to devise a mechanism by which they subtracted people from the vaccine arm to match the censoring of the placebo arm. The paper spent almost no time on analytical methods. We do have this sentence as a clue: "During the blinded period, 51% of participants in each group had 4 to <6 months of follow-up post dose 2; 8% of BNT162b2 recipients and 6% of placebo recipients had ≥6 months follow-up post-dose 2."

For placebo participants, the blinded period is before they were told they got saline shots during the trial and given the vaccine. The people in the vaccine arm were apparently also sent the notices (this wasn't described in the paper), and most of them responded to learn they were vaccinated.

The investigators then censored everyone in the vaccine arm who responded, meaning they de-boarded the train. That is the reason why the proportion of people in the vaccine arm with over 6 months follow up also plunged - from 55% to 8%.

The massive exodus from both arms of the trial appeared numerically similar but the reasons for exiting were night and day. Pfizer said it would be unethical to hold out their 95% efficacious vaccine from the placebo participants for the sake of science. In order to give them the vaccine, they must first be unblinded, i.e. told that they were assigned the placebo shots before. This reason does not apply to the vaccine-arm people as they have already taken both shots.

With the unblinding of the vaccine arm, the researchers threw out a lot of fine data. Instead of estimating the 6-month case rate of the vaccine arm using ~10,000 people, they did so with 10 times fewer participants. Students learn in Stat 101 that sampling error is a function of the square root of the sample size: to halve the error, you need 4 times the sample size. Reverse this. If you reduce the sample to a quarter of its size, you double the error. If you reduce it 9 times, you triple the error. That's one way to quantify the price of the vaccine transition plan. (The dismantling of the placebo arm is inevitable but it is the investigators' choice to inflict it also on the vaccine arm.)

I'd have taken a different path. The VE is based on a ratio of the case rates of the vaccine arm and the placebo arm. By preserving the original sample on the vaccine arm, I would have one good rate estimate (vaccine arm) and one bad estimate (placebo arm). In their analysis, they used two bad estimates.

***

The challenges of observational data do not end there. It's not just about the size of the sample. It's also about the quality.

The participants who stayed on the train are unlikely to be a random sample of the original population. What would you choose to do if you were given a chance to confirm that you have been given the vaccine, and if not, to receive a vaccine that attained a 95% efficacy in the middle of a worsening pandemic? I can only guess at why a small number of participants decided not to unblind themselves. Maybe they already figured out their status. Maybe they think they don't need a vaccination, perhaps because they have lower risk, or they don't care if they are vaccinated or not. No matter what, these are the people who populate the right side of those cumulative case curves which speak to long-term efficacy. It's hard to believe that the ones who stayed on the train are identical to those who left.

I have more to say about this study. The above provides a basic understanding of why this is no ordinary study.

## Comments

You can follow this conversation by subscribing to the comment feed for this post.