Big Data, Plainly Spoken (aka Numbers Rule Your World)

Data autonomous arguments are not data driven

I love "data-driven" arguments. You probably do too if you're reading this blog.

What does it mean to be "data-driven"? One prerequisite is that the conclusion should change as the data change. The analog that I write about frequently on the dataviz blog is to qualify as a data visualization (as opposed to just a visualization), the visual should change as the data change.

***

Recently, there have been numerous arguments that are immutable in the face of changing data.

John Burn-Murdoch at FT wrote this tweet: 

John dislikes this argument:

If there are spare beds, this proves ex-post that mitigation measures were not necessary.

There are different potential problems with this argument.

John is talking about limiting factors. In a complex system, many factors combine to produce an outcome. Typically, only one or a few factors are "limiting" in the sense that if those factors were either relaxed or tightened, the outcome would be directly affected. Meanwhile, there are many other factors for which manipulating them would not alter the outcome. These other factors may matter under other combinations but they don't matter right now. 

Another counter-argument is related to the direction of causality. One reason why the government may shut Nightingale hospitals (for those in the U.S., these are akin to the makeshift beds set up at conference centers) is because mitigation measures were successful in reducing the demand.

Yet another counter-argument is the assumed counterfactual. Because mitigation measures were taken, we could not observe what could have happened if they weren't. The absence of evidence is being taken as evidence of absence. 

***

But I think the best way to counter this type of argument is to point out its inherent data-agnostic autonomous structure:

I) On the one hand, if the outcome is X, then we conclude Y.

II) On the other hand, if the outcome is not X, then we conclude Y.

In other words, in all cases, whether X or not X, we conclude Y.

 

In the concern over Nightingale hospitals, this structure translates to:

I) On the one hand, if there are spare beds, then we conclude that mitigation measures were unnecessary, causing harm without benefit.

II) On the other hand, if they run out of beds, then we conclude that mitigation measures were ineffective.

In all cases, we conclude that mitigation measures were unwise.

The problem is that the same conclusion results regardless of the data (need for spare beds). So this is not a data-driven argument despite the presence of data.

 

***

The Bayesian way of thinking explicitly endorses the premise that conclusions should change as the data changes. Bayesian analysis leads to a posterior probability estimate, which explicitly depends on the data. If you feed the analysis different data, you get a different probability distribution. For an example of this, see this post in which I showed how the Pfizer vaccine efficacy estimate changes, assuming different trial outcomes.

The classicial (frequentist) method of hypothesis testing also produces different conclusions given different data inputs. The "null" hypothesis does not change but with different data, the location of the observed sample with respect to the "null" distribution changes, leading to different p-values.

 

[P.S. 1-1-2021. I switched from "agnostic" to "autonomous" as I think the argument being autonomous of the data conveys the meaning better.]

Happy hearty healthy holiday

A holiday wish.

Food pictures from traveling are in short supply this year so I'm showing you some joy from my kitchen.

Still shopping for a gift? Here are links to my two books: Numbers Rule Your World, and Numbersense.

The second worst kept secret of the pandemic year, and some decimal mischief

The worst kept secret of the pandemic year is Pfizer's vaccine efficacy. The second worst kept secret is... the Moderna vaccine, of course. Today, the FDA committee will recommend approval of this vaccine for emergency use as expected.

The general outline of Moderna's application is the same as Pfizer's so I'm more or less repeating my notes from the previous post about Pfizer's vaccine.

• The vaccine efficacy up to mid November (with about 50% of the trial participants having been observed for two months or longer) is above 90% in the trial population. This does not mean the VE is 90%; DJ Lemahieu of the Yankees batted .364 in 2020, what's the chance that he bats .364 or higher in 2021? Nevertheless, the VE even if it is 80% is better than good. Note again that the FDA's success metric is VE > 30%. The classical statistical analysis (outlined here) addresses this narrow question. When the FDA determines that VE > 30%, it is not simultaneously endorsing that VE = 94.5%, which is the number constantly paraded in Moderna's press releases.
  
  
• The duration of protection is unknown, and there isn't even full evidence for two months yet. The rush to catch up with Pfizer (and front-run the holiday) meant that today's FDA decision is based on an early analysis in which only 25 percent of the participants have passed the two-month-after-the-second-dose requirement. The maneuver to get around this rule is the submission of a supplement, two weeks later, of the full analysis that meets the two-month requirement. The FDA scientists have not had a chance to review the supplement in detail so today's decision is based on (a) the previous analysis and (b) Moderna's submission that the supplementary analysis is not materially different from the previous analysis.
  
  
• Any subgroup analysis is silly. The VE analysis only counts five cases on the vaccine arm so any subdivision by race, age group, severity of cases, etc. creates many subgroups in which the vaccine arm has not yet registered a single case. Be careful about people claiming "directional" evidence of these subgroups; such claims are bogus with these sample sizes.
  
  
• Moderna defines cases in an even stricter way than Pfizer in terms of symptoms. In addition, Moderna only starts counting 14 days after the second dose, which is given one week later than Pfizer's second dose. So the starting day of counting for Moderna is 42 days after the first dose while for Pfizer, it's 28 days after the first dose. In both vaccine trials, the gap between the vaccines and the placebos emerges after two weeks (some people will get infected after the first dose and before the second dose).
  
  
• There is also noise about a single-dose treatment. Unless it is assumed that the second dose is useless, I don't see how that conclusion can be acceptable. See this post.

***

That's not much more to say beyond those expected findings. I want to turn now to register my frustration with how the top-line VE number is being exaggerated for unscientific motives.

Recall that Pfizer has been yelling 95 percent while Moderna replies with "94.5%". The purpose is to have us round 94.5% up to 95% so that we think they are equally effective.

Why didn't Moderna just round up 94.5% themselves to 95%?

Turns out they can't! The 5 cases in the vaccine arm and 90 cases in the placebo arm lead to a VE of 94.46%. They then round this to one decimal place to 94.5%. But if they drop another decimal, they have to round it to 94%.

Moderna was clinging on to that second decimal for dear life. I realized this when on p. 23 of the briefing document, it is disclosed that "there were an additional 18 COVID-19 cases ... but were not able to be adjudicated in time for the interim analysis. Of these.., one was in the vaccine group, and 17 were in the placebo group." Let's assume these cases passed adjudication. So now, we have 6 cases in the vaccine arm vs 107 in the placebo arm. What's the VE? It's 94.41%. With the second decimal below 5, they would have to report 94%.

Don't forget that all of these calculations is for the earlier analysis in which only 25% of the participants have crossed two months of observation. In the supplement, Moderna reported 11 cases in the vaccine arm versus 185 cases in the placebo arm. What's the VE now? 94.1% so even further from 95%.

This bit of decimal mischief unnecessarily sullies the otherwise highly impressive conduct of these vaccine developers. The margins of error around the VE number is in the order of 10 percent so gaming the second decimal point is entirely pointless.

***

One issue I hope the FDA will address is the rapid jump in cases between the first and second analysis. Only two more weeks have passed between the two analyses. Yet, the number of cases doubled (95 to 196). It's hard to know why without the underlying data. Are these new cases happening on the front or the back part of the time line?

Unexpected sightings and unknowns from the Pfizer result

This is part 2 of my review of the FDA briefing document about the Pfizer coronavirus vaccine Phase 3 trial. Part 1 was published here.

Transmissions vs. Symptoms

You may have heard that the Pfizer vaccine (and any of the other ones that are being studied) does not necessarily stop transmission of the coronavirus from person to person. So what is the point of these vaccine trials?

The answer is in the success metric - comparing the case rates of the vaccine group relative to the control group. About 1 percent of the placebo group have been confirmed infected by mid November versus 0.05 percent of the vaccine group. Pfizer does not know how many people were infected but displayed no or mild symptoms.

This metric is chosen in part because of ethics. It would have been easier to track asymptomatic people and transmission paths if trial participants were locked up in a closed environment deliberately infected with the virus. But most scientists revolt at the thought of such an experiment.

Nevertheless, if fewer people are getting sick, fewer will succumb to Covid-19. So it's still a net benefit. Public health officials continue to recommend all mitigation measures to minimize any unnecessary deaths.

 

Vaccine uptake

I know some people are worried about vaccine uptake. I'm not that worried because of expected word of mouth. If this vaccine actually works as it did during the trial, then within a few weeks and by two months after the first vaccinations, people are going to notice the drop in cases, and the word will spread quickly. Of course, the flip side of the coin also holds: if cases don't start declining, Pfizer has a problem!

 

TV stunts

Please don't ask government officials to take the shots on TV. It convinces no one. People who believes in vaccine science don't need reassurance, and the anti-vaxx segment will assume it's a setup anyway. TV stunts juice up TV ratings, and that's it.

 

Unexpected Sightings

Here are a few things I read in the document that surprised me.

I already gave you one in the last post, which is that there were 325 total cases detected among the Pfizer trial participants, not 170 cases in the headline number. The gap is because Pfizer's result discounts anyone who got sick prior to seven days after their second dose. As I described in that post, after including those cases, the overall efficacy achieved during the trial was still a robust 82%.

Secondly, there were six deaths so far among trial participants, including two on the vaccine arm. Evidently, Pfizer did a better job of not drawing attention to deaths than AstraZeneca (link). Of course, deaths may not be caused by the vaccine.

Finally, Pfizer speculates that there may be no immunity coming from prior infection. This is their interpretation of a surprising observation from the trial itself. In one of the analyses of vaccine efficacy, Pfizer excludes people who showed evidence of a prior Covid-19 infection (e.g. via antibody testing) from calculating the case rates. In the placebo arm, there were 670 exclusions, nine of whom were subsequently confirmed infections during the trial (1.3 percent). This 1.3 percent is the same rate as for the remaining placebo arm of people who were not previously infected.

It is widely assumed before that the reinfection rate is zero or close to zero. Another possible explanation is problems with antibody testing; maybe the prior infections were misclassified. That's why one should not exclude data based on assumptions. The analysis that does not exclude the prior infected turns out to be more credible here.

***

The rush to a decision leaves a number of questions unanswered. Here are a few issues I'd be tracking going forward.

Potentially Relevant Factor 1: Duration of observation

Because of the rush, the vaccine developers are conducting analyses when only half the participants have reached two months after treatment (second dose). That was a stipulation by the FDA, discussed here. Many statisticians would have preferred the criterion to be all participants reaching two months. But that was not to be.

The briefing document provides more details on the distribution of follow-up time. Pfizer admitted that they have not quite met the FDA requirement. As of November 14, when the data were frozen for analysis, 44 percent of the participants reached two months of follow-up (below the 50% requirement but we'll let that pass). Thirty-six percent were in the second month of follow-up while 20% were in the first month of follow-up (after the second shot).

Why do statisticians want all participants to reach two months? That's because we want to draw the conclusion that the Pfizer vaccine yields protection for at least two months. The problem with requiring only the median participant to reach two months is that we can conclude only that the median protection is two months, that half the vaccinated people may have protection that lasts fewer than two months.

Next time you hear some talking head say we have proven efficacy for two months, remember that is not quite true. As noted above, 1 in 5 participants have not even reached one month of follow-up, and 1 in 2 participants have not reached two full months of follow-up.

The rule being applied to the median participant rather than all participants also damages any subgroup analysis. There were about 2,000 blacks in the vaccine arm. Does the median black participant have at least two months of follow-up? We do not know because that information has not been released.

If you ask any statisticians who design experiments, we'll tell you decisions have consequences.

 

Potentially Relevant Factor 2: Site-level differences

The trial is being conducted in the U.S. (77% of enrollees), Argentina (15%), Brazil (2%) and South Africa (2%). Nowhere in the briefing document does it break down the data by site. For example, I am curious where the cases were detected - all in the U.S. or spread out across countries? Also, they mislabel blacks as "African Americans," which implies there were no black participants in South Africa or Brazil or Argentina; strangely, Asians are described as Asians, not Asian Americans, when it appears that most if not all Asians would be Asian Americans.

 

Potentially Relevant Factor 3: Confirmation of cases

Most cases are self-reported by participants to Pfizer, who then confirms the infections. While the PCR test is the preferred method, the protocol allows for "quick tests" to be used in cases where PCR testing could not be performed. I'd like to know what percent of the cases were confirmed by PCR.

Further, Pfizer disclosed that a case is a case only if (a) the participant reports at least one of 9 symptoms and (b) a test confirms the infection within "4 days of the symptomatic period". I'm not a doctor so why the positive test must happen in those 4 days. Ordinarily, these little details are not important but for a vaccine trial in which we are talking about dozens or fewer cases, even shifts of 3 or 5 cases may matter.

 

Not Forgetting

Finally, it's useful to list some design decisions that were made to accelerate the trials. Any of these can have unintended consequences - if not, we would have done away with these practices in ordinary trials.

• Extra-short period of observation
• Single blind, not double blind
• Per protocol analysis, noting that in many of these trials, two doses are required weeks apart
• Exclusions both planned and unplanned
• Surveillance is not continuous, relatively infrequent, so that most cases are self-reported
• Not measuring asymptomatic transmission
• Not measuring the level of exposure to the virus (if someone hasn't been exposed to the virus, s/he won't get sick whether vaccinated or not.)

This is not to say these are bad decisions. It's that some risks are knowingly taken, and if anything doesn't meet expectations, these factors might be the reason.

One-dose Pfizer is not happening and here's why

I'm surprised even Dr. Sanjay Gupta has been low-key calling for a single-dose Pfizer vaccine because that's based on a dubious interpretation of the data.

Here's the one-dose vs two-dose story, written in Numbers Rule Your World style (link).

***

The entire Pfizer 50-page briefing to the FDA contains just a single data graphic, and this chart has become the darling of medical reporters.

The chart paints a powerful picture of vaccine science. The lines trace the number of confirmed cases known to Pfizer as the trial progressed, starting with people getting their first shot on "Day 0". The red line tracks the placebo control group: participants who received fake shots were getting sick at a linear clip. The blue line showing the vaccine group initially hugged the red line but after the second week, the blue line started to flatten, and remained flat during the period of analysis (up till Day 100 or so).

The messenger RNA technology in the Pfizer vaccine sends our body a blueprint on how to make antibodies for the novel coronavirus, and this evidence shows it takes about two weeks after the first dose to take effect.

In fact, we learned from the vaccine trial not only that the Pfizer vaccine is super but also that only one dose produces all the protection you need. That blue line bends right as the third week starts; there was not much further benefit after the second dose was given at or after Day 21.

Therefore, instead of the two-dose treatment that Pfizer originally thought was necessary, we should roll out a one-dose treatment. Doing so doubles the number of people who will get protected.

***

Now, let's start over. Go back to square one.

This chart paints a powerful picture of vaccine science. The lines trace the number of confirmed cases known to Pfizer as the trial progressed, starting with people getting their first shot on "Day 0". The red line tracks the placebo control group: participants who received fake shots were getting sick at a linear clip. The blue line showing the vaccine group initially hugged the red line but after the second week, a gap emerged between the two lines, and this gap continues to grow during the period of observation (up till Day 100 or so) as the blue line has almost completely flattened while the red line continues its linear upward trajectory.

Scientists obtain evidence of vaccine efficacy (VE) from the vaccine's share of confirmed cases (VSC). An effective vaccine should suppress the case rate - the baseline of which is measured by the placebo control group. The stronger the vaccine, the wider the gap between the control's case rate and the vaccine's case rate, the smaller the vaccine's share of confirmed cases. The following chart directly plots the VE and VSC as the trial progressed:

I also included the first chart so you can see that the new chart is a re-visualization of the data from the first chart.

The Pfizer vaccine requires two shots. Its benefit emerges over time. At the end of two weeks (Day 14), the case rate in the vaccine arm was 30 percent below that of the control arm (VE = 30%). The VE in the trial (brown line, right axis) breached 50% not long after the second shot was given on Day 21. This is an important moment because any vaccine must hit 50% VE to apply for emergency use authorization from the FDA.

What really impresses the medical community is the continual rise of VE as the vaccine's benefit shows no sign of abating during the next two months (from Day 30 to Day 90). As of the analysis time, the Pfizer vaccine appears to be 80% efficacious about two months after the second dose.

The evidence from the trial confirms Pfizer's two-dose treatment program. The first dose provides bare-minimum protection. The combination of two shots lifts the VE from marginally acceptable to stratospheric.

***

It's a tale of two data graphics and two visual stories - arising from the same underlying data. Both the power of visual data story-telling and its danger are on display. The two stories lead to diametrically opposite recommendations, so only one of these stories can be correct.

The author of the first story has hidden an assumption about what the curve would look like beyond Day 21 if the second dose were scrapped - that the blue line in the first chart would just trace the same path as shown above. This assumption is equivalent to saying the second dose has no value because it asserts that the vaccine arm performs identically with or without the second dose.

In addition, this assumption stipulates that with just one dose, the vaccine efficacy would improve from about 30% after the second week to 80% by around week 75. This is possible but there aren't data to support it.

Alternatively, we make the assumption that should the second dose be scrapped, the vaccine efficacy would be frozen at the level reached on Day 21, and remain there for the rest of the analysis period. This leads to the following path (shown in dashed lines):

The gap between the vaccine and placebo curves is cut by half under this assumption. There are two reasons why this assumption may still be too optimistic. I'd assumed that the 50% efficacy seen in the trial will persist - which is not a given; in fact, if the trial returns a 50% VE, we are only comfortable saying the VE in general is over 30 percent. Besides, it's plausible that the vaccine's effect will wane over time.

For these and other reasons, neither Pfizer nor the FDA has accepted the one-dose argument.

The uncertainty goes away if a new clinical trial is designed to test the one-dose against the two-dose treatment. From a public health perspective, such a trial is desirable as it reduces the cost, the complexity and the speed of vaccination. The flip side is it also reduces revenues, and potentially the efficacy.

 

 

 

These conditions make the Pfizer vaccine result irrelevant to you

Once the vaccines roll out on a massive scale, our pace of learning will quicken, as the sheer numbers will reveal previously unknown, or unclear, issues. There are already a few cases that emerged from the U.K. that involved adverse reactions that are being investigated further. Some commentators believe that these people have conditions that would have excluded them from the Pfizer trial. What they mean is the Pfizer trial results don't apply to them, because anyone with those conditions were excluded from the experiment. (I wrote about exclusions here.)

Such exclusions are routine, and done for the safety of the excluded people. The flip side is those excluded types of people are more likely, perhaps much more likely, to experience adverse events, for that's why the researchers didn't want to include them - for ethical considerations.

There are a lot of exclusions in the Pfizer trial. The following is a full list. If any of these conditions apply to you, then you should know that the trial participants do not have these conditions, and thus the trial result may not apply to you (in terms of efficacy as well as safety results). Consult your doctor.

I have just copied these from the protocol, unaltered (The conditions that apply to the Phase 1 trial only are omitted.)

Note: Condition #2 referring to HIV status supposedly applies only to the Phase 1 and 2 trials but according to the FDA Briefing document, Pfizer only started enrolling HIV-positive people in the latter stage of the Phase 3 trial (this creates a situation in which the results we are seeing for this subgroup include few participants who meet the two-month-post-second-dose requirement.)

***

Medical Conditions:

1. Other medical or psychiatric condition including recent (within the past year) or active suicidal ideation/behavior or laboratory abnormality that may increase the risk of study participation or, in the investigator’s judgment, make the participant inappropriate for the study.

2. Phases 1 and 2 only: Known infection with human immunodeficiency virus (HIV), hepatitis C virus (HCV), or hepatitis B virus (HBV).

3. History of severe adverse reaction associated with a vaccine and/or severe allergic reaction (eg, anaphylaxis) to any component of the study intervention(s).

4. Receipt of medications intended to prevent COVID-19.

5. Previous clinical (based on COVID-19 symptoms/signs alone, if a SARS-CoV-2 NAAT result was not available) or microbiological (based on COVID-19 symptoms/signs and a positive SARS-CoV-2 NAAT result) diagnosis of COVID-19.

8. Immunocompromised individuals with known or suspected immunodeficiency, as determined by history and/or laboratory/physical examination.

10. Bleeding diathesis or condition associated with prolonged bleeding that would, in the opinion of the investigator, contraindicate intramuscular injection.

11. Women who are pregnant or breastfeeding.

Prior/Concomitant Therapy:

12. Previous vaccination with any coronavirus vaccine.

13. Individuals who receive treatment with immunosuppressive therapy, including cytotoxic agents or systemic corticosteroids, eg, for cancer or an autoimmune disease, or planned receipt throughout the study. If systemic corticosteroids have been administered short term (<14 days) for treatment of an acute illness, participants should not be enrolled into the study until corticosteroid therapy has been discontinued for at least 28 days before study intervention administration. Inhaled/nebulized (except for participants in Phase 1 – see exclusion criterion 14), intra-articular, intrabursal, or topical (skin or eyes) corticosteroids are permitted.

15. Receipt of blood/plasma products or immunoglobulin, from 60 days before study intervention administration or planned receipt throughout the study.

Prior/Concurrent Clinical Study Experience:

16. Participation in other studies involving study intervention within 28 days prior to study entry and/or during study participation.

17. Previous participation in other studies involving study intervention containing lipid nanoparticles.

The worst kept secret will be revealed today

Today's FDA meeting will reveal the worst kept secret ever - that the committee will approve the Pfizer vaccine for Emergency Use Authorization (EUA). Given the severity of the pandemic, the urgency of relief, and the promising initial data from the vaccine trial, there really isn't any other possibility!  

Pfizer has submitted a Briefing Document to the FDA to secure EUA, and I'm highlighting parts of this briefing that should interest you.

 

Top-line Conclusion: Trial results are good, no need to embellish

 

Pfizer has been selling this conclusion: the vaccinated participants had a case rate (not the same as infection rate) that is 95 percent lower than the placebo (unvaccinated) participants. 

However, focusing on this selling point is likely to harm the reputation of the vaccine because real-world experience will almost surely refute that. As I pointed out in this post, the PR derby by Pfizer, Moderna and others have moved our expectation from 50 percent VE to 90 percent VE. If such a level of VE is achieved, then people would be right to expect the pandemic to disappear in short order - but that simply isn't the belief of most public health experts.

There are two notable takeaways from the document that I bet you won't hear from the mainstream press:

a) For FDA approval, "The criterion for success was met if the posterior probability that true vaccine efficacy >30% conditioning on the available data was >99.5% at the final analysis." It doesn't matter whether the VE in the trial is 50% or 70% or 90%, they will all be approved based on the 30% threshold. (See yesterday's post to learn why Pfizer's trial performance would have to exceed 50% in order to prove VE > 30%.)

b) We keep hearing that a total of 170 cases were detected during this trial, split 8 cases in the vaccine arm and 162 in the placebo arm. These are cases that were detected within the analysis window. The start of the analysis window is not the day of enrollment or the day in which the treatment began (i.e. the first shot). The analysis window starts seven days after the second shot and ends at the time of analysis (which was November 14). The briefing disclosed that 325 cases, not 170, were documented during the trial. The other half occurred after the first dose and before seven days after the second dose, and are excluded from the headline analysis.

As I previously pointed out (link), this case definition is impractical. Someone who got sick after the first dose before getting the second dose is a failed vaccination. Pfizer submitted a secondary analysis that included all 325 cases, and in this calculation, the VE declines to 82%. There is a margin of error around these VE estimates; the lower bound is 76%. This is still a good result, and I trust this number more than 95%.

The difference between the two analyses (with and without the early cases) tells you that the vaccine wasn't performing anywhere close to 90% efficacy after one dose. During the 28 days after the first dose, and before Pfizer starts officially counting cases, the vaccinated participants have roughly half the chance of getting sick as the placebo participants. Remember that 50% VE in the trial will translate to VE = 30% in general so barely meeting the minimum FDA standard (explained here).

The 50% VE has led to some ill-informed commentary (all over the news channels) that a one-dose regimen should be approved. I address this folly next.

 

Open Issue 1: Should a one-dose treatment with 50% efficacy be approved?

 

This seems to be one of the talking points in the media, most of whom appear not to have read the briefing document, or if they did, they decided they know more than the scientists at Pfizer. Because the Pfizer briefing literally states this:

The efficacy observed after Dose 1 and before Dose 2, from a post-hoc analysis, cannot support a conclusion on the efficacy of a single dose of the vaccine, because the time of observation is limited by the fact that most participants received a second dose after three weeks. The trial did not have a single-dose arm to make an adequate comparison.

There are several problems with concluding that rolling out a one-dose treatment would result in cutting case rates by half. First, as explained in yesterday's post (link), reaching 50% efficacy in a specific trial only shows VE = 30% in general. So at best, the single-dose treatment can be predicted to cut case rates by 30%, not by half.

Second, we have no idea whether the effect of the first shot holds beyond the first three or four weeks because everyone who isn't already sick and hasn't dropped out of the study by day 21 was given the second shot. We are not sure that the top-line VE of 95% holds beyond two months since less than half the participants have reached that point; similarly, we cannot be sure about the single-dose VE beyond four weeks. Because of the trial design, we can answer the former question after more months of follow-up but we will not receive any further data to adjudicate the latter question.

Anyone arguing for a one-dose treatment is making the assumption that there will be no degradation in protection after the first month, without any data to support it. Note that Pfizer researchers elected to test two doses after a Phase 1 trial so a priori, they believe two shots are necessary. In my view, the one-dose treatment should be rejected. Pfizer should design a new trial to test the one-dose treatment against the two-dose control.

 

Open Issue 2: Should those who received placebo be given the vaccine?

 

Pfizer's position is clear: "The Sponsor plans to offer vaccination to participants > 16 year of age who originally received placebo and who become eligible for receipt of [the vaccine] according to local or national recommendations... these participants will be unblinded upon request and will have the opportunity to receive [the vaccine] as part of the study."

Pfizer does not comment on how this affects subsequent analyses, in particular, how this action affects our ability to understand long-term protection, severe cases, and effects on subgroups. There is no doubt that the proposed plan destroys the randomization in the vaccine trial, making it much harder to interpret the results. Those remaining in the placebo arm will clearly not be the same as those who got the vaccine.

But there are some practical and ethical issues that come into play. Is it possible to stop people from seeking the vaccine on their own (and these people will be violating the terms of enrollment)? Can Pfizer ethically turn away such requests?

Further, what is the domino effect on other ongoing trials? If placebo participants got the Pfizer vaccines, the protocol will drop them from the analyses; the claim is that the trials are blinded but it's only single-blind, and it's pretty clear that those who got vaccines may have mild symptoms of various types so bias can creep into these trials.

The randomized controlled trial is the gold standard for proving the efficacy of any new treatment. The randomization of treatment is a foundation to such trials. So it'll be interesting to see how the committee decides this issue.

 

Secondary Findings are Meaningless

Since the trials were designed to get a top-line result as quickly as possible, they have too few samples to draw any reliable conclusions on any of the following topics:

a) period of protection

b) subgroups, such as age groups, comorbidities, racial groups

c) severe cases

d) rare but potentially fatal adverse events

There were only 9 total cases on the vaccine arm used in the analysis. Any kind of breakdown by subgroup means in many subgroups, there would be zero cases on the vaccine arm. Having observed zero cases on a small sample does not merit the conclusion that the vaccine is 100% protective. Pretty much all the margins of errors reported in the subgroup analyses are laughably huge, such as -159% to 99.5% for Asians. Nothing to see there.

Same thing with respect to protection against severe cases. There were only 4 severe cases used in the analysis (split: 1 vaccine, 3 placebo). The interval estimate for VE is -125% to 96%. The statistic screams you don't have enough data.

The absurdity of these intervals has not deterred commentators who just ignore them, and tell us, for example, that the vaccine efficacy for Asians is 74%. What those intervals tell us is that if we ran the Pfizer trial again and again and again, the observed VE for Asians in any one trial would be a number between -125% and 96% (extreme values are rarer but still possible); what they tell us is that we should not draw conclusions from this observation.

The implication of d) is that we must accept an inherent risk of rolling out a new vaccine. The trial would have to be unacceptably huge to detect rare adverse events. With 20,000 participants on each arm, you'd expect on average one instance of a rare problem with 1 in 20,000 chance of occurring (meaning in many such trials, you'd observe zero such events). If the chance is 1 in 100,000, for example, there is basically no chance of finding it during a trial of this size. We will however learn of these rare events when the vaccine is rolled out to millions so active monitoring is essential.

 

Exclusions are Meaningful

There are two types of exclusions - participants who enrolled in the trial but are not included in the analysis.

The first type is designed exclusions. The protocols have stated all kinds of restrictions. Pfizer initially did not allow HIV-positive people in the trial, for example, and then it did enroll some. The types of people who were excluded upfront may seek the vaccine now, and if so, we know neither the efficacy nor the potential harm on them. [P.S. I pulled down the list of exclusions from the Pfizer protocol. You can check the list here.]

The second type are part of the trial but subsequently got thrown out for any number of reasons. For example, someone drops out. Also, anyone who got sick after the first dose is not given the second dose, and so by definition, did not complete the treatment program. In a per-protocol analysis (which is what the FDA accepts as primary), those people are excluded from the analysis population.

The statistical question is if these people had been included (i.e. if they had not dropped out, if they had not gotten sick before the second dose,...), how many would have gotten sick? For example, by my calculation, 533 people were dropped from the vaccine arm after the first dose while 293 were dropped from the placebo arm, and it appears that the reason for dropping these is "protocol deviations that happen up to 7 days after Dose 2". I'd be interested in learning what those deviations were, and why there were 80% more frequent in the vaccine arm.

 

***

Overall, the data present a good story and so I don't doubt that the Pfizer vaccine will be approved.

 

P.S. [12/11/2020: While working on the next blog post, I realize another reason why the subgroup analyses must be viewed with great skepticism for now, and why we must wait for a longer observation period. Recall that under FDA guidelines, these results include up to half of the participants to have been observed for fewer than 2 months. That condition is applied to the aggregate population, which is roughly 20,000 on each arm. Now, when you drill down to subgroups, the subpopulation sizes are much lower, and therefore you're going to have subgroups in which more than half of the participants have been observed for fewer than 2 months. Pfizer can publish the breakdown by length of observation to allay my concerns. In any case, if we wait longer, then everyone would be observed for 3 months, 6 months, etc. so the concern will eventually go away.]

 

 

Using classical statistics to analyze the Pfizer vaccine trial

I previously described how Pfizer told the FDA they're analyzing their trial data, using Bayesian analysis (link). In this post, I describe how classical statisticians would have analyzed the same data.

The beauty of having simultaneous vaccine trials is our ability to compare and contrast the trial designs and outcomes. Even though Pfizer and Moderna are testing vaccines using similar messenger RNA technologies, their trial designs have notable differences. For example, Pfizer counts cases from the first day after the second dose for seven days while Moderna counts cases starting 14 days after the second dose for 14 days. (Remarkably, those two observation windows are disjoint; it would be interesting to hear why each company's researchers made those choices. I have notes on the Moderna protocol here. This timing difference explains why Pfizer was first out of the gate.)

Another major difference is the analysis method. While Pfizer uses a Bayesian analysis, Moderna's statisticians are following the traditional method (sometimes called classical or frequentist).

***

From the previous post, you learn that the Bayesian's answer is a probability curve that gives the probability that the vaccine efficacy (VE) takes on some specific range of values, given what was observed during the vaccine trial. I reproduce the key graph here:

In particular, the FDA is expected to look for the chance that VE is at least 30 percent, i.e., the proportion of the area under the probability curve to the right of the 30% mark. Given the hugely optimistic trial data, the chance is essentially 100% as shown above.

***

Classicists (i.e. Frequentists) start from a different place. They believe the efficacy of a given vaccine has a fixed value, but it is unknown, and is estimated using the data from a randomized controlled trial. The last released result from the Pfizer trial is that 8 of 170 detected COVID-19 cases have occurred in vaccinated participants. This result is reported in two ways: (a) the vaccine share of cases (VSC) is 5% (= 8/170) and (b) the vaccine efficacy is 95% (note, again, this means that vaccination reduces the case rate by 95 percent. It does not mean 95 percent of vaccinated people will be protected from infection.)

The simplest analysis is to take the result from the trial and run with it. This means we are generalizing data from 43,000 people to all 330 million Americans (I'm ignoring the fact that some participants in the trial are outside the U.S.). Statisticians believe this simple analysis is too simple.

Imagine parallel universes in which we would roll out the same Pfizer trial. In each universe, a different set of 43,000 people would participate in the trial, and produce a different number of cases in the vaccine arm - probably leading to a VE close but not identical to 95%. We call this variability due to sampling.

The question then is if we could run these parallel universes, and count the number of cases in the vaccine arm in each, what would the distribution look like?

The shape of this distribution depends on how effective the vaccine is. So a chart like the above must assume a value of VE. The natural assumption is VE = 30% which is the minimum standard to get FDA approval. That's what I assumed when making the above chart.

The featured distribution addresses a very particular question: if the vaccine has 30 percent efficacy, how many COVID-19 cases do we expect to find among the vaccinated? The most likely outcome is 67 cases in the vaccine arm, and this outcome is expected in 6 percent of the parallel universes. But anything is possible: in some scenarios, we would find only 1 case while in others, there might be 140 cases.

Notice how classicists adopt a precise frequentist definition of "chance" as the proportion of all parallel universes. Bayesians interpret "chance" as a subjective probability.

***

Another difference is that classicists provide a precise answer to a limited question: do we have sufficient evidence to show that the vaccine is at least 30% efficacious? Yes or No. Bayesian analysis, by contrast, has the flexibility of answering many different questions, with each answer being a probability. What's the probability that VE > 30%? or VE > 80%?

Classical analysis establishes a cutoff of 54 cases. If there were over 54 cases in the vaccine arm during the actual vaccine trial, then the data are consistent with the vaccine's efficiacy being 30 percent. In this case, the trial fails to show that VE is higher than 30%. If there were 54 or fewer cases in the vaccine arm, then the data are inconsistent with VE = 30%. In this case, we conclude that VE > 30%.

In Pfizer's trial, they found only 8 cases among the vaccinated population, so according to the chart above, the classical analysis comes to the same conclusion as the Bayesian analysis - that VE > 30% is more or less certain.

***

How come we keep hearing that the FDA requires vaccine efficacy of 50% while I keep saying it's 30% (according to the design protocols)?

In the above classical analysis, the cutoff is 54 cases in the vaccine arm - out of 164 total cases. So to show VE > 30%, the vaccine's share of cases (VSC) during the trial must be below 54/164 = 33 percent. This value of VSC can be translated back to VE: VSC of 33 percent is equal to VE of 50 percent. In other words, if the VE is 50 percent (or better) as measured in the Pfizer trial, then we have proven that VE > 30 percent in general.

Note that the first VE number (50 percent) is computed from one sample of 43,000 people, and the second VE number (30 percent) refers to (the average of) all parallel universes. If the first number is 45 percent or 40 percent or 35 percent, statisticians are not comfortable that the second number is over 30 percent. This is because of variability from sample to sample.

To reiterate, the FDA minimum requirement is not 50 percent but 30 percent. However, in order to prove that the vaccine has 30 percent efficacy (over the entire population, not just in a single sample), the vaccine must show 50 percent or higher efficacy during the vaccine trial.

This post about frequentist analysis is inspired by the blog post titled "A Vaccine Trial from A to Z" by Stephen Senn, contributed to Deborah Mayo's blog.

***

Underlying all of this is the logic of hypothesis testing. This is a topic I covered in more detail in Numbers Rule Your World in Chapter 5. Get your copy of the book here.

 

P.S. For those who wonder about deriving the cutoff of 54 cases in the vaccine arm: the Pfizer protocol states that 164 total cases triggers the full analysis. Showing VE > 30% is the same as showing VSC < 41.2%. During the trial, out of 164 total cases, how many will appear in the vaccine arm, given that there is 41.2% chance of each case being in the vaccine arm? That's a binomial distirbution with N = 164 and p = 41.2%. The cutoff is then based on a tail probability of 2.5%. This means if VE = 30%, there is still a 2.5% chance that 54 or fewer cases appear in the vaccine arm. According to our decision rule, we would have concluded that VE > 30%. This means we are allowing a 2.5% chance of falsely concluding that VE > 30%.

The importance of matching data

We keep hearing that Covid-19 cases may be rising but the death rate is dropping. This is usually supported by appealing to the trends of cases and deaths. Here is an example for the state of Illinois:

Both cases and deaths are expressed as index values relative to April 1st so they can be directly compared. If you pick a date and compute the death rate as deaths divided by cases, the graph above seemingly shows a declining death rate.

While we all wish this to be true, it is too good to be true. The simple analysis described above ignores the timing of deaths. Typically, someone who dies from Covid-19 does not die on the same day s/he gets diagnosed. Death may come up to a month or longer after the positive test. Take the deaths from December 1st. These patients likely tested positive mid October to mid November. So to compute the death rate, we should divide by cases from about a month ago, not the cases from December 1st.

Notice that cases surged in November, and most of those infected won't show up in mortality counts until December. If we divide deaths on Dec 1st by cases on Dec 1st, the surge of cases is what drives the death rate down. The declining "death rate" is less due to lower deaths but fast-growing cases.

A clearer analysis requires dividing deaths of each day by the cases from about 20 days ago. I described how this is done back in April in this post about Lombardia in Italy. Today, I apply this methodology to current data for the states of Illinois and Iowa.

***

In Illinois, the lag between cases and deaths is around 25 days.

The thin gray line is the same line as in the first graph. Here, I lag it by 25 days, shifting the curve to the right. Doing so matches the deaths from one week with the cases from 25 days ago. You can see the matched dates on the bottom axis.

Both lines start at 100 on July 1st. You can see that deaths pretty much follow the shape of the trend of cases. This indicates that the chance of dying from Covid-19 given that one has tested positive has been relatively stable during these many months.

***

In Iowa, the lag between cases and deaths is around 17 days.

A similar picture emerges.

It's difficult to compare state-level statistics because each state has its own set of rules for reporting. Also bear in mind that all dates are reporting dates, and don't necessarily reflect the day of infection or the day of death. So the error bar around the exact time of lag is pretty wide.

It's time to visit Florida again - graphically

In May, I showed the following chart that presents a way to understand excess deaths in Florida (link to post):

It was only two months into the pandemic: because actual deaths from death certificates take time to count, it was just a sign of things to come. We start with a projection of expected deaths as by seasonal flu ("pneumonia & influenza") made based on prior flu seasons, expressed as a percentage of total deaths by any cause. In the first two months of the pandemic, the reported deaths by seasonal flu was as much as double the typical percentage of all deaths. On top of that, there were deaths related to Covid-19, which were roughly equal to deaths by seasonal flu during April.

There was no particular reason why seasonal flu deaths should be significantly above normal during the tail end of the 2019-20 flu season so it was suspected that Florida might have been undercounting Covid-19 deaths.

Months later, most of the dust has settled on these earlier months, and many of those deaths appeared to have been reclassified as Covid-19 related. I revisited this analysis below.

***

Let's start with seasonal flu statistics for the past five seasons (from 2015-6 to 2019-2020).

Flu-counting starts in late September/early October each season. According to the CDC, flu season typically reaches a peak in the winter, and lingers on till as late as May. Pneumonia & influenza accounts for 1 to 2 percent of total deaths in any given week of the year, on average in the four seasons prior to 2019-2020. The 2019-20 flu season was a bit late arriving so when Covid-19 cases started getting detected in March, flu deaths were just easing from the seasonal peak.

The above chart - specifically, the thick gray line - establishes what we expect the flu fatalities to be during the pandemic months from March to Oct 2020. The orange lines show the actual deaths attributed to seasonal flu. These counts are definitely smaller than the numbers I found back in May. The revised counts make more sense since the flu is seasonal. (I left out the November data because they are incomplete at the time of writing.)

What happens when we layer on the Covid-19 deaths?

The spike of data ran off the page.

I kept the scale of the lower part the same as before. In March and April, while pneumonia and influenza accounted for 1 to 2 percent of all deaths in Florida, when Covid-19 were lumped with the other two causes, they reached about 8 percent of all deaths, roughly four times as many. In the summer months, over a quarter of the deaths in Florida were linked to Covid-19, pneumonia or influenza.

In a normal flu season, we might see 50 to 100 deaths per week in Florida due to pnuemonia or influenza. Since April, Florida has suffered Covid-19 deaths in the hundreds, and during the peak in the summer, over a thousand per week, roughly 10 times above expectation.

Here's a gif that shows the top of the chart: