Big Data, Plainly Spoken (aka Numbers Rule Your World)

The children

A quarter of cases reported in the U.S. are now in children

This is what the data are showing. The talking heads tells us there is nothing to think about - it's simply because children are not vaccinated.

But there is plenty to think about:

1) Children return to school. Especially where kids are not vaccinated, schools may be testing kids regularly. As our politicians made famous, more testing leads to more cases.

2) As testing ramps up among children, the number of tests conducted in vaccinated adults has dropped because public health officials discourage vaccinated people with mild symptoms from getting tested. Vaccinated people with mild disease are inadvertently hurting our fight against Covid-19 in two ways: many are spreading the virus, and many are not getting tested, thus biasing our datasets. I'd love to see testing statistics split by vaccinated vs unvaccinated. Have anyone seen such data?

This previous post explains why it is very hard to interpret case trends when seleciton bias permeates the dataset. I present a framework to think about these issues.

3) It's also weird that we should worry about cases in children since for months, the same talking heads have said these vaccines have never been designed to prevent mild cases (a claim that is debunked just by opening the first page of any of the clinical trial protocols, or perhaps finding the videos from 2020 in which the same talking heads can be found extolling 95% efficacy which was computed using cases.). As is well known, the risk of severe disease and deaths in children is tiny so if we should be alarmed by cases in children, then we must be worried about spread. If we are worried about spread by infected children, why are we not worried about spread by infected, vaccinated adults?

4) A quick trip to the Census reveals that about a quarter of the U.S. population are under 20.

5) There are many possible reasons why children is an increasing share of cases. Many of these reasons are artifacts of selection biases that pollute all of our very poor-quality datasets. These are artifacts - decoys that confuse us. We will continue to make very bad decisions if we continue to take biased datasets and gloss over their flaws.

 

 

 

 

 

 

 

Receipts from a year of blogging the pandemic

Today is a good day to review some of the things you've read on this blog all through the pandemic.

***

As governments pushed very hard for third doses of the Covid-19 vaccine, it may feel like ancient history when - back in January, only about six months ago - many experts were pushing for one-dose mRNA vaccines, claiming that a single dose gives you close to 90% effectiveness.

I sounded the alarm in a post called One Dose Vaccine Elevates PR Over Science (January 2021), which began with:

"I fear that the U.K. policy of one-dose vaccines will backfire, and cause the pandemic to continue longer than necessary."

We are now certain that those who argued for single doses advocated a policy that likely resulted in unnecessary suffering.

In another post, appearing right after the Pfizer EUA in December 2020, I laid out the reasons why the data cannot support a one-dose regimen. (One Dose Pfizer is Not Happening and Here's Why) Nothing I said in December has aged. They are all still true.

***

In that same post, I predicted that "Partial protection provides a convenient excuse for vaccinated people to do away with inconvenient mitigation measures." Little did I know that CDC would subsequently endorse this folly by telling vaccinated people to drop their masks.

The CDC guidance was based on hope-fueled, over-interpretation of the data. Until recently, most experts have claimed that the vaccine stops infection, even asymptomatic infection, and spread. Now, they retracted those claims. But none of those outcomes were ever formally measured in the clinical trials.

Eight months ago, when vaccinations were just starting, you read here: "I think two doses are closer to 70 percent effective in reality, and we don't know that the vaccine stops asymptomatic spread, and so continuing to reduce contacts is advisable." Many places that removed those restrictions have been forced to reimpose them.

***

"One of the key lessons of managing this pandemic so far is that good data drive good decisions, and bad data drive bad decisions. Unfortunately, policymakers have signed up for bad data, so no one should be surprised if future policies turn out badly."

That was the conclusion of another post from January, Actions have Consequences: the Messy Aftermath of One-Dose Pfizer. The situation, as of August, has not improved. At every turn, governments have failed to collect relevant data, sufficient data, and good-quality data. In fact, they have actively interfered with the ability to learn from data.

The central example from that January post explains one (of many) reasons why real-world vaccine effectiveness studies have wildly exaggerated the effectiveness of the mRNA vaccines.

"A fundamental best practice of running statistical experiments on random samples of a population is that once the winning formula is rolled out to the entire population, the scientists should look at the real-world data and confirm that the experimental results hold."

"The action of the U.K. government (and others who may follow suit) has severely hampered any post-market validation. It is almost impossible to compare real-world evidence with the experimental result, because most people are not even getting the scientifically-proven treatment per protocol!"

That's right. The original vaccine efficacy measure came from clinical trials in which the participants followed a precisely-timed two-dose regimen. In the real world, many countries deviated from the prescribed protocol, adopting policies such as one doses, or extending the dose interval from three weeks to three months, or mixing and matching vaccines. It is an insult to science to pretend we are comparing apples to apples.

These issues did not catch anyone by surprise. When I raised those concerns in January, Pfizer was just being rolled out and not a single real-world study has commenced.

A particularly galling detail is very telling. In the U.K., the average duration between two doses is 80 days (almost 3 months!) When the U.K. studies apply the (in)famous 2D+14 case-counting window - that is to say, the researchers nullify all cases that occur after the first shot and before 14 days after the second shot, they are discounting cases for more than three full months! The rationale for the case-counting window is that the vaccine requires time to attain optimal effect - surely, a vaccine that needs 94 days to become effective isn't one that has practical value!

***

It is sobering to look at the data today (well, yesterday when I pulled this chart), courtesy of the fine folks at OurWorldinData:

Israel and the U.K. won countless headlines in the first half of the year when their case rates dropped to historic lows - they were the loudest in attributing the entire drop in reported cases to one and only one cause: widespread vaccinations. Today, they have some of the highest infection rates, much higher than in South Africa and India, which had experienced brutal surges but have relatively low vaccination rates.

In the first week of May, I wrote a post called Curve Watching During the Pandemic. Even when the cases were low in Israel and the U.K., if one was willing to look at non-conforming data, one would have noticed gaping holes in the idea that vaccinations were the sole explanations for the trends at the time.

In that post, I concluded: "Simple one-factor models aren't going to work to explain trends across countries and time. A good causal model should include a baseline trendline, a vaccine factor, plus lockdowns and other mitigation measures." To this day, none of the studies covered by the media follow this strategy.

***

It is well past time to admit that the real-world vaccine studies have dramatically over-stated the effectiveness of the mRNA vaccines. This is a failure of science. Not only, it is a systematic failure because I do not know of a single study that has committed the opposite error of under-estimating the VE! It's clear science has not fielded an A team in this crisis.

On this blog, I reviewed many of the influential real-world studies that created the narrative that vaccines are as effective in the real world as they were in clinical trials. I have indicated a wide variety of problems with such studies, and all the reasons why their conclusions are over optimistic.

If you are interested in research methodologies, read this series of posts from March 2021:

Real-world Studies: How to Interpret Them (Mayo clinic/nference)

Real-world studies: limits of knowledge (Mayo clinic/nference)

The Confusing Picture in Israel and in the Israel Study (Israel Clalit)

Note on a Simpson's Paradox in Real World Vaccine Effectiveness Studies (Denmark)

What the Danish Study tells us about the CDC Study on Real-World Effectiveness (Denmark, CDC)

Eventually in July, Public Health England published a study that provided some raw data which I harnessed to explain these abstract concepts. This exercise led to a post called Real World Vaccine Studies Consistently Overstate Vaccine Effectivness. In this post, I laid out several key adjustments that should be made to real-world VE calculations, and how these adjustments would have resulted in far less rosy pronouncements. My conclusion about mRNA vaccines: "A realistic estimate of VE is probably closer to 60%, which is an excellent number."

This is no mere squabbling amongst scientists. I repeat what I said above: "Good data drive good decisions, and bad data drive bad decisions." Throughout this crisis, the policymakers did not work with good data, by this, I include good analytical findings.

***

In my very first review of a real-world study (Mayo Clinic/nference), I stated the fundamental challenge facing real-world studies: "The simplest first analysis is to compare the case rate of the vaccinated people to the case rate of the unvaccinated people. This is hopelessly flawed because in a real-world study, we must not assume 'all else equal.' People who have received the vaccines at this point are apples and oranges to those who haven't."

That was March 2021. Fast forward to August, and I'm saddened to report that the situation has deteriorated. Unfortunately, the more recent studies have almost all adopted the "simplest first analysis" that is "hopelessly flawed".

Take the recent full court press relating to the first few days of data after Israel started giving third doses to older people. On the next day, the case rate amongst those who got the third shot is immediately below those who only had two shots. To a statistician, this gap presents the best estimate of selection bias we know of - no vaccine can be expected to work at full strength within one day so the more likely explanation for it is that people who rushed to the front of the line have lower baseline propensity to get infected by the coronavirus. In other words, the third dose unintentionally reveals who the lower-risk people are.

I discuss this in a recent blog post, which showed up on the dataviz section of my blog. This selection bias is yet another reason why the early real-world studies - even those using more advanced methods - over-estimated the VE. Thus, I concluded: "Statistics is about grays. It's not either-or. It's usually some of each. ... When they rolled out two doses, we lived through an optimistic period in which most experts rejoiced about 90-100% real-world effectiveness, and then as more people get vaccinated, the effect washed away. The selection effect gradually disappears when vaccination becomes widespread. Are we starting a new cycle of hope and despair? We'll find out soon enough."

***

This is a very long post to say thank you for your past year of support, and tell your friends about this blog. I can't promise I will get everything right but so far, the record looks good :)

 

P.S. The big news of this week is the FDA's "full approval" of the Pfizer vaccine. I am not devoting more space on the blog for it than this paragraph because there is nothing to discuss: the only additional information that was publicized since the interim analysis in December was the "6-month" update, which is nothing but (link). This was not a decision based on science. In fact, on cable news yesterday, the experts were not talking about the science. The entire rationale for this decision sounded as if someone at the FDA spent time reading consumer surveys. I refer you to Peter Doshi who nicely summarized the issues at BMJ (link).

 

Tokenism in statistics

Tokenism is an act of virtue signaling: taking an action in order to create an appearance of having solved a problem but the appearance runs ahead of reality. The canonical example is the hiring of "token minorities" to demonstrate that businesses have inclusive hiring practices. The symptom is the appearance of a single female or minority face in a group picture. Remember this Buzzfeed reveal?

From a statistics perspective, tokenism works by elevating the anecdote - the outlier example being peddled as representative.

***

This post is my third set of notes inspired by Pfizer's 6-month study on the safety and efficacy of its Covid-19 vaccine. The other two posts are here and here.

Today, I draw attention to this paragraph (bolding is mine):

The current report focuses on phase 2/3 safety assessments in ≥16-year-old and prespecified VE assessments in ≥12-year-old participants through up to 6 months post-immunization. Because enrolment of 12-15-year-olds began on October 15, 2020, 6-month post-immunization data are currently unavailable for this age cohort. Shorter duration safety, immunogenicity, and efficacy data for 12-15-year-olds are reported separately; however, data for this cohort are included in overall VE analyses reported here.

The original trial involved people 16 years old or above, and 44,000 participants enrolled between end of July and end of October 2020. Later, a teenager trial was added, with enrollment of ~2,000 between October and January, 2021. The overlap between the two trials was extremely short, as at interim analysis of the adult trial (conducted in mid November), Pfizer said there were only 100 12-15-year-olds with data.

The interim analysis generated a result of the form: "Vaccine efficacy for ≥16-year-old is 96% from 7 days after 2nd dose to median 2 months." This new study led to a result of the form: "Vaccine effiacy for ≥12-year-old is 91% from 7 days after 2nd dose to median 4 months." (I explained in a previous post why I don't think it is appropriate to say 6 months.)

***

Did we just prove that the vaccine works just as well for 12-15 year olds? It may seem so but this is an example of what I'm calling "tokenism" in statistics.

In order to have a statistically significant result for teenagers, we must have enough samples of participants between 12 and 15 years old. The teenager trial did not include enough people, which was why the researchers pre-specified an immunogenicity outcome (antibody levels) as the primary endpoint.

Adding the 2,000 teenagers to the study with 44,000 adults does not produce more teenager outcomes. In fact, it's such a small portion of the aggregate sample it's not likely to have moved the aggregate result. What's more, because of the later enrollment dates, almost no one in the teenager group has reached 4 months post 2nd dose so their effect on the cumulative case curves is strictly limited to short-term efficacy, the part of the curve for which most of the adult participants contributed data.

Nevertheless, after merging the two age groups, the researchers are able to phrase their finding as "Vaccine effiacy for ≥12-year-old is 91% from 7 days after 2nd dose to median 4 months." This is not a false statement. It is tokenism.

To see tokenism clearly, I find it useful to speak facetiously. If I find 2,000 cadavers and inject half of them with the vaccine, and then add this group to my dataset of 44,000 living people, now I can say my vaccine works for both living and dead!

***

In a previous post, I explain why the headline VE result "from 7 days after 2nd dose to up to six months after 2nd dose" is better interpreted as "up to about 4 months after 2nd dose". This is another example of tokenism.

The published statement paints the picture that the researchers have data for participants up to six months after 2nd dose - one presumes that even if not all participants have six months of follow-up, most would have. Yet, inside the paper, they disclosed that only 7 percent of the evaluable participants have been followed up for at least six months, a sharp drop due to allowing placebo participants to get the vaccine earlier than planned.

Tokenism does not require lying. The businesses have hired the minority workers but the hiring has made barely a dent on the aggregate statistics.

***

Tokenism is hard to diagnose unless the consumer of the analytical finding has the time and desire to dig into the details of the study. An outcome for >= 12-year-olds on top of an impressive outcome for >=16-year-olds suggests the new study contained a sufficiently large sample of 12-15 year olds. A follow-up period described as "up to six months" suggests that most of the participants have been observed for about six months.

In the same way, a picture of an outing with a couple of token women signals that the company promotes workforce diversity.

 

 

 

 

 

Statistical statements turn on a few words

In the previous post, I started to review the newly released study about the "6 month" efficacy and safety of the Pfizer-Biontech vaccine.

This post examines the central efficacy claim of the paper. Its interpretation turns on a few words.

The paper starts with the finding that "VE against COVID-19 was 91% ... through up to 6 months of follow-up among evaluable participants..." and ends with the claim that "the data in this report demonstrate that BNT162b2 [Ed: the name of the Pfizer vaccine] prevents COVID-19 effectively for up to 6 months post dose 2..."

On the surface, this sounds like a straightforward extension of the interim analysis published in December 2020 that led to the emergency use approval of the Pfizer-Biontech vaccine. That study generated the now-famous VE of 96%. At the time of the study, roughly (but not quite) 50% of the participants have been tracked for two months after their second doses. Fast forward four months. Using exactly the same methodology, pre-specified in the trial protocol, the scientists updated the VE number "up to 6 months of follow-up", resulting in a revised VE of 91%, so there have been negligible attenuation of the vaccine's power.

That's not quite 2what happened, though. Strictly speaking, the quoted statements describe the collected data. However, the meaning of these statements turn on a few words. I'll first focus on why they said "up to 6 months" rather than "6 months".

(In a related post, I explain why when I read journal papers, I don't make assumptions of what the researchers may or may not have done. I focus on what they commit to paper.)

***

This headline result is on line 4 of the following table:

The famous VE of 96% was based on a case counting window that runs from 7 days after second dose (2D+7) to mid-November, when just under half of the study population have reached two months post second dose. Recall the train metaphor from the last post: the participants are seated in different carriages based on the dates of enrollment (embarkment).

So what's new in this study? The head of the train has moved from 2 months to 6 months of follow-up while the rear of the train also moved 4 months forward so almost all participants should have reached the 4-month mark, meaning their case history contributes to the case rate from 2D+7 up to 2D+4 months.

Given the focus on the "6 month" point by the investigators, I was expecting to find the words six months in this table. They are not there. The closest thing we can see is >= 4 months, which is the last line of the above table.

The reason why six months can't be found is that a proper six-month efficacy analysis has become impossible when the FDA allowed the vaccine developers to offer the vaccine to everyone in the placebo group. In the paper, this is known as the "vaccine transition option". Between December 2020 and March 2021, all but ~2,000 participants jumped off the train... before they arrived at the six-month marker. Once they are off the train, there is no re-boarding so no further updates on "long term efficacy" is possible.

***

The paper glosses over how the vaccine transition option totally changed the game. I'll now fill in the details. The following chart shows how the transition option caused follow-up times to be cut short across the board.

Without the option, as the Pfizer trial was originally designed, we should have 55 percent of the participants reaching 6 months or longer after second dose by March 2021, with all but two thousand odd people exceeding 4 months of follow-up. We should have had solid evidence on efficacy "up to 6 months".

With the transition option, the train was stopped, and almost all of the 44,000 passengers exited. What's more, the conductor instructed passengers to use exits at the back of the train as the doors near the front would not open. As a result, the proportion with at least 6 months of follow-up crashed from 55% to 7% while the number with fewer than 4 months of follow-up doubled. The vast majority of participants (about 65%) left the train with >=4 and <5 months of follow-up.

What the investigators meant by "up to 6 months" is really "four to seven months," with 80% of "evaluable participants" with fewer than 5 months of follow-up time. The last row in that VE table labeled ">= 4 months" is really better described as "4 to <5 months".

***

Those who decided to stay on the train for 6 months or longer are not your regular passengers - they consist solely of people who do not want to know if they were given the vaccine or the placebo, and so for some reason, they don't care if they are inoculated with a 96%-efficacious vaccine during a deepening pandemic. We don't know the psychology of those individuals but we are pretty sure they are not the average person.

This is another reason why the numbers on the right side of the case curve (>= 5 months) are neither reliable nor valid. If they show the error bars around those rates, they will be huge. If they conduct a survey, they will possibly learn the reasons why most exited the train but a few remained on it.

***

Elsewhere in the paper, the case-counting window was described as "from 4 months to the data cut-off".

If this were an ordinary clinical trial, these words suggest that the data were censored at the time of the interim analysis. Censoring is a technical term that signals the curtailment of the follow-up period. Common reasons for censoring include the participant reaching an endpoint (once someone gets infected, the outcome is known, removing the need to keep testing the person for infection), or the participant has dropped out or gone missing.

A different type of censoring is triggered by performing an interim analysis. At the full analysis, each participant has amassed the required follow-up time and every participant can be analyzed for the identical amount of follow-up. At interim analysis, the train is stopped at the half way point, so that the early enrollees have been observed for more time than the later enrollees. Every participant who has not previously been censored is now censored through an act of the investigators'.

Interim analyses have become common in clinical trials and so, censoring by "data cut-off" follows. But what happened in the Pfizer trial is not censoring by data cut-off. It is censoring by the vaccine transition option. Study participants censored themselves by exiting the train. This is a type of "informative censoring," which requires bias correction (not discussed in the paper).

March 13, 2021 was the data cut-off date for this study. In a standard interim analysis, censoring by data cut-off means anyone who hasn't been censored for some other reason will have their follow-up period stopped on March 13. When the censoring is due to the vaccine transition option, each individual has a different end date - the date on which they accepted the transition option. Almost everyone who accepted did so before March 13. Thousands exited in December 2020 alone.

The use of "4 months to data cut-off" creates the impression that the censoring occurs just like any interim analysis that stops follow-up on data cut-off. In reality, almost nobody was censored on data cut-off date; they were censored before data cut-off date, when they were told whether they received the vaccine or placebo.

***

In this post, I covered two examples of statistical statements that turn on a few words. "Up to 6 months" and "to data cut-off" don't mean what we might think they mean. That's because the vaccine transition option was not anticipated or pre-specified.

In summary, the six-month study is primarily about the 4th month after 2nd dose. There is some decline in efficacy but not large. The current estimate is less reliable than the previous analysis because the vaccine option curtailed follow-up time across all participants. Besides, we are unable to learn about efficacy beyond four months - that is the cost of giving the placebo participants the vaccine. I also have doubt that the people who stayed on board are representative of the average participant.

 

P.S. [8-9-2021] Here are my other posts about this study: post 1, post 3

How to count the fare after letting most passengers leave the train

We don't live in normal times and so the vaccine trials have been accelerated in order to deliver the promise of stemming the pandemic.

What does a normal clinical trial look like? In the most fundamental setting, you recruit participants, screen them for eligibility, randomly split them into two equal groups, give one group the medicine being tested, and the other group a placebo, wait a pre-determined follow-up time, and when the trial ends, you tally the outcomes in both groups and compare them.

In designing and running randomized controlled trials (RCTs), every effort is taken to make sure that the two groups of participants are identical and identically treated so that any difference in outcomes can be attributed solely to the medicine (vs placebo) and nothing else.

***

One way to accelerate trial results is to conduct an interim analysis. In the simplest setting, one conducts an analysis half way through the trial. This procedure assumes that the first half of the trial is predictive of the second half. It also assumes that the sequence of enrollment is random. 

Think of participants as boarding a train from the front to the rear. During a normal follow-up period, the train moves a fixed distance forward.  Passengers can only alight at a fixed disembark point on the platform, so at the end of the journey, the rear of the train has stopped there. In other words, every participant is tracked for the same amount of time, say one year.

For an interim analysis, the train is stopped midway to its destination. An inspector boards the train to check everyone's documents. The people at the front of the train has traversed a longer distance than the people at the back of the train. In statistical jargon, we say that the outcomes are "censored". 

At interim analysis of the vaccine trials, someone may have just received their first shot a week before the date of analysis, and so the investigators have followed them for one week only - and they can only contribute to the case counts up to 1D+7 days (7 days after first dose).

If you look at a cumulative case curve like this at interim analysis, imagine the front of the train running from left to right from Day 0 to Day 238. The interim analysis stops the train. The passengers at the front of the train have moved a further distance to the right than those who boarded last. The number of participants who can contribute to the case counts decreases as you go along the time axis from left to right.

As I have pointed out before (link 1, link 2), most vaccine trials have some form of staged enrollment, usually recruiting lower-risk participants first, and so the assumption of random sequencing of enrollment fails. These are the types of biases that appear when one deviates from the standard design. Investigators should correct for this and other biases.

At final analysis, all passengers have exited the train at the same disembark point. That means all participants have experienced the entire timeline. So the bias correction is only necessary at interim analysis but not at final analysis.  

A different adjustment - unrelated to bias - is that the bar for statistical significance is raised at interim analysis to account for additional uncertainty due to shorter follow-up. (Nevertheless, statistical significance has become collateral damage of the pandemic.)

***

That is a long setup for the topic of today's post, which is the "6-month" analysis of the Pfizer vaccine trial. A preprint of the scientific results has finally been published last week (here). The report is notable not just for what it says but what it doesn't say. My goal is to help fill out the details so you can digest the analysis with full context.

The first thing I noticed on the summary page of the report is that the investigators carefully inserted the words "up to", which includes the possibility that only a small proportion of people have reached 6 months of follow-up after the second dose. 

Enrollment in the Pfizer trial took place between end of July and end of October of 2020, and this interim "6 month" analysis froze the data on March 13, 2021. At this point, the head of the train is at about 7.5 months out while the rear of the train is at 4 months so we know for sure not everyone has reached 6 months of follow-up.

A key sentence in the paper discloses that about 55% of the participants have reached 6 months following their second doses: "During blinded and open-label periods combined, 55% of BNT162b2 [Ed: name of the Pfizer vaccine] recipients had ≥6 months follow-up post-dose 2."

It's important to realize that the above sentence did not state simply that "55% of trial participants had ≥6 months follow-up post-dose 2." The wording makes all the difference!

They didn't say the simple sentence because it's not true. The statement would have applied under normal circumstances but we live in extraordinary times. Since December, when Pfizer obtained EUA for the vaccine, a steady stream of placebo participants have been given the vaccine - they have effectively disembarked from the train before reaching the destination (link). By Pfizer's own publicity (link), they aimed to offer all placebo participants the vaccine by March 1, two weeks before the analysis date. 

As a result of the early-than-planned exits, only a fraction of the original placebo arm had ≥6 months follow-up post-dose 2. How many remained? How about 6%? This fact was disclosed in the adjoining sentence: "6% of placebo recipients had ≥6 months follow-up post-dose 2".

This is the data challenge that the analysts faced due to the decision to dismantle the placebo group. By design, 55% of the placebo group should have had 6 months of follow-up by March 13 but because of the "vaccine transition option", the number reaching that goalpost shrank from 55% to 6%... from about ~10,000 to ~1,000. To use the terminology from above, 9,000 or so persons have been censored at earlier dates so they have zero impact on the later part of the case curve. The later part of the curve is where we find "long-term efficacy".

For the interim analysis published in December, 2020, the FDA required a median follow-up period of two months following the second dose. Fast forward four months, and the middle of this train should have reached six months, triggering the six-month update. And yet, this is just theory that ignores the vaccine transition plan for the placebo group. In reality, the six-month mark is at the 6th percentile of follow-up times, not the 50th percentile. Nevertheless, passengers who left (and received the vaccine) are not allowed to re-board the train, and thus conducting the analysis later will make no difference.

Tellingly, in the VE table (Figure 2), the last line computes VE >= 4 months after dose 2. That's strange when the study is used to confirm "six month safety and efficacy." 

***

As an analyst, the crisis you're facing is that the vaccine transition option effectively destroys the careful RCT design, turning the data into observational ("real-world") data. This means the key to success is bias identification and correction. 

Unfortunately, that's not the path taken. The paper makes no mention of how many placebo participants chose to receive the vaccine, and when they did so. Timing matters a lot. It does not provide background statistics comparing those who transitioned, and those who didn't, nor any differences between those who transitioned earlier relative to those who transitioned later.

What the investigators did instead is to devise a mechanism by which they subtracted people from the vaccine arm to match the censoring of the placebo arm. The paper spent almost no time on analytical methods. We do have this sentence as a clue: "During the blinded period, 51% of participants in each group had 4 to <6 months of follow-up post dose 2; 8% of BNT162b2 recipients and 6% of placebo recipients had ≥6 months follow-up post-dose 2."

For placebo participants, the blinded period is before they were told they got saline shots during the trial and given the vaccine. The people in the vaccine arm were apparently also sent the notices (this wasn't described in the paper), and most of them responded to learn they were vaccinated.

The investigators then censored everyone in the vaccine arm who responded, meaning they de-boarded the train. That is the reason why the proportion of people in the vaccine arm with over 6 months follow up also plunged - from 55% to 8%.

The massive exodus from both arms of the trial appeared numerically similar but the reasons for exiting were night and day. Pfizer said it would be unethical to hold out their 95% efficacious vaccine from the placebo participants for the sake of science. In order to give them the vaccine, they must first be unblinded, i.e. told that they were assigned the placebo shots before. This reason does not apply to the vaccine-arm people as they have already taken both shots.

With the unblinding of the vaccine arm, the researchers threw out a lot of fine data. Instead of estimating the 6-month case rate of the vaccine arm using ~10,000 people, they did so with 10 times fewer participants. Students learn in Stat 101 that sampling error is a function of the square root of the sample size: to halve the error, you need 4 times the sample size. Reverse this. If you reduce the sample to a quarter of its size, you double the error. If you reduce it 9 times, you triple the error. That's one way to quantify the price of the vaccine transition plan. (The dismantling of the placebo arm is inevitable but it is the investigators' choice to inflict it also on the vaccine arm.)

I'd have taken a different path. The VE is based on a ratio of the case rates of the vaccine arm and the placebo arm. By preserving the original sample on the vaccine arm, I would have one good rate estimate (vaccine arm) and one bad estimate (placebo arm). In their analysis, they used two bad estimates. 

***

The challenges of observational data do not end there. It's not just about the size of the sample. It's also about the quality.

The participants who stayed on the train are unlikely to be a random sample of the original population. What would you choose to do if you were given a chance to confirm that you have been given the vaccine, and if not, to receive a vaccine that attained a 95% efficacy in the middle of a worsening pandemic? I can only guess at why a small number of participants decided not to unblind themselves. Maybe they already figured out their status. Maybe they think they don't need a vaccination, perhaps because they have lower risk, or they don't care if they are vaccinated or not. No matter what, these are the people who populate the right side of those cumulative case curves which speak to long-term efficacy. It's hard to believe that the ones who stayed on the train are identical to those who left.

I have more to say about this study. The above provides a basic understanding of why this is no ordinary study.

P.S. [8/9/2021] Here are the other posts: post 2, post 3

Every headline produced by real world studies has been retracted and rewritten

Back in March, I reviewed several early "real-world studies" of vaccine effectiveness (here, here, here, here), including the Dagan et. al. study from Israel that eventually became one of the most frequently cited studies on the unreasonable effectiveness of mRNA vaccines.

The researchers of these early studies acknowledged the difference between observational datasets, and data coming from randomized clinical trials; specifically, the problem of selection bias inherent in the former but obviated by the design of the latter. Matching methods were deployed to reduce the effect of selection before generating a vaccine effectiveness number that mirrors vaccine efficacy in an RCT.

I also later reviewed several studies (Denmark, CDC, CDC 2) that utilized regression adjustments. Regression adjustment is an exercise in wishful thinking: for example, an age-adjusted estimate of vaccine effectiveness is the value of VE after removing the effect of age. Mathematically, the regression adjustment imposes a common age distribution on the vaccinated and unvaccinated subgroups, so age becomes a non-factor.

However, the age-adjusted regression estimate is relevant only if
(a) the estimate of VE for any given age range is reliable, and
(b) the specific age distribution imposed on both subgroups is realistic.

In the present situation, it is hard to believe either of those assumptions. For vaccination campaigns everywhere prioritize high-risk subgroups, particularly older people with or without comorbidities.

The headlines coming out of those early real-world studies were loud and clear: real-world evidence confirmed that the vaccines performed as well as and maybe even better than the clinical trials indicated. The media highlighted VE estimates of over 90% on all reported cases of Covid-19.

As a reminder, this was a CNN headline from February 2021.

The first sentence of the article cites the 90% number: "Pfizer-BioNTech's Covid-19 vaccine appears to reduce symptomatic coronavirus infections by more than 90% in the real world, Israeli researchers said Sunday."

If you want the other view, Fox News said the same thing.

***

By end of May, amidst reports of breakthrough infections, the story started to evolve. The new headlines now focused on VE estimates of close to 100% on reported severe cases, hospitalizations, or deaths. Readers were admonished for ever believing that vaccines should prevent disease, even though the primary endpoints for all clinical trials and early real-world studies were prevention of symptomatic, PCR-positive Covid-19 cases in the selected case-counting window.

Since June, the U.K., despite having one of the world's fastest vaccination programmes, began to suffer another surge in cases. The new headlines now drew attention to hospitalizations rather than cases.

The report cites a Public Health England study, saying "although the Delta variant reduces the effectiveness of vaccines against symptomatic infection, two doses of COVID-19 vaccine still protect against severe disease."

Israel, the early star in vaccinations, faced a similar scenario, and their reporters followed the same path, delivering headlines such as this:

It's not about infections anymore; only hospitalizations matter.

It's not just the headlines that changed. This new batch of studies employs different methods of analysis. Matching and regression adjustments were both sent to the scrap pile. The new analyses now wantonly ignore selection and other kinds of biases, and treat the observational data as if they were collected from randomized clinical trials. They assume the vaccinated and unvaccinated groups are balanced on all variables, an assumption not remotely plausible.

Here is an example of such a calculation, lifted from Matan Holzer's twitter account (see title of bottom chart):

In these type of studies, vaccine "efficiency" for hospitalizations is based on the ratio of the odds of hospitalizations for vaccinated vs unvaccinated to the odds of vaccination. The same ratio is obtained by dividing the hospitalization rate of the vaccinated by the hospitalization rate of the unvaccinated. This is the analysis that would have been performed to evaluate VE in a clinical trial - although those trials did not contain enough participants to attain statistical significance on the hospitalization metric.

Such analyses gave rise to headlines that proclaim that only unvaccinated people are sick enough to go to hospitals.

What are the data supporting the above headline (link)? It's the share of hospitalizations that are attributed to the unvaccinated people.

In the meantime, the following chart from BBC shows how the new wave of infections is  growing even faster than in the last surge, which happened before vaccinations.

***

By mid July, hospitals in the U.K. and Israel are admitting a rising number of fully vaccinated patients. So, the story is shifting again.

The previous metric comparing vaccinated and unvaccinated groups as if they were randomly assigned is abandoned. Experts now pretend that they did not introduce this analysis to the public, and criticize those looking at this number as a probability illiterate - failing to grasp Bayes' Rule. (here) Nate Silver chimed in:

I agree that the efficiency metric is poorly conceived, a fact that was as obvious in May as it is in July. I don't like the "story-first" mentality adopted by many scientists, which led to all those headlines about almost no vaccinated people getting hospitalized.

As the hospitalization story falls apart, the media is lining up behind the VE metric on deaths. We are being told that the vaccines have broken the link between infections and deaths, which reminds me of when they said the vaccines severed the link between infections and hospitalizations.

The math has been clear on this from the start: hospitalizations lag cases, and deaths lag both. We are going to find out in the coming weeks whether this latest narrative sticks

We need a post-mortem on the real-world studies, a honest evaluation on why these findings have low predictive value.

 

 

P.S. [7/23/2021] The same story is unfolding in Israel. Weeks ago, they happily cited naive estimates of vaccine effectiveness - first on infections, later on severe disease. Now that those metrics don't show what they used to show, suddenly those metrics are worthless. This Times of Israel article is highly informative.

Real-world vaccine studies through the lens of vaccine trials

Real-world studies are hard because observational data are packed with known and unknown biases. One thing is for sure: any analysis that doesn't correct for known biases is clearly flawed.

When discussing real-world studies of Covid-19 vaccine effectiveness, I have often lamented the ill effect of "case-counting windows". Recall that none of the vaccine studies count all cases from the day the participants enter the trial and get their first shot. In the case of Pfizer, all Covid-19 cases that occurred prior to the 7th day after the second dose were removed, in effect, nullified.

This procedure is feasible in the vaccine trial, when the same case-counting window can be applied to both vaccine and placebo arms. In real-world studies, the unvaccinated people do not take placebo (saline) shots, and therefore, it is impossible to locate 7 days after the second shot. As a result, the case-counting window removes cases only from the vaccinated group and not from the unvaccinated group.

tldr; Scroll to the end of the post

***

I'll show you the effect visually. We start with the cumulative case curve that was displayed in the Pfizer FDA Briefing document.

This chart is straightforward. As time passes, the number of cases grew on both vaccine and placebo arms. At the time of the interim analysis (which led to the emergency use authorization), the cumulative case rate for the vaccine arm was 0.25%, or 25 out of 10,000 was found to be sick while that for the placebo arm was 1.4% or 140 out of 10,000. Because of the trial setup, we can assume all else being equal, and so the vaccine is said to have cut the case rate by just over 80%. The vaccine efficacy is 80%.

But Pfizer said VE was 95%, not 80%. This is because of the case counting window. All cases on either arm that occurred prior to 2D+7 days are nullified. Like this:

This case-counting strategy knocks down the case rates for both placebo and vaccine arms. The placebo group now has a cumulative case rate of 0.8% at interim analysis, while the vaccine group has a cumulative case rate of 0.04%. The vaccine is now said to have cut the case rate by 95%. (But this number is not comparable to the 80% number above. The 95% efficacy applies only to people who did not get sick prior to 2D+7 while the 80% number applies to everyone in the vaccine trial.)

What happened is purely arithmetic. The second calculation is based on a subset of the data used in the first calculation.

***

Next, we do a thought experiment. We pretend that the data came from a real-world study instead of a clinical trial. We will therefore relabel the placebo group as "unvaccinated" and the vaccine arm as "vaccinated". We will now compute a vaccine effectiveness using the methods from real-world vaccine studies.

Here is the picture:

The cumulative case rate for the unvaccinated group is 1.4% while that for the vaccinated group is 0.04%. Thus, the vaccine apparently cut the case rate by 97% - almost perfect protection, even better than what was seen in the vaccine trial.

What just happened is that I took the orange line (for unvaccinated) from the first chart and the blue line (for vaccinated) from the second chart. The blue line is uncontroversial: given the dates of vaccinations of the vaccinated people, the analyst can compute how many days after the first shot someone got sick with Covid-19, and therefore apply the case-counting window, removing all cases occurring prior to 2D+7. This mimics the formula used in the vaccine trial.

When it comes to the unvaccinated group, the analyst could not apply a case-counting window. That's because the unvaccinated group did not take any injections - unlike the placebo participants in the vaccine trial who took saline shots. As a result, the analyst just counts all the cases in the unvaccinated group during the entire study period. This case rate is then compared to the case rate of the vaccinated - but restricted to various case-counting windows.

The beauty of this thought experiment is that it allows us to estimate how much pro-vaccine bias has been introduced by this case-counting window. On the second chart, we know that if case-counting could be applied to the orange line, the case rate would have been 0.8%. On the third chart, we know that the case rate was treated as 1.4% when case-counting became infeasible. Thus, the baseline case rate is inflated by 70% due to this asymmetric application of case counting.

Without a doubt, these real-world studies produce vastly inflated estimates of the true vaccine efficacy. (And here, I have only covered one of many biases in these studies.)

 

TLDR;

Real-world data are messy because unlike a clinical trial, there is no design to constrain biases. Real-world studies are evalulated on whether they have sufficiently corrected for known biases. One useful trick is to run the real-world study methodology through an unbiased dataset (e.g. data coming from a clinical trial or randomized experiment.) One hopes that the RW methodology yields the same answer as when one analyzes the clinical trial using techniques for analyzing trials.

The analysis here shows that the most popular real-world study methodology over-estimates the effectiveness of vaccines. The reason is that the case-counting window is applied symmetrically in the trial analysis but only to the vaccinated subgroup in a real-world study. This asymmetry causes the baseline case rate to be inflated by 70% (in the case of Pfizer), and vaccine effectiveness is the relative improvement over that baseline.

The case-counting window for Pfizer starts at 7 days after the second dose, meaning that all reported infections happening before that day are nullified when analyzing the clinical trial. In the real-world studies, these earlier infections are nullified only for the vaccinated group but not for the control group. This is a pro-vaccine bias built into real-world data. The only remedy is a correction performed at analysis. A first-order correction is to divide the unvaccinated case rate by a factor of 1.7 before computing vaccine effectiveness. (This correction however does not adjust for many other biases found in real-world data.)

 

Story time: vaccinated people cannot spread Covid19

If you ask me for advice, I say get the vaccine, but do not let your guard down afterwards. Sure, go out more, and re-connect with friends but continue to wear masks and maintain distance in crowded places.

But some of you say, the CDC tells us if one is fully vaccinated, one can treat the pandemic as over. Throw all (pre)caution to the wind.

Let's dissect that recommendation. If you are fully vaccinated, you can still get Covid-19 but they claim you will not get severely sick. They encourage you to return to pre-pandemic living. So they must believe that fully vaccinated people with mild disease cannot spread the virus. They must also believe that fully vaccinated people with asymptomatic infections cannot spread the virus either.

These are important points because half the U.S. population is now fully vaccinated, and if fully vaccinated cases can spread the virus, then potentially lots of people are spreading the virus, invisibly. While we don't get severely ill, the more sick people roam around, the more the virus replicates, and the higher the chance that a rare mutation surfaces that causes another lethal wave of Covid-19.

Starting around April, self-described experts began making claims that "fully vaccinated people cannot spread Covid-19". For example, in this Johns Hopkins piece, an expert said "I think the preponderance of the evidence supports the fact that vaccinated individuals are not able to spread the virus."

***

A month or two ago, I gave a talk to University of Denver students, with the key message being "don't judge a research study by its abstract; learn what research methodology it used to draw its conclusions." See this post for more.

At this stage of the pandemic, three research methods dominate Covid-19 research: clinical trials, real-world studies, and lab experiments.

If we want to prove that vaccinated individuals are unable to spread the virus, what kind of study do we need?

We can immediately rule out the lab experiment. Testing blood samples for antibodies will never yield data on clinical outcomes such as symptoms, infections, transmission or spread.

Spread is a very hard outcome to assess. You need to first identify infected people, and then do contact tracing to figure out the transmission chains.

Can clinical trials yield such data? Absolute not, if we use the vaccine trials as they were designed. Those trials only tested participants for infection once every month or two, and most cases were self-reported, meaning that they only identified a fraction of the infections. Moreover, no contact tracing was attempted so we have no idea who transmitted the virus to whom. Therefore, these trials yielded no data about spread.

What about real-world studies? While we have big sample sizes for these studies, they also contain nothing about transmission chains. The aforementioned Johns Hopkins article cites this CDC study. I fail to see how the data proved that vaccinated people cannot spread the virus. The paper shows that vaccinated people are less likely to be infected (on this, I have concerns about inadequate debiasing of the data), and then they stretch that result to claim that the vaccine stops transmission.

Nevertheless, the study made no attempt to figure out who transmitted the virus to whom. For example, how do we know that one of the vaccinated and Covid-positive participants did not infect a dozen unvaccinated co-workers? I don't know the answer, nor do I think that is the most likely scenario but I fail to find any evidence supporting the claim that vaccinated individuals do not spread Covid19 in that study.

So what we have is "story time". Readers of the study are lulled to sleep by the nice dataset that was collected to measure overall vaccine effectiveness, and as they doze off, the researchers feed them the story of preventing spread, which has little if anything to do with the data.

 

P.S. [7/14/2021] In addition to story time, we have contradictionary evidence that gets ignored or dismissed. For example, the Las Vegas Review Journal just reported that 10 vaccinated health-care workers got Covid-19, possibly all during a pool party. Eight of these are fully vaccinated. Only 1 unvaccinated attendee got sick. So 10 out of 11 cases from that party were vaccinated. 

They only know about these cases because the health care workers decided to get tested. There are probably more cases because many attendees may not have gotten tested. Note, however, that fully vaccinated people are more likely to ignore mild symptoms and not get tested.

If fully vaccinated, PCR-positive people cannot spread Covid19, each of these 10 healthcare workers were infected by an unvaccinated, infectious person or persons. Even in this small example, we don't know who spread the virus to whom.

Regrettably, the Review-Journal printed a fallacy that I have pointed out from day the first clinical trial result was published. It said "In laboratory tests, the Pfizer and Moderna vaccines were about 95 percent effective in preventing symptomatic cases of COVID-19. That means that about 1 in 20 people could become sick after being vaccinated, though their infections would be less likely to be severe." In these trials, the case rate in the placebo group - people who got saline shots - was only around 1 percent. If their interpretation were correct, then the vaccine bumped the chance of getting infected from 1 percent to 5 percent - 5 times worse! See this post for how to interpret vaccine efficacy.

If you're not counted, you don't count

If you're not counted, you don't count.

The first law of counting is to count everything, and I do mean everything - not just the things you think you care about.

***

Imagine - as I did in Chapter 1 of Numbersense (link) - that a college advertises that 95% of its graduates found jobs within 6 months of graduation. It turns out the school collects data from a post-graduation survey - it's not disclosed anywhere in the marketing materials that it's 95% of survey respondents, not 95% of graduates. What's also not reported is the survey's response rate, which was 20%.

Nevertheless, the college administrators stand behind their statistics. They say the respondents and non-respondents did not show statistically significant differences in their gender distribution, average age, and type of degree; therefore, they assume that non-respondents also have a 95% employment rate within 6 months.

Do we think graduates who failed to land jobs have the same chance of filling out an employment survey as those who are gainfully employed after graduation? You can draw your own conclusion.

If you only have biased data on 20% of the graduates of this school, do you think you have a full understanding of the employment prospects of its graduates?

***

At the start of the Covid-19 pandemic - think March or April 2020, the U.S. suffered from a shortage of diagnostic tests. The official policy at the time was to test only people with severe symptoms.

We knew next to nothing about the novel coronavirus at the time. The data on cases became the basis of our knowledge - not only of the severity of the pandemic but also its trends, its demographics, its symptoms and injuries, and so on.

But triage testing leads to biased data. When the pandemic also grows quietly through asymptomatic transmission, our data led to a partially blind, misleading assessment of the situation. I warned about this in a Wired column - subsequently, colleges have shown that comprehensive testing is crucial to controlling the spread of Covid-19. Random testing of small samples of the population is a viable alternative we can still implement today to monitor new variants.

***

A curious thing happened during the pandemic to our unemployment statistics. Eight million Americans not only lost their jobs overnight but suddenly lost all interest in working. That's according to official statistics, which showed that the number of people in the "civilian labor force" dropped by 8 million.

People "not in the labor force" have always been a strange species. The government claims that these people either do not want work at all, or have not been looking for a job for an extended period of time. Their unemployed status is ignored in the unemployment rate. Oddly, as more people drop out of the labor force, the unemployment rate improves, all else being equal. This group is simply not counted; they are neither employed nor unemployed.

If one has the responsibility of reducing the unemployment rate, one will focus on unemployed people who are counted in the data, not the ones who are not counted. When you are not counted, you don't count.

In Chapter 6 of Numbersense (link), I explain how the government computes the unemployment rate in detail. Not counting a group of unemployed people hides some of the bad news. As I pointed out in this previous blog post, the sharp exodus out of the labor force at the start of the pandemic proves the serious shortcoming of the concept. None of the people who just lost their jobs due to Covid-19 are people who do not want work at all; their recently-ended employment proves otherwise. There also was not enough time that passed to qualify them as having given up on a job search.

***

The Bureau of Labor Statistics (BLS) does count everyone, including data on how many people have been removed from the labor force tally. These data just show up in an appendix, with much less attention and care.

We can't say as much about how the CDC counts Covid-19 cases in 2021. As this New York Times story disclosed, the CDC "stopped investigating breakthrough infections among fully vaccinated people unless they become so sick that they are hospitalized or die."

Just as BLS defines unemployed people as not really unemployed if they "don't want to work", CDC defines infected people as not really infected if they are fully vaccinated and are not hospitalized for severe Covid-19 symptoms. In this way, our data have come full circle, back to the same sad state that it was in March or April 2020. It tells us only about severe cases. To the extent that new variants spread through asymptomatic infections, we are as blind to that possibility as we were a year ago.

Furthermore, the change in counting rules upsets any trend analysis. Consider a real-world study of vaccine effectiveness. If the researchers report that the case rate of fully vaccinated people is much lower than the case rate of unvaccinated people, is it because of the vaccine or because we count mild or asymptomatic cases for unvaccinated people but not for vaccinated people?

It's not just fun and games with counting. Don't forget: infected, hospitalized, or dead people who got sick before 14 days are also treated as not really infected, hospitalized or dead. When you're not counted, you don't count. The CDC turns a blind eye to them because they happen to catch the disease prior to reaching so-called full vaccination status. Why do their lives not matter? That's something I have been wondering about throughout the pandemic, and I still don't have a good answer.

What do we know about the Sinopharm vaccine? 2

The other day, I posted some notes about the JAMA paper featuring the interim analysis of the Sinopharm vaccine trial, a Covid-19 vaccine developed in China using the tried-and-true inactivated virus technology. The trials were conducted in UAE and Bahrain and the follow-up period for the median participant has reached 3 months since the second dose, which is a longer follow-up period than the earlier trials.

One takeaway from Part 1 and also my previous reviews of vaccine trial results is that vaccine efficacy (VE) numbers are not easily comparable. Trials were conducted in different countries at different times; definitions of cases were not standardized; case-counting windows varied. The duration of follow-up also matters because we expect the effect of any vaccine to wane over time.

***

In Part 2 of my review, I explore eFigure 1 in Supplement 3 of the JAMA report. This extremely important diagram traces how "cases" are qualified to be counted in the VE metric. All vaccine trials - not just Sinopharm - count only adjudicated, lab-confirmed, symptomatic Covid-19 cases.

The final VE metric in the Sinopharm trial included only 142 cases across 2 treatment arms and 1 placebo arm. e eFigure1 (partly shown on the right) revealed that there were 962 suspected Covid-19 cases (with symptoms) at the start of the process. That number was cut by 50-60% to 420 based on PCR testing negative.

The adjudicators then used their judgement to remove another 30-50% of the symptomatic, PCR-positive cases. We have 255 cases standing at this point.

Then the case counting window is applied, and another 113 cases vanished from the count. That leaves 142 cases that ultimately entered the VE calculation, resulting in the headline number of 70-80%.

For the sake of argument, if we got rid of the adjudication step, then the VE would have been reported as 37-56%. Because this calculation includes cases from first dose, compare it to 50-66% that was mentioned in the previous post. Thus, adjudication increased VE by roughly 10%.

Adjudication is not a unique feature of the Chinese trial. All vaccine trials employed adjudicators. The other pharmas have been far less forthcoming about what happened during this process. Pfizer, for example, disclosed to the FDA that they had 3,410 suspected cases (link) but I have not found any details on how that number got winnowed down to 170 cases actually counted in the headline 95% VE number.  

***

If you have the stomach for it, I am about to open another can of worms. This happens to be a very material issue common in many settings. When comparing two numbers, one can compute a ratio (sometimes called an index) or a difference. Ratios are very tricky to deal with, and easy to misinterpret. The VE is a ratio of case rates (think of it as a relative risk index).

In the Chinese trial, the headline VE was reported as 73-78%, based on 142 adjudicated cases of Covid-19 across the three arms. In one of the sidebar analyses, the scientists added "asymptomatic" cases, defined as cases found by PCR tests for which the participants did not report experiencing qualifying symptoms.

The following table shows the data disclosed in the report. You can see that the vaccines are clearly not as effective at reducing "asymptomatic" cases, as the differences between 16, 10, and 21 are not statistically significant.

In order to simplify the calculations, I will work with the following idealized example. I'm assuming 10,000 people in two arms. The vaccine arm counts 35 cases of which 20 are symptomatic. The placebo arm has 115 cases of which 100 are symptomatic. We can compute the vaccine's share of total cases to be 23% so that the corresponding VE is 70%. (In this calculation, I include asymptomatic cases.)

An alternative efficacy metric is to compute the difference in the case rates. This means 1.15% (115/10000) for placebo minus 0.35% (35/10000) for the vaccine, resulting in a difference of 0.80%. This number has a straightforward interpretation: for every 10,000 people vaccinated, 80 fewer get sick from Covid-19 relative to placebo.

What happens if we ignore asymptomatic cases from both arms as the pharmas did? The difference in case rates remain at 80 per 10,000 because we remove 15 from each side. This makes sense intuitively since the vaccine is assumed to be ineffective at reducing asymptomatic cases (15 vs 15). Therefore, whether we count symptomatic cases or all cases, the vaccine reduces the number of cases by 80 per 10,000 relative to placebo.

As a relative risk ratio, the VE actually increases from 70% to 80% after removing asymptomatic cases. Note though that the 70% reduction is off the total case rate while the 80% reduction is off the symptomatic case rate, which is a smaller number. This points to the challenge of interpreting VE as a ratio. All the different VE ratios based on different inclusion and exclusion criteria have different base rates and so they cannot be compared to each other! Eighty perent is not always better than 70 percent: 70 percent of a big number can be more impactful than 80 percent of a small number.

You can now see why the game of the vaccine trials is to restrict the VE calculation to regions where the vaccine performs well. (I'm using "regions" in an abstract sense which includes the geographical sense.) Such restrictions lead to a higher ratio reduction but the result is only relevant to those regions, usually where the base rate is lower. In other words, it's a conditional number, as I explained in the last post. It's a higher reduction off an often unintuitive, restricted base rate. The case-counting window is the same game as asymptomatics but with a different name.

***

Another frequently ignored issue in these vaccine trials is the weak blinding. The Chinese trial is actually double-blind whereas the Pfizer, Moderna, etc. are single blind (very commonly mis-reported as double-blind because medical reporters just assume all clinical trials are double-blind).

Blinding is a protective measure that prevents investigators or participants from behaving differently - conscious or not - because they know which arm individuals are in. When the placebo is saline solution, it's actually very hard to blind participants because those getting vaccines are much more likely to experience a variety of symptoms, such as numbness, headaches, mild fevers and so on.

As a result, a few designs use non-saline placebos. The Chinese trial used an adjuvant in the placebo. An adjuvant is an additive in the vaccine that helps it work better. It's unclear to me whether this placebo is preferred to saline. It depends on the chance that the adjuvant causes mild side effects similar to the vaccine (which is a good thing in terms of masking treatment status) and the chance that the adjuvant by itself could be beneficial (which makes the base rate higher and raises the bar for the vaccine). 

The only other exception I know of was one of the Astrazeneca trials, in which the placebo was a different unrelated vaccine. Participants might be less able to guess which arm they belong to in that trial. 

***

Finally, none of these trials is really serious about measuring asymptomatic cases. The only exception is again in one of the Astrazeneca trials: they actually asked participants to send in nasal swabs every week of the trial. In the typical vaccine trial, they only do PCR testing (a) if symptoms are self-reported and (b) at infrequent intervals, including before each shot, and then once a month or longer. With infrequent surveillance, only a fraction of the asymptomatic infections can be found.