Big Data, Plainly Spoken (aka Numbers Rule Your World)

The wish that won't quit

 

It amazes me how the media keep inviting experts that push their gut feelings by cherry-picking only the science that confirms their beliefs. I am talking about the wishful thinking that is single-dose vaccines.

If you believe these experts, the evidence is strong, verging on undeniable, that we should skim on the Pfizer, Moderna and Astrazeneca vaccines and only administer one dose. They have most often cited the calculation by the U.K. government which I discussed here. This same argument was parroted by two Canadians in a more recent push for one dose.

The arguments for sticking with what we know are being buried. Here is an excellent summary of the main points. I covered all of these earlier than most in this blog post.

Since that time, there have been many studies that bolster the case for sticking with two doses. You've heard about these studies except they buried the lead in each case.

***

We recently heard there is a study that showed half doses of Moderna might be enough. Such a study did exist, it's an interim analysis of a small-scale randomized trial - but the big story in that study is that the antibody levels did not exceed that obtained through natural infection until after the second doses. I wrote about that study here.

We recently heard about a study in which the Mayo Clinic found 89% efficacy in their real-world study. Great news. What the media experts did not tell you is that this VE was measured from 36 days after the first dose, thus including the effect of the second dose. If measured from first dose, the VE is 60%. Therefore, real-world evidence shows that the VE of a single shot is nowhere near the 90% claimed by these experts.

We recently heard of a study involving Israel's largest healthcare provider, verifying that the Pfizer and Moderna vaccines have real-world efficacy similar to those achieved in the clinical trials. Great. But did the media experts tell you what else they found? The vaccine effectivness at 14 days after the first dose is in the 40-60% range depending on how they defined cases. It was not until after the second shot where effectiveness edged towards 90%. Therefore, real-world evidence shows that the VE of a single shot is nowhere near the 90% claimed by the experts.

***

What I wrote above are directly lifted from those scientific papers. It does not require any interpretation or analysis. I'm just relaying what the researchers said in those papers. The media and their experts are filtering out what don't fit their worldview, and you're worse off for it.

Of course, I have taken notes while reading those studies so in a future post, I'll provide an evaluation of these real-world studies.

 

 

 

How to rise above the cacophony of sometimes phony data analyses

Data analyses are flooding the airwaves. They paint a confusing, contradictory picture of all aspects of the pandemic. Sadly, most of the work are hastily put together, and low quality. Here are some things to keep in mind when you look at these studies.

What happened in an experiment will not happen again

If a vaccine trial returns a 90% vaccine efficacy, we can say for sure the vaccine’s efficacy is precisely 90% for the 30,000 or so participants in that specific vaccine trial. We are not empowered to conclude that the VE for billions of people who will eventually get the shots will be 90% - no matter how many epidemiologists repeat this falsehood on TV.

That’s because the trial involved a small sample of people, and there is a margin of error around the 90% number. If the range estimate is 70% to 98%, then we say “with 95% confidence” that the VE for the general population is above 70%. That’s still an impressive number, and it has the advantage of respecting scientific principles.

Imagine repeating the same clinical trial with different choices of 30,000 participants repeatedly. The margin of error says 95% of the series of VE estimates from these trials will fall inside the 70% to 98% range. The chance of any of these trials repeating the exact 90% VE is practically zero.

Credit: Jeremy Schultz/Flickr

 

The randomized clinical trial (RCT) is the gold standard for establishing cause--effect. The lack of randomization in “real-world” studies opens a can of worms.

In a vaccine trial, we compare the case rates of those who are vaccinated to those who aren’t, which are the primary ingredients of the vaccine efficacy formula. We are currently being served weekly new studies that also compare vaccinated people to unvaccinated people. The media are telling us that these new studies are better because (a) they are more recent (b) they have much larger sample sizes and (c) they constitute “real-world” evidence. A popular theme is that these studies fill in the gaps left by the vaccine trials.

The noise you’re hearing are the chuckles from incredulous statisticians. Since RCTs are regarded as the gold standard for causal inference, no “real world” evidence should be modifying, and certainly not correcting, findings from a scientific experiment. Doing so is like hiring a C student to tutor an A student, and siding with the C student when they disagree.

There is one critically important property missing from all real-world studies: the randomization of treatment. (See, however, so-called natural experiments.) In an RCT, a coin flip determines who gets the vaccine but in the real world, who gets the shots is anything but random. Most countries have priority lists and specific types of people are getting the vaccines earlier than others. In any “real world” study, the unvaccinated group is different from the vaccinated group, not just by vaccination status but also by many other factors, both known and unknown. Any of those other factors could contribute – majorly or minorly – to the observed difference between the two groups. Randomizing treatment ensures that on average, these other factors will not bias the finding in an RCT, a condition that does not exist in real-world studies.

A partial solution is to define a better control group. We’d take a subset of the unvaccinated people who look like those who are vaccinated. This is an effort to manufacture artificially the randomization condition. This matching process is highly subjective, and we can only match people using measurable and influential factors. There is no law that dictates that every important factor is measurable. As an illustrative example, perhaps people with higher-speed Internet connections are more likely to land vaccination slots. Since medical researchers do not have individual data on people’s Internet speeds, we cannot control for this effect. Even the most obvious adjustments have problems. For example, young people can be excluded because they don’t qualify for vaccination yet but young people who are front-line workers are getting shots.

Another common issue is incurable imbalance: if all care home residents have been vaccinated, we won't find care home residents in the unvaccinated group. The biggest problem with matching studies is lack of transparency. The typical disclosure is vague and confusing – I can’t even figure out what they did. None of these studies publish their data or code.

So, a real-world study that corrects for selection bias is better than one that doesn’t but in no case is an observational study superior to an RCT. Larger sample sizes produce more precise estimates (with lower margins of error) but they do not auto-correct biases. More precise estimates derived from biased data pose even greater risk precisely because they feel more solid when the biases are ignored.

The randomized clinical trial is a victim of abuse.

In reviewing the torrent of studies that have come out of the vaccine trials, I can’t help but notice that analysts are mercilessly abusing the RCT framework. I’m going to echo Andrew Gelman’s criticism of the standards of research in psychology. We’re concerned about the high probability of false-positive findings.

The VE numbers from Pfizer, Moderna, Astrazeneca, etc. are not comparable to each other. That’s because each team measures case rates differently. Pfizer drops all cases prior to 7 days after the second shot; Moderna and Astrazeneca drop cases prior to 14 days after the second shot. Then, you have people reanalyzing the data, who use their own case-counting windows. The U.K. government published a post-hoc analysis of the Pfizer data, counting only cases between 15 and 28 days after the first dose; this is echoed in a Canadian re-analysis of the same data, in which they set the window of 14 days to 20 days. A more recent “real-world” study out of Scotland counted from 28 to 34 days after the first dose. This is no small matter as VE is an improvement metric relative to a baseline, and we can’t keep our heads straight which the baseline applies.

Many scientists justified the choice of when they count cases, on record no less, by pointing to the cumulative case curve, arguing for counting when the vaccine’s curve starts flattening because that’s exactly how long it takes for the vaccine to do its magic. This type of post-hoc thinking is what teachers warn students not to do. These clinical trials were sized to measure the endpoint, and not sized to identify a turning point on a case curve. Imagine repeating this trial many times over. Assuming that the VE after 90 days rises to roughly 90 percent in each trial, it is likely that the turning point on the curve is not precisely at day 14 (or whenever it was in the first trial).

Analysts appear to have looked at their individual datasets, identified the period in which the vaccine showed the best performance against the placebo, defined their VE metric to hone in on that time window, and justified the definition with some sciency blather. This is a recipe for over-estimating the true performance.

The proof is the absence of new studies that go through similar post-hoc thinking to adjust down the findings of the respective RCT. In fact, the re-analyses of the Pfizer data, which moved the estimated VE of the first dose from around 50% to 90% is a nice example of a C student correcting an A student.

To prevent the danger of post-hoc theorizing, scientists agree to pre-register their studies so they have to define how they would count cases ahead of time. Unfortunately, those protocols are written loosely to allow a wide range of options, such as 7 days, 14 days, 21 days, etc. as possible evaluation metrics. This maybe exposes how little investigators know about the behavior of their inventions.

If only the start of the case-counting window were the only “researchers’ degree of freedom”. The researchers also follow their noses about (a) the duration of the case-counting window (b) whether a positive test is required to confirm cases (c) which test is allowed for confirmation and the parameters for running the test (d) the maximum length of time between reporting symptoms and testing positive (e) what symptoms are included and excluded as qualifying (f) how many symptoms are qualifying. As I showed in this post, VE drops fast with just a few more cases on the vaccine arm so any of these decisions matters.

The RCT framework does not protect us against these potential abuses. RCTs are the gold standard, but they aren’t fool-proof.

Story time after RCT

The potential problems outlined in the previous section relate mostly to trial design decisions. The vaccine RCTs are also being suffocated by post-hoc hacking. I already devoted an entire post to this phenomenon of “story time”. Not all results obtained from analyzing data from RCTs have the crucial randomization property. That’s because analysts can destroy the property by cherry-picking the data. We’re lulled into thinking these lesser findings have the same status as a proper RCT finding.

The most infamous example of such analysis is the 90% VE attributed to the so-called low-dose, standard-dose subgroup in the Astrazeneca-Oxford trial (link). This VE is indeed based on comparing case rates of a vaccine arm to that of a placebo arm. In a true RCT, those two arms are statistically the same except for what was placed in the two shots. Nevertheless, the low-dose subgroup happened as an accident, and it subsequently emerged that this subgroup contained only younger participants, have earlier enrollment dates, have longer intervals between doses, etc. And yet, on the front page of the Lancet paper summarizing this trial, the investigators stated “In participants who received two standard doses, vaccine efficacy was 62.1%... and in participants who received a low dose followed by a standard dose, efficacy was 90.0%...” By this point, there has been no disclosure about the “accident,” and anyone reading this sentence is likely to assume that there were two randomized dosage schedules used in this trial.

Likewise, much ink has been spilled on dose intervals. Many analysts have compared subgroups that took their second shots later and earlier. Even though the underlying data were collected in an RCT, dose intervals were not randomized. And yet, these results were published and publicized using methods for analyzing RCTs and not methods for analyzing observational studies. In some cases, the researchers even disclosed that these subgroups differed on numerous important factors, and still they plodded on.

**

In this era, even results coming out of RCTs must be scrutinized. Early analyses, post-hoc analyses, deep-dive analyses, and side analyses typically discard the crucial randomization property, and should be treated as observational studies.

Of course, results from observational ("real-world") studies should be scrutinized even more. In a future post, I’ll outline how to review analyses of observational data.

The slow suffocation of science by data

What we're witnessing in recent weeks is a slow suffocation of science - by the hyperspeed and rulelessness of modern technologies including data and social media.

The dominant themes in pandemic reporting from late 2020 to February have been the holiday spikes, virus variants, from the U.K. then South Africa, vaccinations, forthcoming vaccines, and the dramatic decline in cases.

All sorts of reasons have been concocted to explain the trendlines. So-called experts make passionate arguments mostly based on conjecture built on top of some cherry-picked data.

I took a few minutes to pull together the following chart, so we all can join the cacophony.

Thanks to the team at OurWorldinData for collecting and making the data accessible.

I have chosen a range of countries representing many different experiences. For example, Israel (blue line) has been loudly claiming victory on the vaccination front. The U.K. (pink) likewise has self-congratulated on their vaccination work. There is increasing chatter in the U.S. along the same lines. Meanwhile, Canada (second from bottom) has been criticized for a slow vaccine roll-out. So has the E.U. South Africa (green) also has a slow roll-out, is the epicenter of the much-feared South African variant, and even stopped using the Astrazeneca-Oxford vaccine.

Valid causal factors must explain the trend in every place where the factor is active. Some factors, like the Thanksgiving holiday, are local while many factors, such as vaccinations, are global.

***

After making the above chart, I also checked in on the Swedish experiment:

Isn't it funny that when the trend isn't looking good, the voices disappear? Journalists should be chasing after those epidemiologists and professors to ask whether they repudiate their prior views, or if they have an explanation for what transpired.

The mainstream media have been vocal in stamping out misinformation. And yet, they have given a lot of space early on to this renegade group of scientists. Not correcting the record, and informing their audience that the theory has buckled under the weight of reality, also contributes to misinformation.

***

A sharp drop in cases occurred in Japan starting mid-January, where they only just approved a vaccine (mid February). Here's what happening in Asia Pacific.

***

If you're a business analyst who've been asked to explain the trends in your company's performance indicators, you already know this type of trend analysis is very hard work - you can't just sling things at the wall. Offering explanations for some cherry-picked data is not science; it's a mockery of the scientific method.

If this post makes you question the experts you hear night after night on the news, I'd have succeeded.

 

P.S. [2/22/2021] Right on cue, the New York Times put this article up today. To put it succinctly, the two statements can both be independently true: that the vaccines are helping, and that the cases have recently sharply declined. Two true statements do not a link establish. About the situation in the U.K., they say "Vaccines don’t explain it: Even though a quarter of the population has been vaccinated, only the earliest recipients had significant protection by Jan. 10, when cases there started to drop. Those early doses mostly went to health-care workers and elderly patients already in the hospital."

 

Notes on AstraZeneca-Oxford vaccine trial results 4

I'm not done with my notes on the Astrazeneca-Oxford vaccine trial results. Previous installments are 1, 2 and 3. This post and this post are also both related to it.

***

Trickle-down Biases

When I examined specific pieces of analyses in the Lancet paper, I've previously explained the biases that emerge when we can no longer assume that the groups being compared are randomized. While those details are tasty, we should also pay attention to biases that trickle down from the top.

Firstly, the AstraZeneca-Oxford (AO) trials targeted health and social care workers, which typically account for 60-90 percent of the analysis populations. This targeting makes sense because those front-line workers are at high risk of exposure to the virus, and so we'd like to have good data on the vaccine's effect on them. However, this decision also makes it harder to generalize the results to the rest of the population - simply because health and social care workers as a class is not "interchangeable" with the rest of the population. There are three major differences: a) their workplaces are much more strict about measures such as masking and cleaning; b) they are healthier than the general population; and c) they probably have more frequent testing. Remember this when digesting the information about risk of severe illness, hospitalization and death.

Secondly, the AO trials adopted a methodological sequencing of trial enrollment, roughly speaking, from healthier to less healthy. Younger participants (under 56) were accepted in June and July while older participants (56-69) came in later, followed by the oldest (70 and over). Other subgroups like HIV-positive people entered even later. This means that we must not assume that participants from one part of the trial can be directly compared to another part of the trial. In a typical clinical trial, we assume that the characteristics of early enrollees are similar to those of late enrollees. In this case, beware of comparisons in which one subgroup contains more early enrollees than another subgroup.

The sequencing problem is not limited to the AO trial. You may recall that in some of the American trials, the pharmas announced that they were targeting more minorities toward the end of the trials. This creates a bias as well.

Thirdly, the U.K. and Brazil AO trials are both extremely heavily biased toward white participants (90%, 70%), and so it's tough to judge whether the vaccine would work for minorities. Recall that subsequent to the Lancet paper, it was reported that the AO vaccine was far less effective in the South Africa trial. The underperformance has widely been attributed to the South African variant. You know what, the South African trial was 70% blacks. How do we know race isn't a primary factor?

***

The LD/SD Debacle

I have left the juiciest part till the end because I don't want this section to overshadow the good work that these researchers did. I also do not have any insider knowledge, and so will stick to what was disclosed in their official documents - which represents undoubtedly the scientists as seen through the eyes of AstraZeneca lawyers.

In AO's first press release about the trial, they highlighted a 90% vaccine efficacy finding in a group that received a half dose as the first dose of the vaccine. Some smart reporters picked up on the anomaly - as they didn't recall that the AO trial includes a treatment arm that has a half-dose first shot. AstraZeneca management then explained that this arm was accidentally created when they discovered that some of the participants were mistakenly given half doses.

Alarm bells ring off if you're statistically trained. It really doesn't help that AstraZeneca did not address this issue upfront. It's like a furniture dealer delivering me a new sofa set, forgetting to let me know that a cushion is missing - are they hoping that I'd not notice?

Statisticians also know that such a mistake can really upset a well-designed clinical trial. Is there any assurance that the accident fell on the study population "at random"? The subsequent explanation from Astrazeneca did not address this important question. The situation got even muddier when the Oxford researchers denied the "accident" explanation, claiming that it was part of the plan.

So, I was very eager to read the official disclosure in the Lancet paper, as surely they must address it head on. According to the paper, the original design of the AO trial certainly did not contain an LD/SD (low dose, standard dose) treatment arm. In late July, they amended the trial protocol after deciding that the early enrollees were given a half dose instead of a full dose. To complicate matters more, at that time, the trial design stipulated a single dose only. (In other words, if the trial were stopped at that stage, no one in the trial would have been given the SD single shot as originally planned.) So, they actually made two major alterations to the pre-specified plan: adding a second dose for all participants; and creating a new treatment arm that would be the originally-planned SD/SD, while retaining the existing treatment arm that would become LD/SD.

In my view, a more objective way to describe the LD/SD cohort is the "early enrollees" (June to July 10) and the SD/SD cohort is the "late enrollees" (July 9-August). This descriptor does not require any adjudication. We currently refer to them as LD/SD and SD/SD because the AO team told us to. In the Lancet paper, they spent paragraphs on why this interpretation is justified.

With the AO interpretation, one of the trial findings is that people who received LD/SD treatment enjoyed a 90% vaccine efficacy compared to those who got SD/SD. A different way of describing the same analysis is that people who enrolled before July 10 enjoyed a 90% vaccine efficacy compared to those who enrolled later. Regardless of which description, this comparison is marred by built-in biases. As described above, later participants in the U.K. trial are older because of sequential enrollment. The condition of the pandemic was also evolving. Because of the accident, they don't have randomized comparison groups. A proper experiment that investigates dosage should have both groups run simultaneously, and participants randomly assigned to either half or full doses.

The AO interpretation depends entirely on accepting their explanation of how they discovered that the early shots were half doses, and how they knew which participants were given half doses, after the shots went into arms. They devoted many paragraphs to this but since the story is so complicated, I'll address it in the next post.

 

 

 

How to tell if someone is a serious data analyst

The unexpected turns in the pandemic have exposed which data analysts are serious, and who aren't. At the current moment, anyone who proclaims that vaccines are the cause of the dramatic receding of the coronavirus has proven to be not serious. They are analysts who embrace every nugget of confirming data while ignoring inconvenient observations.

What are some of these inconvenient data points?

• The number of cases has come down drastically in many places, including many that have not yet rolled out their vaccination campaigns. (e.g. India)
  
  
• The level of decline in many places far exceeds what is a reasonable expectation from the data observed in the vaccine trials. In almost every place, the vast majority of the population have not been vaccinated.
  
  
• Some organizations have issued deceptive analyses which the media should have flagged as potential misinformation. The much-reported analysis by the largest healthcare provider in Israel compared what they defined as effectiveness measured from second shot (2D) to 2D+7 days to the VE number from the Pfizer trial, which counted cases from 2D+7 days to a median two months later. In other words, the case-counting window of those two numbers do not overlap for even a single day! Talk to us in a month or so.
  
  
• If these providers are tracking all those they have vaccinated over time, they should publish all the data (from the first shot) as opposed to selecting a few days. That way, we can access how different the real-world experience is from what happened during the trial. (To be fair, I expect those two to differ because we're not measuring the same thing. The idea that those two should match perfectly is bonkers.)

It may be hard to imagine how complicated it is to get an estimate of real-world vaccine effectiveness. In a clinical trial, we have access to a cohort of people who are given two placebo shots and are being followed up alongside the vaccinated cohort. We don't have a placebo cohort in real life.

An unvaccinated person is not the same as someone who's been given saline shots. It's long been known that the placebo effect is real - you can give people sugar pills and some of them will get better. This means any real-world measurement is over-optimistic.

**Now I'm grabbing you with me while I jump into the deep end

We've identified a group of people who are unvaccinated. We even go one step further and match these people's characteristics to those who have been vaccinated. This means the age-group distribution is statistically the same across both vaccinated and unvaccinated groups, and so on. Remember the Israeli healthcare provider counted cases between 2D (second shot) and 2D+7.

Here's the catch. When is 2D in the unvaccinated group? You know what, we cannot define 2D because none of these people got any shots! In fact, only those who are symptomatic and get tested will show up at a facility - the rest are just people in databases with a "vaccination = No" flag. Take the unvaccinated person who just tested positive. Where in this timeline did the person test positive? How do we know the timing of this case is within that 2D to 2D+7 window?

The answer is we don't. It is not even possible to define this. Depending on what we define as Day 0, this person can be counted in any number of analyses. This is yet another reason why we need to look at longer time windows. Any analysis based on short time windows is junk.

***

One way to get around this issue is to match the timing. Take each vaccinated individual in the analysis population, find a "double" via demographic matching in the unvaccinated group, exclude anyone who has reported symptoms by Day 0 as defined by the vaccinated group, and then, mark off the corresponding 2D to 2D+7 window for each matched individual. As per protocol, if the unvaccinated double gets infected before 2D, that case cannot count. (If you don't narrow the case-counting window for the unvaccinated, then the analysis is hopelessly favorable to the vaccinated group!)

The key lesson is if you don't have experimental data, the data analysis gets a lot more complex (more stimulating, but also fraught with risk). The analyst must release full details of the analysis; otherwise, it's impossible to evaluate its merit.

 

 

 

Eight reasons why someone can still get infected after vaccination

The efficacy achieved at various Covid-19 vaccine trials is a marvel of medical science. Because of ballyhooing around 90%-plus numbers, the media roll eyes each time they discover someone who tests positive despite getting vaccinated. This is not surprising, though. Here are 8 ways it happens - they are hiding in the text of the trial reports.

1) Mirage of measurement. Realize that when the investigators computed 90%+ efficacy during the vaccine trials, they only counted cases that surfaced from 7 to 21 days after the second shot (the exact day depends on individual trials). Mathematically, this definition nullifies all cases that happened right after getting the first shot, during the interval between shots, and the 6 to 20 days after the second shot. Paradoxically, the trial data showed that a lot more people get infected during those time windows than during the analysis frame! (See this post for a visualization of the time-line.) 

2) The trials report lower vaccine efficacy prior to the second shot (see chart below and explanation here). The vaccine tempered down the growth rate of cases over time, and in the case of Pfizer/Biontech, eventually attaining about 80 percent reduction. (If we count all cases after the first shot, the VE was 80%, not 95%.) During the Pfizer trial, the efficacy was roughly 50% on the day before the second shot so it's clear some people will get infected in the 20 days after taking the first shot.

3) The infections after vaccination may be asymptomatic. With the exception of the U.K. edition of the AO trial (I discussed the findings here), all other trials with published results so far have measured only symptomatic, and PCR-confirmed cases. By not counting asymptomatic cases, the trial results mislead us into inferring there were no asymptomatic cases. (Nevertheless, it is reasonable to believe that one reason for finding fewer symptomatic cases among the vaccinated trial participants is the partial suppression of asymptomatic spread - although the vaccine trials were not designed to confirm this hypothesis, and no amount of post-hoc analyses can prove it.)

4) New variants of the coronavirus may evade the approved vaccines. There is emerging evidence that these vaccines may be less effective (or ineffective) against certain new variants of the coronavirus. AO just reported that their South African trial (finally a double-blind trial) showed no efficacy, and it is known that the dominant variant there is new.

5) The unlucky few. Even if a vaccine is 90% effective (pretending that this number measures all infections from the first shot to 60 days after the second shot), this means one's chance of getting infected has been cut by 90 percent, and is not zero.

6) The baseline may deteriorate. Vaccine efficacy is defined relative to placebo (i.e. the baseline rate of cases among an unvaccinated population). If the baseline rate rises, for example, due to viral mutation, the case rate among the vaccinated will also increase, even if the 90% reduction is maintained. This is why epidemiologists are very concerned about the higher transmissibility of new variants.

7) The unlucky whole. Last but not least is statistical error. Statistical analysis takes a sample of data (the thousands of trial participants) and generalizes the result to entire populations (tens of millions and more). Every result has a margin of error, usually 5 percent. The trial result used to assert 90% efficacy could be an outlier. The real efficacy may be less than reported.

8) Vaccinations affect our behavior. Many people expect to return to normal life after getting the vaccine. The baseline rate of infection measured during the vaccine trials occurred in the context of the lockdown and mitigation policies of the time. If the level of contacts in the population jumps as a result of the vaccination campaign, the baseline rate may increase, and so follows the expected case rate among the vaccinated.

***

Even if the case trend does not quickly drop, it's not clear the vaccine is ineffective. On the other hand, even if the vaccine is 50% effective, it would still be helpful in the fight against the pandemic. Finally, approval by the FDA (and other regulators) is not an endorsement of the 90% efficacy number, despite that number being touted all day long. The threshold for emergency approval of these vaccines was 30% efficacy (in some places, 20%). (If the advisors were asked to vouch for the 90% number, I'm sure many of them would have refused.)

Detour to the world of make-believe

Those readers who have taken my invitation to jump in the deep end with the Astrazeneca-Oxford (AO) vaccine trial results have to hang on the pool's edge for another day. I'm getting distracted by another headline-grabbing supplementary analysis just advanced to the media, which makes two new claims:

• that the AO vaccine works better with a longer delay of the second dose,
  
  
• that a single dose of the AO vaccine confers 76% efficacy, which suggests that the second shot is not needed (even though, so far, all those who publishes such findings on the one hand advises taking the second shot, which suggests they don't believe their own results).

As I described it in the last post, when it rains, it pours. This is a pair of supplementary analyses in which the researchers once again slice and dice the data to fit their questions. Even though the data came from a randomized controlled trial, none of these supplementary analyses addresses comparisons of randomized subgroups.

***

First, let's examine the following finding:

In the SD/SD group, after the second dose, efficacy was higher with a longer prime-boost interval: VE 82% [...] at 12+ weeks, compared with VE 55% [...] at < 6 weeks.

If you read my last post on the Lancet paper, you know that they've already performed two other ad hoc analyses in a determined campaign for longer dose intervals. Those prior analyses considered cutoffs of 6 weeks and 8 weeks. Now, the goalpost has been moved to 12 weeks. This new analysis included more data: longer follow-up time for the people previously studied, additional participants (primarily from the Brazil cohort) who did not meet the minimum follow-up time requirement last time, and participants of other trials not included before, including Phase 1 cohorts (excluded in the prior analyses), and a newer trial in South Africa.

I did not see a discussion of U.K. and South African variants, nor why those datasets are similar enough to be pooled. Even if we ignore that difference, we know that the South Africa cohort will only affect the front part of the case trend because the trial started later.

Are you ready to get in the deep deep end with me? Hold tight.

***

This type of analysis is highly seducing. But it is filled with danger, and these analysts have fallen down the rabbit hole. It's not easy to explain the flaw of the analysis methodology. Let me start by visualizing the time-line of the trial experience. Imagine the following schedule for a single trial participant:

This participant enrolls and receives the first shot during Week 1, and then returns for a second shot in Week 4. S/he cannot contribute to the case count until Week 6, at least 14 days after the second dose. The case-counting window ends when the data are locked down for analysis. In this diagram, I assume this happens in Week 25. Thus, this participant will be counted as a confirmed case if it happens between Week 6 and 25 (a window of 19 weeks).

What if the participant gets sick in Week 5? According to the rules set by the pharmas, this case will be ignored. The participant is in effect counted as not infected despite being infected. This is the pink part of the diagram.

What if the participant gets sick before the scheduled second dose? According to the trial protocols, this participant will be excused from the second shot - this means s/he does not have a complete treatment, and so will be dropped from the analysis population. This is shown in the red part of the time-line.

Because this participant took his/her second shot during Week 4, s/he will be included in a low-delay subgroup (interval <= 6 weeks) in the dose interval analysis.

Now let's compare this participant to another one in the long-delay subgroup (interval >= 12 weeks). Notice the bottom time-line: this person will contribute to the case count between 14 days after the second shot and the data lockdown date, which totals 11 weeks.

The diagram exposes serious problems with the analysis methodology.

First, the case-counting window for the low-delay subgroup is consistently longer than that for the high-delay subgroup. The vaccine efficacy formula uses cumulative counts. So, the analysis is biased against the low-delay subgroup because longer windows lead to higher counts. This is a great real-world example of why it is necessary to fix the duration of case-counting for every participant.

Now, consider someone from the placebo group who gets sick during Week 13. Imagine this participant has been re-randomized into the vaccine group. If she gets her second shot in Week 4, then she will be counted a confirmed case. If she takes her second shot in Week 12, then she counts as an exposed but uninfected individual! This is why I disagree with designating that pink area in the timeline. It is much better to count cases from Day 1 of the trial - something known as an intent-to-treat (ITT) analysis.

It gets worse. Imagine someone who would get sick in Week 10. If this participant takes the second shot during Week 4, she counts as a case. If the same person was scheduled to take the second shot during Week 12, she would have been disinvited, and removed from the trial population. This exposes a surviorship bias. Anyone who gets sick during the first 12 weeks after the first dose will be removed from the efficacy calculation in the long-delay subgroup so it is not surprising that the long-delay subgroup showed higher efficacy. It's a mirage of how they defined vaccine efficacy.

***

Because of the definition of the case-counting window as starting 14 days after the second shot, a different way to look at how the VE numbers are computed is shown below:

Since VE is computed on cumulative counts, the short-delay subgroup is bound to have more cases just because the case-counting window is longer (all else being equal). In the following annotation of Figure S-2 from the new study, I indicated several other factors that can explain the apparent counterintuitive finding that the longer-delay subgroup has a higher VE than the lower-delay subgroup.

Sharp-eyed readers may notice that this new analysis includes the LD/SD subgroup while the two dose-interval analyses in the Lancet paper don't. The reason given for excluding the LD/SD subgroup last time was that since the second dose was added months after the enrollment began, the dose interval was greater than 8 weeks for everyone. The fact that this subgroup reappeared would not alter the dose interval. So the LD/SD subgroup resides on the right side of the above chart, to the right of 56 days. The further to the right, the higher the weight of the LD/SD subgroup. What do we know about the LD/SD subgroup? It's the subgroup that showed VE of 90%, which is very likely to be just a statistical anomaly. (In fact, in the latest study, incorporating a longer case-counting window, the VE of that subgroup has dropped to 83%.)

***

Similar issues plague the analysis of a single-dose treatment, so I'll make this short. Remember that the trial design was switched from testing single dose to two doses so there is officially no treatment arm of single dose. This group consists of "participants [who] chose not to receive the second dose" and people who experienced "delays in the administration of the second dose" (for whom they observe up to the second dose).

I have already explained before that this is no longer a randomized trial. The single-dose subgroup differed from the two-dose subgroup significantly in "age, sex, health and social care worker status, dose(LD/SD, SD/SD), country, ethnicity, and follow-up time (p < 0.001)."

The position the researchers take regarding these facts is like most scientific research reports. Disclosure is an act of exorcism. They proceeded as if such biases did not exist, after they acknowledged they do exist.

***

The new paper has not passed peer review. Reviewers should consider the problems outlined here when they evaluate its merits.

 

[P.S. 2/9/2021. The Oxford team has announced this week that the South African trial has shown their vaccine has no efficacy, especially against the dominant virus variant there. This raises the ethical question of why they included the data from South Africa in this analysis of the dose interval. It is clear that the vast majority of the South African cohort received their second dose before 6 weeks, which means they are over-weighted in the short-delay subgroups relative to the long-delay subgroups. Since this cohort has much worse overall efficacy than the other country cohorts, this is a significant factor explaining the variations by dose interval.] 

Notes on the Astrazeneca-Oxford vaccine trial results 3: it's raining numbers

In part 3 of my series on the AstraZeneca-Oxford (AO) vaccine trial results, as published in the Lancet paper (link), I'm giving you a health warning: we're going down the deep end. (The previous installments are here and here.)

As previously discussed, the AO trial experienced major mid-course corrections, upsetting the original test design, which necessarily complicates interpreting the data. Every number that is issued is now loaded with caveats, and rarely can one compare an efficacy calculation with another.

Reading through these analyses is like juggling balls, and forced to juggle more and more balls as I take in each supplementary analysis. The key is to not get tired and lower your guard.

Here is Table 3 of the paper that presents a series of supplementary analysis they used to shed light on various specific questions. While reading this table, I'm always anchored to the headline vaccine efficacy (VE) number of this Phase 3 trial, which is 70%, a pooled result across separate U.K. and Brazil trials.

The Origin of the 90% Efficacy Claim

The first analysis shown in Table 3 had a starring role in Astrazeneca's press release, in the paragraph directly following the overall efficacy result. This analysis claims that after a two-dose treatment of the AO vaccine, the VE reaches 90% if the first dose is a "half" dose. This statement implies that other participants in the trial did not get a "half" first dose. I can infer that the remaining subgroup had a VE lower than 70%, and Table 3 confirms that it is 60% if the first dose is a "standard" dose. Knowing the other side raises an immediate question, as it is unusual that decreasing dosage from the "standard" should improve efficacy.

This would have been an astounding finding if the trial had incorporated two randomized treatment arms, respectively getting lower doses (LD) and standard doses (SD) in the first shot. But the trial design stipulated just one treatment group getting two standard doses (SD/SD). The LD/SD subgroup was written into the trial when it was determined that a quarter of the first doses contained less vaccine than expected. This means the difference between the LD/SD and SD/SD subgroups is more than just the size of the first dose.

In fact, the paper disclosed June 10 as the point of demarcation: every participant in the LD/SD subgroup enrolled prior to June 10 whereas everyone in the SD/SD subgroup enrolled after that date. A quick glance at the case trend of the U.K. indicated that cases were trending downward in June, reaching a temporary plateau in early July.

No amount of magic can cure this factor because the two subgroups are disjoint on the basis of enrollment dates.

It turns out that this subgroup analysis does not include the entire SD/SD subset. We should not directly compare the 70% overall VE with the 90% subgroup VE since the latter excludes the Brazil contingent. The researchers justified the exclusion because no one in Brazil got a half dose. The same researchers previously convinced regulators that the Brazil cohort is similar enough to the U.K. cohort so they can conduct a pooled analysis. So, effectively, they have argued for pooling in one analysis but not the other.

Another sub-subgroup excluded from this subgroup analysis is the older cohort (ages 56 and over). This decision is justified because the LD/SD cohort did not contain any older people; that was because the design stipulated a staged enrollment, starting with lower-risk people and then moving to higher-risk people. As explained above, the LD/SD cohort was finished before the SD/SD cohort began. If they had included all ages, age would become another disjoint factor. In the case of enrollment time, disjointedness isn't a show-stopper but age and country are. This discretion is technically called "researchers' degrees of freedom". With freedom comes responsibility.

The implication is that these individual analyses generate a host of VE numbers, none of which can be directly compared to the others. These numbers are correlated, and yet they do not necessarily have to paint a consistent picture. When several numbers conflict with one another, it reveals the non-randomness lurking behind these comparisons, which denies our ability to clearly explain the discrepancies. 

The Case of the Delayed Second Shot 

That's not the end of the potential confounding factors, though. Virtually everybody in the LD/SD subgroup had to be called back for a second shot when AO changed the protocol in late July, implemented in August. As a result, virtually everyone got their second shot more than 8 weeks (56 days) after the first. By contrast, 40 percent of the SD/SD group took their second shot less than 8 weeks after the first.

To investigate the timing of the second dose, the investigators conducted another analysis by drilling down further. Now they removed everyone in the SD/SD group who received their second shot prior to 8 weeks. Why should the cutoff be set at 8 weeks and not, say, 3-4 weeks (which is the dose interval favored by other pharmas)? It's because the LD/SD subgroup did not contain any participants with dose intervals less than 8 weeks.

So the cutoff is determined by what data was available for analysis. And what data was available was decided by ... you guessed it, the mid-course alteration of the design. As indicated above, this dropped about 40% of the SD/SD group while retaining 100% of the LD/SD group.

On first brush, this created another counterintuitive result. The VE of the U.K. SD/SD subgroup aged 18-55 who received the second shot longer than 8 weeks after this first is 66%, a bit higher than that for the same but who took the second dose fewer than 8 weeks after the first (60%)!

Don't forget that we are no longer analyzing two randomized treatment groups except for dose intervals. When the shorter-delay sub-subgroup were taken out of the analysis data, did these participants exit evenly across the enrollment period? Could someone who got a first shot later receive a second shot earlier than someone else? We don't know because the participants rather than the investigators appeared to control when they accepted the second shot. 

Further, the range around the 66% estimate is from 25 percent to 85 percent, which basically tells you this analysis is worthless, and the gap between 66% and 60% is meaningless.

The Moving Goalpost

Because the regulators were taken by a theory that the second dose should be delayed, they ordered yet another analysis.

This time, the investigators dumped the LD/SD subgroup altogether while recalling the Brazil cohort. They also called back the older people. There exist demographic differences between the U.K. and Brazil cohorts that received the SD/SD treatment. The UK cohort is twice as likely to be 56 and over. The Brazil cohort was two-thirds white while the U.K. cohort was 90% white. Ninety percent of the Brazil cohort were health and social care workers, compared to 60% in the U.K. Notably, the LD/SD cohort who enrolled earlier in the U.K. trial was 90% health and social care workers but is excluded.

Despite these demographic differences, the data from two countries are pooled for this analysis, because the primary factor being investigated is the dose interval. The study population is split by when they received the second dose using a cutoff of 6 weeks. Why 6 weeks and not 8 weeks? It's anybody's guess. I suspect it's partly driven by the later start date of the Brazil trial, which means they have fewer long-interval participants -- the mirror image of the LD/SD cohort. It couldn't be placed earlier because there wouldn't be enough low-delay participants in the U.K. trial.

This analysis again gives a surprising finding, that those who got the second shot in fewer than 6 weeks did worse (53%) than those who took it in longer than 6 weeks (65%). The 12% difference in vaccine efficacy appears meaningful.

Well, well, well. The range estimate of the short-delay sub-sub-group is ...-3% to 79%. Yes, that is negative 3%, and the range is over 80%. Moreover, don't be lulled into thinking that the only difference between these two subgroups is the dose interval. The Brazil cohort was much more likely to have shorter intervals: 73% of the short-delay subgroup came from Brazil while 70% of the long-delay subgroup came from the U.K. Did the short-delay sub-subgroup do worse because of the dose interval or because those participants were much more likely to be in Brazil? Or, because the Brazil cohort is less white?

These numbers can't be compared to the overall 70% VE either because that number contains the LD/SD subgroup.

When it rains, it pours

The parade of "exploratory" analyses did not stop there. Somewhere embedded in the Lancet report is this gem of a sentence:

Although there is a possibility that chance might play a part in such divergent results, a similar contrast in efficacy between the LD/SD and SD/SD recipients with asymptomatic infections provides support for the observation (58·9% [95% CI 1·0 to 82·9] vs 3·8% [−72·4 to 46·3]).

The researchers are making an assertion that a lower dose might after all be better than a higher dose, since the gap between 59% and 4% is so bigly that it could not have been due to chance. To be fair, they provided us the range estimates. Did I see 1% to 83% and negative 72% to 46%? These are some crazy, crazy intervals. Sad, sad intervals. As any statistics student should know, the size of the confidence interval is a function of the sample size. The more samples we have, the narrower the interval, the greater our ability to rule out random chance.

 

Notes on AstraZeneca-Oxford vaccine trial results 2

This is a continuation of a series of posts on the Astrazeneca-Oxford (AO) vaccine trial results. The previous installment is here. These comments are based on the peer-reviewed paper in the Lancet (link).

At the end of the last post, I mentioned that the AO trial experienced multiple meaningful mid-course corrections. I'll discuss the implications of these changes in this post.

Deviations from original protocol

Credit: Joel Kramer/Flickr

An architect publishes the blueprint for a new building, it's approved by regulators, and construction commences. Midway during the build, the architect revises the blueprint. The regulators approve the changes. The building is completed. When the new tenants inspect their new abodes, they are surprised to find huge pillars in the middle of the floor that weren't on the floor plans they were shown when they made the purchase. They feel both violated and helpless. If the change were extra pipes in the toilet, they might not feel as badly.

Reading the Lancet paper (link) is like inspecting the new building, and learning that AO made many alterations to their pre-specified trial protocols. Several of these changes are not cosmetic in nature but foundational. The following major changes were disclosed in the Lancet paper:

a) The vaccine arm was swtiched from one dose to two doses

b) The vaccine arm was split into two subgroups, one of which was supposed to have received a lower dose than expected at the first shot

c) Two separate trials - one in the U.K. and one in Brazil - were combined in a pooled analysis.

 

The one-dose, two-dose debate started much earlier than we thought

The vaccine treatment in the AO trials was switched from one dose to two doses, months after the trial started. It is ironic that the U.K. government is now using the trial results to justify a one-dose treatment (officially, the guidance is not to worry about when to take the second shot) when the investigators felt strongly enough to switch to two doses midway through the trials, which required altering the pre-specified protocols. Critically, when making that change, they could have retained a one-dose arm but decided against it.

What makes this decision even more consequential is that AO eventually carved out a subgroup of the vaccine arm - those who received a lower dose at first dose (more later). By the time the protocol was altered (in late July, and implemented early August), this entire subgroup had already received the first dose, which means their treatment would have already been complete, all of them at Day 30 of follow-up or later, when suddenly they were asked to receive a second dose. This late protocol revision partly explains why many participants got the second shot much later than the 3-4 weeks recommended by Pfizer/Moderna or the 4-6 weeks targeted in the revised AO protocol.

While the Brazil trial started later than the U.K. trial, it too started out with a one-dose treatment, later switched to two doses without retaining a one-dose arm.

Splitting the vaccine treatment arm in two

Some coverage of the AO results highlights a 90% VE number, which is a lot more impressive than 70%. This number comes from the Lancet paper, describing a subgroup of trial participants (labeled LD/SD - low dose/standard dose) who are said to have inadvertently received a lower ("half") dose followed by a full dose of the vaccine. About a quarter of the vaccine arm has been classified as LD/SD.

The first thing to realize is that if there is a subgroup that did better than average VE of 70%, then the rest of the vaccine arm must have done worse than average. There is no way around this math. So, if we split the vaccine arm into two subgroups, LD/SD and SD/SD, then the VE of the SD/SD group fell to 62%. Any commentator who mentions 90% without mentioning 62% is cherry-picking the data. An exception can be made if one has a cogent argument as to why the SD/SD group is invalid - but, in this case, the SD/SD group represents the desired treatment!

The second thing to realize is that the trial design did not designate three randomized arms, LD/SD, SD/SD, and control, so that each arm can be compared to another while assuming "all else equal" except for the treatment. The comparison of LD/SD and SD/SD was presented as if the data came from a proper randomized controlled experiment but that is far from the truth. According to the researchers, the LD/SD subgroup arose unintentionally when they discovered that some participants got a lower-than-expected volume of the vaccine in the first dose (more on this in a future post).

As a result, the LD/SD and SD/SD subgroups differ not only by the amount of vaccine but a host of other factors, which include at least the following:

i) age: everyone in LD/SD was 55 years or under while 20 percent of SD/SD was over 55.

ii) country of abode: everyone in LD/SD came from the U.K. trial while the SD/SD participants were split half-half between U.K. and Brazil.

iii) timing of enrollment: everyone in LD/SD received the first shot before anyone in SD/SD got the first shot.

iv) timing of second dose: essentially no one in LD/SD received the second shot before week 8 while 40% of the SD/SD U.K. people and 80 % of the SD/SD Brazil people did

v) occupation: 90% of LD/SD are health and social care workers compared to 60% of SD/SD U.K. and 88% of SD/SD Brazil.

This mess is a case study for why statisticians insist on careful test design and high quality of test execution. The attempt to salvage this trial led to the counterintuitive result that LD/SD had better efficacy than SD/SD. Nevertheless, this result could have been due to any combination of younger people, more health and social care workers, more Britons, or earlier enrollment (when the pandemic was less serious) in the LD/SD subgroup. Those are the known differences; there will also be unknown differences because the treatment was not randomly assigned. It is impossible to separate the confounded effects without making assumptions.

The counterintuitive result may even be a matter of pure chance. The original trial was sized to analyze the vaccine group in aggregate, not split into two parts. Thus, an analysis of these subgroups necessarily result in wider error bars. This is one reason why AO requested a pooled analysis combining results from the U.K. and Brazil trials. When combined, the SD/SD group was judged to be 62% effective, with a confidence interval of 40% to 75%. Without adding the Brazil cohort, the VE is 60%, with a confidence interval of 28% to 78%. This presents two problems: first the 50% range is unacceptably wide, and the lower end of 28% is below the critical number of 30%. (See this previous post for why the FDA is looking for VE > 30%.)

Pooling data from two trials

Another alteration to the pre-specified trial protocol is to pool together the data from the U.K. and Brazil trials for analysis. This is different from Pfizer running one trial across different sites in different countries.

The potential problems is similar to the LD/SD issues. The following list are some of the known differences between the two trials:

i) timing: the U.K. trial started about a month before the Brazil trial

ii) placebo: in the U.K. trial, the placebo originally was the meningococcal vaccine, then it became saline when the second shot was added while in Brazil, all placebo shots were saline

iii) timing of second shot: 60% of the U.K. (SD/SD) group got their second shot after 8 weeks while 80% of the Brazil (SD/SD) group got their second shot before 8 weeks

iv) race: 90% of the U.K. (SD/SD) group are whites compared to 60% in Brazil.

v) occupation: 60% of the U.K. (SD/SD) group are health and social care workers compared to 88% in Brazil

vi) the state of the pandemic, including public health policies, differs significantly between the two countries

It's not that any of these differences destroys the analysis but by pooling the data, an implicit assumption is made that none of those (and other unknown) differences matters. This decision was made to expedite the issuance of actionable results - another example of a shortcut taken in response to the public health emergency.

Let me put it this way: I don't envy the data analysts; these protocol deviations cook up a big mess.

***

In summary, the significant alterations of the protocol pollute the original randomization of treatment, making it much harder to understand the data because the groups being compared no longer differ only by the treatment but by a host of known and unknown biases. In the Lancet paper, these less rigorous analyses are described as "exploratory" so when you see that word, you know the analysts are treating the non-randomized data as  if they are randomized.

The reader must be very careful when interpreting any result from this AO analysis. Pretty much every analysis is based on a different set of participants, and most of these subsets cannot be interpreted as randomized.

Notes on AstraZeneca-Oxford vaccine trial results 1

I have split this post into 3 or 4 posts since I have made lots of notes and comments on the results of the AstraZeneca-Oxford (AO) vaccine Phase 2/3 trial, published in the Lancet (link).

These notes are intended for those who care about the details of these studies, how they were conducted, where shortcuts were taken, what assumptions were made in computing outcomes, and how to interpret the result - from the perspective of a statistician. They do not make a case for rejecting this or other Covid-19 vaccines. I emphasize that these vaccines are useful in our fight against the coronavirus, though I worry that their effectiveness has been overstated.

The AO vaccine has some strong non-medical advantages. AstraZeneca is making it available broadly and at cost. The vaccine can be stored in regular refrigerators and so significantly simplify distribution to countries that can't afford to buy expensive cold storage equipment.

Headline Result

The vaccine efficacy (VE), aggregate of the U.K. and Brazil trials, is 70% (range: 55% to 81%). This means vaccination cuts the chance of getting sick with Covid-19 by 70 percent. I have said it often and must say it again: this does not mean 70 out of 100 vaccinated people are protected. Imagine goalies in the Premier League: the best goalie is about 67 percent better than the worst goalie in goals against shots. This does not mean that the best goalie will keep out 2 out of 3 shots! In fact, the best goalie saves 4 out of 5 shots.

This egregious misinterpretation of VE is so prevalent now I hear it almost every night on cable news. And the NYC health department even posted this untruth on twitter (see right).

As you've learned from my previous notes on Pfizer and Moderna, each pharma defines cases differently. For AO, they only start counting cases from 14 days after the second dose (all cases that occur before this time are dropped on the floor). Unlike Pfizer and Moderna, AO did not tightly control when the second dose was administered. So each participant was observed for a variable amount of time. Some received the second dose 6 weeks after the first dose (Day 42) while some took the shot 12 weeks after their first dose (Day 84), etc. By contrast, the gap between doses was 21 days for Pfizer, and 28 days for Moderna. One reason for the variable time of the second dose is the mid-course switch from single dose to two doses (more on this below) but even after taking that into account, the data still showed that AO did not control the timing of the second dose, despite saying in the protocols that the "target" timing should be four to six weeks (Day 28 to 42).

The primary analysis counts "confirmed cases," similar to Pfizer and Moderna, requiring both a positive PCR test and at least one of four symptoms (a shorter list than the other two pharmas). If any symptomatic COVID-19 case is counted, the VE drops slightly to 67%.

The long delay of the second dose means the case-counting window for each participant is much shorter than in the Pfizer/Moderna trials. Remember that cases are only counted from 14 days after the second dose. Someone who took the second dose at Day 84 (12 weeks after the first dose) won't count as a case until Day 99 or after. If the analysis was conducted on Day 120, then this participant's case-counting window was just 20 days. I computed this not because it's fun and games but because this has practical implications. If I get the first shot of AO today, and then get the second shot 12 weeks later, the published VE of 67% applies to me only for the period from Day 99 to Day 120.

By my calculation, the first participant in the SD/SD group of the U.K. trial has a case-counting window of about 65 days. This means the vast majority of the participants in the AO trial do not have at least two months follow-up after the second shot, the supposed FDA requirement which nonetheless was not strictly adhered to by Pfizer or Moderna.

All the caveats apply to the AO result like the others. Infections will definitely occur after the first dose and before 14 days from the second dose. Asymptomatic infection and transmission are not of primary interest (more below). Subgroup analyses have too few samples to be convincing. Most participants have been observed for fewer than two weeks, and so the duration of protection is uncertain. It's a single-blind study, not double-blind.

As with the other trials, the researchers did not find any significant safety concerns.

Statistical Uncertainty

Despite the large sample sizes of these vaccine trials (relative to the typical drug trial), the measured outcomes still have a lot of statistical uncertainty. The confidence interval around the headline 70% VE is from 55% to 81%. This is a very wide range. This interval says that a vaccine with VE as low as 55% (and as high as 81%) could conceivably register a VE of 70% in any single trial. In any experiment, we'd like the confidence interval to be narrower, and larger sample size is the key to improving the precision.

The low precision is also why we shouldn't pay much attention to subgroup analyses. Restricting an analysis to age groups, ethnicities or other subgroups reduces the sample sizes further, resulting in even wider confidence intervals.

Asymptomatic Cases

One of the high points of the AO trial is the attempt to measure asymptomatic cases. It is now widely acknowledged that up to 50% of transmission is by people who are not experiencing symptoms.

Recall that neither Pfizer nor Moderna bothered to measure asymptomatic cases. In order to count as a case, the trial participant must report symptoms, and then their infection must be confirmed by a positive PCR test. This was a baffling design flaw because it is feasible to test trial participants frequently for the virus. The operation would have been no more complex than the broad-based testing plan implemented by many U.S. colleges that re-opened their campuses in the fall!

AO showed that it is possible to measure asymptomatic infections. In the AO trial, participants were asked to mail in nasal swabs weekly. These swabs were tested, and asymptomatic cases were identified. Nevertheless, the headline number excluded asymptomatic cases. When asymptomatic cases were added to the totals, the VE dropped from 70% to 56%. (Remember: these also exclude cases detected prior to 14 days after the second dose.)

The good news is the vaccine is still effective. There are two insights worth further investigating here.

Of the cases attributed to the placebo arm, 40/153 = 26% are asymptomatic while for the vaccine arm, it's 43%. One explanation for the wide gap is that the vaccine reduces case severity. Another explanation is that people on the placebo arm were more likely to self-report symptoms.

It turns out that not all of the swabs received could be traced back to trial participants. Of the negative swabs, 98% were identified. Of the positive swabs, 81% were identified. This implies that the asymptomatic cases are under-counted. There isn't an easy explanation why the identification rate is lower if the swab is positive. We don't know if the probability of identification differ by vaccine vs placebo.

Deviations from original protocol

Credit: Joel Kramer/Flickr

An architect publishes the blueprint for a new building, it's approved by regulators, and construction commences. Midway during the build, the architect revises the blueprint. The regulators approve the changes. The building is completed. When the new tenants inspect their new abodes, they are surprised to find huge pillars in the middle of the floor that weren't on the floor plans they were shown when they made the purchase. They feel both violated and helpless. If the change were extra pipes in the toilet, they might not feel as badly.

Reading the Lancet paper (link) is like inspecting the new building, and learning that AO made many alterations to their pre-specified trial protocols. Several of these changes are not cosmetic in nature but foundational. The following major changes were disclosed in the Lancet paper:

a) The vaccine arm was swtiched from one dose to two doses

b) The vaccine arm was split into two subgroups, one of which was supposed to have received a lower dose than expected at the first shot

c) Two separate trials - one in the U.K. and one in Brazil - were combined in a pooled analysis

I'll discuss the implications of those alternations in the next post.

 

[P.S. 2/3/2021. A reader sent me this Tech Review article that says Pfizer now wants to track asymptomatic cases as well. Of course, I like this development. The reporter said the reason the Pfizer and Moderna trials ignored asymptomatic cases was that "it's expensive and complicated to measure". This is a sad excuse, as the AstraZeneca-Oxford trial showed. I hope Pfizer has a better plan - the example given in the article is quite useless: someone was swabbed once at the end of January, months after his second shot in September. Pfizer can do better. As the article pointed out, the AO method measures cases not spread but I think their result showed it's a minor concern.]

[P.S. 2/4/2021. The second part of this series is here.