I have split this post into 3 or 4 posts since I have made lots of notes and comments on the results of the AstraZeneca-Oxford (AO) vaccine Phase 2/3 trial, published in the Lancet (link).
These notes are intended for those who care about the details of these studies, how they were conducted, where shortcuts were taken, what assumptions were made in computing outcomes, and how to interpret the result - from the perspective of a statistician. They do not make a case for rejecting this or other Covid-19 vaccines. I emphasize that these vaccines are useful in our fight against the coronavirus, though I worry that their effectiveness has been overstated.
The AO vaccine has some strong non-medical advantages. AstraZeneca is making it available broadly and at cost. The vaccine can be stored in regular refrigerators and so significantly simplify distribution to countries that can't afford to buy expensive cold storage equipment.
Headline Result
The vaccine efficacy (VE), aggregate of the U.K. and Brazil trials, is 70% (range: 55% to 81%). This means vaccination cuts the chance of getting sick with Covid-19 by 70 percent. I have said it often and must say it again: this does not mean 70 out of 100 vaccinated people are protected. Imagine goalies in the Premier League: the best goalie is about 67 percent better than the worst goalie in goals against shots. This does not mean that the best goalie will keep out 2 out of 3 shots! In fact, the best goalie saves 4 out of 5 shots.
This egregious misinterpretation of VE is so prevalent now I hear it almost every night on cable news. And the NYC health department even posted this untruth on twitter (see right).
As you've learned from my previous notes on Pfizer and Moderna, each pharma defines cases differently. For AO, they only start counting cases from 14 days after the second dose (all cases that occur before this time are dropped on the floor). Unlike Pfizer and Moderna, AO did not tightly control when the second dose was administered. So each participant was observed for a variable amount of time. Some received the second dose 6 weeks after the first dose (Day 42) while some took the shot 12 weeks after their first dose (Day 84), etc. By contrast, the gap between doses was 21 days for Pfizer, and 28 days for Moderna. One reason for the variable time of the second dose is the mid-course switch from single dose to two doses (more on this below) but even after taking that into account, the data still showed that AO did not control the timing of the second dose, despite saying in the protocols that the "target" timing should be four to six weeks (Day 28 to 42).
The primary analysis counts "confirmed cases," similar to Pfizer and Moderna, requiring both a positive PCR test and at least one of four symptoms (a shorter list than the other two pharmas). If any symptomatic COVID-19 case is counted, the VE drops slightly to 67%.
The long delay of the second dose means the case-counting window for each participant is much shorter than in the Pfizer/Moderna trials. Remember that cases are only counted from 14 days after the second dose. Someone who took the second dose at Day 84 (12 weeks after the first dose) won't count as a case until Day 99 or after. If the analysis was conducted on Day 120, then this participant's case-counting window was just 20 days. I computed this not because it's fun and games but because this has practical implications. If I get the first shot of AO today, and then get the second shot 12 weeks later, the published VE of 67% applies to me only for the period from Day 99 to Day 120.
By my calculation, the first participant in the SD/SD group of the U.K. trial has a case-counting window of about 65 days. This means the vast majority of the participants in the AO trial do not have at least two months follow-up after the second shot, the supposed FDA requirement which nonetheless was not strictly adhered to by Pfizer or Moderna.
All the caveats apply to the AO result like the others. Infections will definitely occur after the first dose and before 14 days from the second dose. Asymptomatic infection and transmission are not of primary interest (more below). Subgroup analyses have too few samples to be convincing. Most participants have been observed for fewer than two weeks, and so the duration of protection is uncertain. It's a single-blind study, not double-blind.
As with the other trials, the researchers did not find any significant safety concerns.
Statistical Uncertainty
Despite the large sample sizes of these vaccine trials (relative to the typical drug trial), the measured outcomes still have a lot of statistical uncertainty. The confidence interval around the headline 70% VE is from 55% to 81%. This is a very wide range. This interval says that a vaccine with VE as low as 55% (and as high as 81%) could conceivably register a VE of 70% in any single trial. In any experiment, we'd like the confidence interval to be narrower, and larger sample size is the key to improving the precision.
The low precision is also why we shouldn't pay much attention to subgroup analyses. Restricting an analysis to age groups, ethnicities or other subgroups reduces the sample sizes further, resulting in even wider confidence intervals.
Asymptomatic Cases
One of the high points of the AO trial is the attempt to measure asymptomatic cases. It is now widely acknowledged that up to 50% of transmission is by people who are not experiencing symptoms.
Recall that neither Pfizer nor Moderna bothered to measure asymptomatic cases. In order to count as a case, the trial participant must report symptoms, and then their infection must be confirmed by a positive PCR test. This was a baffling design flaw because it is feasible to test trial participants frequently for the virus. The operation would have been no more complex than the broad-based testing plan implemented by many U.S. colleges that re-opened their campuses in the fall!
AO showed that it is possible to measure asymptomatic infections. In the AO trial, participants were asked to mail in nasal swabs weekly. These swabs were tested, and asymptomatic cases were identified. Nevertheless, the headline number excluded asymptomatic cases. When asymptomatic cases were added to the totals, the VE dropped from 70% to 56%. (Remember: these also exclude cases detected prior to 14 days after the second dose.)
The good news is the vaccine is still effective. There are two insights worth further investigating here.
Of the cases attributed to the placebo arm, 40/153 = 26% are asymptomatic while for the vaccine arm, it's 43%. One explanation for the wide gap is that the vaccine reduces case severity. Another explanation is that people on the placebo arm were more likely to self-report symptoms.
It turns out that not all of the swabs received could be traced back to trial participants. Of the negative swabs, 98% were identified. Of the positive swabs, 81% were identified. This implies that the asymptomatic cases are under-counted. There isn't an easy explanation why the identification rate is lower if the swab is positive. We don't know if the probability of identification differ by vaccine vs placebo.
Deviations from original protocol
An architect publishes the blueprint for a new building, it's approved by regulators, and construction commences. Midway during the build, the architect revises the blueprint. The regulators approve the changes. The building is completed. When the new tenants inspect their new abodes, they are surprised to find huge pillars in the middle of the floor that weren't on the floor plans they were shown when they made the purchase. They feel both violated and helpless. If the change were extra pipes in the toilet, they might not feel as badly.
Reading the Lancet paper (link) is like inspecting the new building, and learning that AO made many alterations to their pre-specified trial protocols. Several of these changes are not cosmetic in nature but foundational. The following major changes were disclosed in the Lancet paper:
a) The vaccine arm was swtiched from one dose to two doses
b) The vaccine arm was split into two subgroups, one of which was supposed to have received a lower dose than expected at the first shot
c) Two separate trials - one in the U.K. and one in Brazil - were combined in a pooled analysis
I'll discuss the implications of those alternations in the next post.
[P.S. 2/3/2021. A reader sent me this Tech Review article that says Pfizer now wants to track asymptomatic cases as well. Of course, I like this development. The reporter said the reason the Pfizer and Moderna trials ignored asymptomatic cases was that "it's expensive and complicated to measure". This is a sad excuse, as the AstraZeneca-Oxford trial showed. I hope Pfizer has a better plan - the example given in the article is quite useless: someone was swabbed once at the end of January, months after his second shot in September. Pfizer can do better. As the article pointed out, the AO method measures cases not spread but I think their result showed it's a minor concern.]
[P.S. 2/4/2021. The second part of this series is here.
Comments