I'm not done with my notes on the Astrazeneca-Oxford vaccine trial results. Previous installments are 1, 2 and 3. This post and this post are also both related to it.
***
Trickle-down Biases
When I examined specific pieces of analyses in the Lancet paper, I've previously explained the biases that emerge when we can no longer assume that the groups being compared are randomized. While those details are tasty, we should also pay attention to biases that trickle down from the top.
Firstly, the AstraZeneca-Oxford (AO) trials targeted health and social care workers, which typically account for 60-90 percent of the analysis populations. This targeting makes sense because those front-line workers are at high risk of exposure to the virus, and so we'd like to have good data on the vaccine's effect on them. However, this decision also makes it harder to generalize the results to the rest of the population - simply because health and social care workers as a class is not "interchangeable" with the rest of the population. There are three major differences: a) their workplaces are much more strict about measures such as masking and cleaning; b) they are healthier than the general population; and c) they probably have more frequent testing. Remember this when digesting the information about risk of severe illness, hospitalization and death.
Secondly, the AO trials adopted a methodological sequencing of trial enrollment, roughly speaking, from healthier to less healthy. Younger participants (under 56) were accepted in June and July while older participants (56-69) came in later, followed by the oldest (70 and over). Other subgroups like HIV-positive people entered even later. This means that we must not assume that participants from one part of the trial can be directly compared to another part of the trial. In a typical clinical trial, we assume that the characteristics of early enrollees are similar to those of late enrollees. In this case, beware of comparisons in which one subgroup contains more early enrollees than another subgroup.
The sequencing problem is not limited to the AO trial. You may recall that in some of the American trials, the pharmas announced that they were targeting more minorities toward the end of the trials. This creates a bias as well.
Thirdly, the U.K. and Brazil AO trials are both extremely heavily biased toward white participants (90%, 70%), and so it's tough to judge whether the vaccine would work for minorities. Recall that subsequent to the Lancet paper, it was reported that the AO vaccine was far less effective in the South Africa trial. The underperformance has widely been attributed to the South African variant. You know what, the South African trial was 70% blacks. How do we know race isn't a primary factor?
***
The LD/SD Debacle
I have left the juiciest part till the end because I don't want this section to overshadow the good work that these researchers did. I also do not have any insider knowledge, and so will stick to what was disclosed in their official documents - which represents undoubtedly the scientists as seen through the eyes of AstraZeneca lawyers.
In AO's first press release about the trial, they highlighted a 90% vaccine efficacy finding in a group that received a half dose as the first dose of the vaccine. Some smart reporters picked up on the anomaly - as they didn't recall that the AO trial includes a treatment arm that has a half-dose first shot. AstraZeneca management then explained that this arm was accidentally created when they discovered that some of the participants were mistakenly given half doses.
Alarm bells ring off if you're statistically trained. It really doesn't help that AstraZeneca did not address this issue upfront. It's like a furniture dealer delivering me a new sofa set, forgetting to let me know that a cushion is missing - are they hoping that I'd not notice?
Statisticians also know that such a mistake can really upset a well-designed clinical trial. Is there any assurance that the accident fell on the study population "at random"? The subsequent explanation from Astrazeneca did not address this important question. The situation got even muddier when the Oxford researchers denied the "accident" explanation, claiming that it was part of the plan.
So, I was very eager to read the official disclosure in the Lancet paper, as surely they must address it head on. According to the paper, the original design of the AO trial certainly did not contain an LD/SD (low dose, standard dose) treatment arm. In late July, they amended the trial protocol after deciding that the early enrollees were given a half dose instead of a full dose. To complicate matters more, at that time, the trial design stipulated a single dose only. (In other words, if the trial were stopped at that stage, no one in the trial would have been given the SD single shot as originally planned.) So, they actually made two major alterations to the pre-specified plan: adding a second dose for all participants; and creating a new treatment arm that would be the originally-planned SD/SD, while retaining the existing treatment arm that would become LD/SD.
In my view, a more objective way to describe the LD/SD cohort is the "early enrollees" (June to July 10) and the SD/SD cohort is the "late enrollees" (July 9-August). This descriptor does not require any adjudication. We currently refer to them as LD/SD and SD/SD because the AO team told us to. In the Lancet paper, they spent paragraphs on why this interpretation is justified.
With the AO interpretation, one of the trial findings is that people who received LD/SD treatment enjoyed a 90% vaccine efficacy compared to those who got SD/SD. A different way of describing the same analysis is that people who enrolled before July 10 enjoyed a 90% vaccine efficacy compared to those who enrolled later. Regardless of which description, this comparison is marred by built-in biases. As described above, later participants in the U.K. trial are older because of sequential enrollment. The condition of the pandemic was also evolving. Because of the accident, they don't have randomized comparison groups. A proper experiment that investigates dosage should have both groups run simultaneously, and participants randomly assigned to either half or full doses.
The AO interpretation depends entirely on accepting their explanation of how they discovered that the early shots were half doses, and how they knew which participants were given half doses, after the shots went into arms. They devoted many paragraphs to this but since the story is so complicated, I'll address it in the next post.
Comments