Big Data, Plainly Spoken (aka Numbers Rule Your World)

Lessons for communicating science emerging from the vaccine campaign

It felt like eons ago when I worried about the obsession over 90% vaccine efficacy, and I'm afraid the chicken has come home to roost. This week, the FDA approves the Johnson & Johnson vaccine, which failed to deliver a 90% number capable of exciting the headline writers. The reported efficacy is in the 60-70% range (I will offer commentary on that later because once again, every trial uses its own measurement of cases and decides which periods to count & not count so none of these numbers are directly comparable anyway - one of the scientific embarrassments of the vaccine trial enterprise so far.)

TV experts are on air breathlessly verbalizing how this new vaccine is a big game changer. It's easier to transport and store, it's one dose, etc. One thing they've avoided like the plague is the question on everybody's mind - at least those intending to take the vaccine: Why are you touting a 65% effective vaccine as pandemic-beating when you've been telling us we have 95% vaccines?

What else are the TV experts not addressing? Since they unremittingly misinterpreted vaccine efficacy as the proportion of people protected (which I have tried to debunk again and again on this blog), does this mean they believe that the J&J vaccine will leave 30-40% of those vaccinated people unprotected? In addition, how will a vaccinated person know if they are protected or unprotected from the shot? (The answer is we won't know.) So do they recommend that everyone who took the J&J vaccine assume they are not protected?

Questions, questions, questions....

***

These are self-inflicted wounds. It's a failure to think of communications in the long term. It's completely predictable, and I've predicted it.

Looking back at some of my posts, I'm surprised that it's only been two months since the first vaccines were approved. So the posts were not as old as I imagined.

On December 1, I titled a post "The press-release derby has set an unrealistic bar for the coronavirus vaccine." In a follow-up post, when the Pfizer vaccine was approved, I wrote "Pfizer has been selling this conclusion: the vaccinated participants had a case rate (not the same as infection rate) that is 95 percent lower than the placebo (unvaccinated) participants. However, focusing on this selling point is likely to harm the reputation of the vaccine because real-world experience will almost surely refute that." Later, when the Moderna vaccine was approved, I was frustrated that the company appeared to be competing with Pfizer on decimal games.

In reality, the FDA panel were not tasked with rubberstamping the 95% number. They would have approved any vaccine that demonstrated at least 30% vaccine efficacy during trial. They are approving a J&J vaccine that has substantially lower efficacy.

But day after day after day, government officials and TV talking heads kept telling people to take these shots because they are miracles that deliver 95% protection. Now, they hope we'll forget they ever said that.

As background, we know that a good portion of the American population has been taking flu shots every year, despite the "vaccine" being only 30% effective. They could have set the bar at 50% and communicated that anything above 50% is good enough. Pfizer and Moderna are great vaccines because the vaccines deliver over 50% efficacy. Instead, they worked overtime to set the bar at 90%. Now, they're stuck.

***

The other hole they dug was the misinterpretation of the vaccine efficacy metric. They misled us by saying out of every 100 people who take the shots, 90 are protected from infection. I've corrected this many times, e.g. here, with a fuller explanation here

The first Big Lie is the concept of protection. Remember that epidemiologists insisted that we shouldn't count cases from the day after you enter treatment (first shot), and we shouldn't count cases from the day after you complete treatment (second shot), and we shouldn't even count cases that happen some 7, 14, or 21 days after completing treatment, so when they say 90% are "protected", they don't care if you got Covid-19 outside their case-counting window.

In their eyes, you are deemed "protected" even if you got ill in between shots, or within 7, 14, or 21 days of your second shot.

We really can't put lipstick on that pig. What is a data communications specialist going to do with that? It's not something the general public can understand, or will accept.

Now say we have a vaccine with 50% VE. It is still a useful tool, and the FDA will approve it. By the same (flawed) logic, this vaccine will protect 50% of those who take the shots. That would imply one out of two people taking this vaccine will still be unprotected even after getting inoculated.

This paints the picture of a binary world in which someone is either protected or not protected by the vaccine, with no possibility of partial protection. Maybe some reader knows the answer. My understanding is it's an open question but it's unlikely to be binary.

This stumps our data communications specialist again. By this logic, each vaccinated person has a 50% chance of being "protected", and a 50% chance of not. It's like the flip of a coin. Sounds good when the coin is 90% biased to one side, doesn't sound that great when the coin is "fair".

The correct interpretation of 50% VE is that it cuts one's chance of infection by half. If you're a front-line worker with high risk of exposure, it cuts your chance of infection by half. (Could be more but we don't have data to look at segments of the population.)

***
We can do the best science, and have the best results. But the strength of the results don't guarantee acceptance. Making a mistake in an experiment or a calculation can be fixed more easily than delivering a first impression to the public that subsequently needs to be erased and replaced.

 

P.S. [3/2/2021] Thinking about this some more. I think a reasonable communication strategy now is to say the J&J vaccine is great for those who are at lower risk of Covid-19. This includes people with low risk of exposure (e.g. live in sparsely populated areas), people who are very careful about staying at home and other mitigation measures, people who believe they are "strong" and don't need maximum protection, etc. For these subgroups, the J&J vaccine is one shot only, and the logistics of getting it might be easier because of the less cumbersome storage and transportation.

 

 

 

The wish that won't quit

 

It amazes me how the media keep inviting experts that push their gut feelings by cherry-picking only the science that confirms their beliefs. I am talking about the wishful thinking that is single-dose vaccines.

If you believe these experts, the evidence is strong, verging on undeniable, that we should skim on the Pfizer, Moderna and Astrazeneca vaccines and only administer one dose. They have most often cited the calculation by the U.K. government which I discussed here. This same argument was parroted by two Canadians in a more recent push for one dose.

The arguments for sticking with what we know are being buried. Here is an excellent summary of the main points. I covered all of these earlier than most in this blog post.

Since that time, there have been many studies that bolster the case for sticking with two doses. You've heard about these studies except they buried the lead in each case.

***

We recently heard there is a study that showed half doses of Moderna might be enough. Such a study did exist, it's an interim analysis of a small-scale randomized trial - but the big story in that study is that the antibody levels did not exceed that obtained through natural infection until after the second doses. I wrote about that study here.

We recently heard about a study in which the Mayo Clinic found 89% efficacy in their real-world study. Great news. What the media experts did not tell you is that this VE was measured from 36 days after the first dose, thus including the effect of the second dose. If measured from first dose, the VE is 60%. Therefore, real-world evidence shows that the VE of a single shot is nowhere near the 90% claimed by these experts.

We recently heard of a study involving Israel's largest healthcare provider, verifying that the Pfizer and Moderna vaccines have real-world efficacy similar to those achieved in the clinical trials. Great. But did the media experts tell you what else they found? The vaccine effectivness at 14 days after the first dose is in the 40-60% range depending on how they defined cases. It was not until after the second shot where effectiveness edged towards 90%. Therefore, real-world evidence shows that the VE of a single shot is nowhere near the 90% claimed by the experts.

***

What I wrote above are directly lifted from those scientific papers. It does not require any interpretation or analysis. I'm just relaying what the researchers said in those papers. The media and their experts are filtering out what don't fit their worldview, and you're worse off for it.

Of course, I have taken notes while reading those studies so in a future post, I'll provide an evaluation of these real-world studies.

 

 

 

How to rise above the cacophony of sometimes phony data analyses

Data analyses are flooding the airwaves. They paint a confusing, contradictory picture of all aspects of the pandemic. Sadly, most of the work are hastily put together, and low quality. Here are some things to keep in mind when you look at these studies.

What happened in an experiment will not happen again

If a vaccine trial returns a 90% vaccine efficacy, we can say for sure the vaccine’s efficacy is precisely 90% for the 30,000 or so participants in that specific vaccine trial. We are not empowered to conclude that the VE for billions of people who will eventually get the shots will be 90% - no matter how many epidemiologists repeat this falsehood on TV.

That’s because the trial involved a small sample of people, and there is a margin of error around the 90% number. If the range estimate is 70% to 98%, then we say “with 95% confidence” that the VE for the general population is above 70%. That’s still an impressive number, and it has the advantage of respecting scientific principles.

Imagine repeating the same clinical trial with different choices of 30,000 participants repeatedly. The margin of error says 95% of the series of VE estimates from these trials will fall inside the 70% to 98% range. The chance of any of these trials repeating the exact 90% VE is practically zero.

Credit: Jeremy Schultz/Flickr

 

The randomized clinical trial (RCT) is the gold standard for establishing cause--effect. The lack of randomization in “real-world” studies opens a can of worms.

In a vaccine trial, we compare the case rates of those who are vaccinated to those who aren’t, which are the primary ingredients of the vaccine efficacy formula. We are currently being served weekly new studies that also compare vaccinated people to unvaccinated people. The media are telling us that these new studies are better because (a) they are more recent (b) they have much larger sample sizes and (c) they constitute “real-world” evidence. A popular theme is that these studies fill in the gaps left by the vaccine trials.

The noise you’re hearing are the chuckles from incredulous statisticians. Since RCTs are regarded as the gold standard for causal inference, no “real world” evidence should be modifying, and certainly not correcting, findings from a scientific experiment. Doing so is like hiring a C student to tutor an A student, and siding with the C student when they disagree.

There is one critically important property missing from all real-world studies: the randomization of treatment. (See, however, so-called natural experiments.) In an RCT, a coin flip determines who gets the vaccine but in the real world, who gets the shots is anything but random. Most countries have priority lists and specific types of people are getting the vaccines earlier than others. In any “real world” study, the unvaccinated group is different from the vaccinated group, not just by vaccination status but also by many other factors, both known and unknown. Any of those other factors could contribute – majorly or minorly – to the observed difference between the two groups. Randomizing treatment ensures that on average, these other factors will not bias the finding in an RCT, a condition that does not exist in real-world studies.

A partial solution is to define a better control group. We’d take a subset of the unvaccinated people who look like those who are vaccinated. This is an effort to manufacture artificially the randomization condition. This matching process is highly subjective, and we can only match people using measurable and influential factors. There is no law that dictates that every important factor is measurable. As an illustrative example, perhaps people with higher-speed Internet connections are more likely to land vaccination slots. Since medical researchers do not have individual data on people’s Internet speeds, we cannot control for this effect. Even the most obvious adjustments have problems. For example, young people can be excluded because they don’t qualify for vaccination yet but young people who are front-line workers are getting shots.

Another common issue is incurable imbalance: if all care home residents have been vaccinated, we won't find care home residents in the unvaccinated group. The biggest problem with matching studies is lack of transparency. The typical disclosure is vague and confusing – I can’t even figure out what they did. None of these studies publish their data or code.

So, a real-world study that corrects for selection bias is better than one that doesn’t but in no case is an observational study superior to an RCT. Larger sample sizes produce more precise estimates (with lower margins of error) but they do not auto-correct biases. More precise estimates derived from biased data pose even greater risk precisely because they feel more solid when the biases are ignored.

The randomized clinical trial is a victim of abuse.

In reviewing the torrent of studies that have come out of the vaccine trials, I can’t help but notice that analysts are mercilessly abusing the RCT framework. I’m going to echo Andrew Gelman’s criticism of the standards of research in psychology. We’re concerned about the high probability of false-positive findings.

The VE numbers from Pfizer, Moderna, Astrazeneca, etc. are not comparable to each other. That’s because each team measures case rates differently. Pfizer drops all cases prior to 7 days after the second shot; Moderna and Astrazeneca drop cases prior to 14 days after the second shot. Then, you have people reanalyzing the data, who use their own case-counting windows. The U.K. government published a post-hoc analysis of the Pfizer data, counting only cases between 15 and 28 days after the first dose; this is echoed in a Canadian re-analysis of the same data, in which they set the window of 14 days to 20 days. A more recent “real-world” study out of Scotland counted from 28 to 34 days after the first dose. This is no small matter as VE is an improvement metric relative to a baseline, and we can’t keep our heads straight which the baseline applies.

Many scientists justified the choice of when they count cases, on record no less, by pointing to the cumulative case curve, arguing for counting when the vaccine’s curve starts flattening because that’s exactly how long it takes for the vaccine to do its magic. This type of post-hoc thinking is what teachers warn students not to do. These clinical trials were sized to measure the endpoint, and not sized to identify a turning point on a case curve. Imagine repeating this trial many times over. Assuming that the VE after 90 days rises to roughly 90 percent in each trial, it is likely that the turning point on the curve is not precisely at day 14 (or whenever it was in the first trial).

Analysts appear to have looked at their individual datasets, identified the period in which the vaccine showed the best performance against the placebo, defined their VE metric to hone in on that time window, and justified the definition with some sciency blather. This is a recipe for over-estimating the true performance.

The proof is the absence of new studies that go through similar post-hoc thinking to adjust down the findings of the respective RCT. In fact, the re-analyses of the Pfizer data, which moved the estimated VE of the first dose from around 50% to 90% is a nice example of a C student correcting an A student.

To prevent the danger of post-hoc theorizing, scientists agree to pre-register their studies so they have to define how they would count cases ahead of time. Unfortunately, those protocols are written loosely to allow a wide range of options, such as 7 days, 14 days, 21 days, etc. as possible evaluation metrics. This maybe exposes how little investigators know about the behavior of their inventions.

If only the start of the case-counting window were the only “researchers’ degree of freedom”. The researchers also follow their noses about (a) the duration of the case-counting window (b) whether a positive test is required to confirm cases (c) which test is allowed for confirmation and the parameters for running the test (d) the maximum length of time between reporting symptoms and testing positive (e) what symptoms are included and excluded as qualifying (f) how many symptoms are qualifying. As I showed in this post, VE drops fast with just a few more cases on the vaccine arm so any of these decisions matters.

The RCT framework does not protect us against these potential abuses. RCTs are the gold standard, but they aren’t fool-proof.

Story time after RCT

The potential problems outlined in the previous section relate mostly to trial design decisions. The vaccine RCTs are also being suffocated by post-hoc hacking. I already devoted an entire post to this phenomenon of “story time”. Not all results obtained from analyzing data from RCTs have the crucial randomization property. That’s because analysts can destroy the property by cherry-picking the data. We’re lulled into thinking these lesser findings have the same status as a proper RCT finding.

The most infamous example of such analysis is the 90% VE attributed to the so-called low-dose, standard-dose subgroup in the Astrazeneca-Oxford trial (link). This VE is indeed based on comparing case rates of a vaccine arm to that of a placebo arm. In a true RCT, those two arms are statistically the same except for what was placed in the two shots. Nevertheless, the low-dose subgroup happened as an accident, and it subsequently emerged that this subgroup contained only younger participants, have earlier enrollment dates, have longer intervals between doses, etc. And yet, on the front page of the Lancet paper summarizing this trial, the investigators stated “In participants who received two standard doses, vaccine efficacy was 62.1%... and in participants who received a low dose followed by a standard dose, efficacy was 90.0%...” By this point, there has been no disclosure about the “accident,” and anyone reading this sentence is likely to assume that there were two randomized dosage schedules used in this trial.

Likewise, much ink has been spilled on dose intervals. Many analysts have compared subgroups that took their second shots later and earlier. Even though the underlying data were collected in an RCT, dose intervals were not randomized. And yet, these results were published and publicized using methods for analyzing RCTs and not methods for analyzing observational studies. In some cases, the researchers even disclosed that these subgroups differed on numerous important factors, and still they plodded on.

**

In this era, even results coming out of RCTs must be scrutinized. Early analyses, post-hoc analyses, deep-dive analyses, and side analyses typically discard the crucial randomization property, and should be treated as observational studies.

Of course, results from observational ("real-world") studies should be scrutinized even more. In a future post, I’ll outline how to review analyses of observational data.

Tantalizing unreliable new data on half doses and single doses

A reader sent me an interim analysis of a new Phase 2 trial of the Moderna vaccine in which they tested whether the dosage could be cut by half. An economist pointed to this result as a big win, as it could double the population of vaccinations while retaining two doses, and the same supply of vaccines.

This paper deserves more coverage.

The trial had three arms, one given the treatment of two "full" doses - the same as in trial that resulted in Moderna's emergency use authorization, one given two "half" doses, and the third arm given placebo (saline) shots.

The key result is shown in the following graph from the paper:

Each arm was sub-divided into two age groups (18-54, and 55+). As with the Phase 3 trial, Moderna relied on infrequent follow-up visits and self-reported symptoms. The chart showed the three follow-up moments, once at "Day 29" (i.e. during the appointment to receive the second dose), once at "Day 43," roughly 2 weeks after the second dose, and once at "Day 57", about 1 month after the second dose.

The chart uses a log2 scale (you can expect a Junkcharts post on this tactic soon). The metric for this Phase 2 trial is not cases but antibody levels. The analysis utilizes two controls. The placebo places the "zero" level while the "Conv" level measures the average amount of antibodies found in the blood of those who are known to have been infected with SARS-CoV-2 ("convalescent"). A viable vaccine should induce production of antibodies well above the level obtained from natural infections.

The twitter users used this paper to push the case for halving the dosage for the Moderna vaccine. They are comparing the gray columns to the red columns, declaring them "similar enough". The evidence is not as strong as it seems. The log2 scale squishes the large values while spreading out the small values. If you look at the numbers on those columns, the red columns are definitely taller than the gray bars. Take the 3rd column for 18-54, half dose, and the 3rd column for 18-54, full dose. The values are 189 and 239, the latter being 26% higher than the former.

Nevertheless, the level of antibodies from natural infections is only 48 so both those values may be sufficient to fight the virus. (Note, as the paper's authors admit, no one at this stage knows the minimum level of required antibodies. The connection between antibody levels and vaccine efficacy has not been established.)

***

The paper has another significant finding, which is relevant to the hot-again debate on whether a single dose is sufficient. Recall that Moderna administers the second dose around Day 28. The chart above shows a huge jump in amount of antibodies from Day 29 to Day 43. That appears to be the effect of the booster shot (possibly with an assist from the first dose). Pointedly, this study also did not include a single-dose arm.

Further, the level of antibodies on Day 29 is way below the "Conv" level. Don't be fooled by the log2 scale. The 55+ age group had antibody levels of 18, less than 40% of the "Conv" level. That spiked to well above the "Conv" level after the second shot.

The authors also compared the last two columns of each experimental group. This shows the change in antibody levels in the 3rd and 4th weeks after the second shot. I'm not sure how this finding of a shorter observation period of an indirect metric could possibly alter the finding of the Phase 3 trial which measured confirmed cases for a median follow-up of almost two months after the second shot. We already know that the vaccine's effect holds for roughly two months.

***

Unfortunately, the level of scientific discourse in the pandemic era has sunk like an egg in water. The following is still true about a study like this:

• It is a Phase 2 study
  
  
• This is an interim (early) analysis, meaning the study population has been followed up to one month after the second shot
  
  
• There were only 100 people in each of those groups, and a total of 600 people in the entire experiment. This is 50x smaller than the 30,000 people in the Phase 3 trial.
  
  
• The sample size is so small that the researchers did not perform statistical analysis - which is a way of saying we can't generalize this result to the larger population. (That's why I wrote about the "law of small numbers" last week.)
  
  
• Another obstacle to generalizing this result is that when you only have 600 people, you have to narrow the criteria for enrollment. The sampling error is a function of sample size, and the inherent variability of the participants. If the participants have to look more alike, they are going to look more dissimilar as a group compared to the general population.
  
  
• Lastly, I will let the authors of the paper do the talking. They described the following limitations of their experiment: "The study population was not designed at the time to be representative of those at risk for SARS-CoV-2 infection or COVID-19. ... the present study was not designed for statistical comparison of superiority or equivalence between doses of [the Moderna vaccine] ... Although antibody responses were generated ... that, on average, exceeded those of convalescent sera, at the present time, a serological correlate of protection against SARS-CoV-2 remains to be determined."

These descriptions of limitations always give me a chuckle, just like reading the "risk factors" section of business prospectuses, after which you wonder how any investors could possibly open their wallets.

After concluding that the half doses and the full doses both produced good antibody responses, the researchers say their study wasn't intended to draw such a conclusion (neither of superiority nor of equivalence). In addition, the connection between level of antibodies and propensity to get infected has not been established. What the study does show clearly is that the level of antibodies spiked about 3 times after the second shots.

 

P.S. [2/23/2021] Post about the graphical design of the chart above is now up on Junk Charts. Find it here.

The slow suffocation of science by data

What we're witnessing in recent weeks is a slow suffocation of science - by the hyperspeed and rulelessness of modern technologies including data and social media.

The dominant themes in pandemic reporting from late 2020 to February have been the holiday spikes, virus variants, from the U.K. then South Africa, vaccinations, forthcoming vaccines, and the dramatic decline in cases.

All sorts of reasons have been concocted to explain the trendlines. So-called experts make passionate arguments mostly based on conjecture built on top of some cherry-picked data.

I took a few minutes to pull together the following chart, so we all can join the cacophony.

Thanks to the team at OurWorldinData for collecting and making the data accessible.

I have chosen a range of countries representing many different experiences. For example, Israel (blue line) has been loudly claiming victory on the vaccination front. The U.K. (pink) likewise has self-congratulated on their vaccination work. There is increasing chatter in the U.S. along the same lines. Meanwhile, Canada (second from bottom) has been criticized for a slow vaccine roll-out. So has the E.U. South Africa (green) also has a slow roll-out, is the epicenter of the much-feared South African variant, and even stopped using the Astrazeneca-Oxford vaccine.

Valid causal factors must explain the trend in every place where the factor is active. Some factors, like the Thanksgiving holiday, are local while many factors, such as vaccinations, are global.

***

After making the above chart, I also checked in on the Swedish experiment:

Isn't it funny that when the trend isn't looking good, the voices disappear? Journalists should be chasing after those epidemiologists and professors to ask whether they repudiate their prior views, or if they have an explanation for what transpired.

The mainstream media have been vocal in stamping out misinformation. And yet, they have given a lot of space early on to this renegade group of scientists. Not correcting the record, and informing their audience that the theory has buckled under the weight of reality, also contributes to misinformation.

***

A sharp drop in cases occurred in Japan starting mid-January, where they only just approved a vaccine (mid February). Here's what happening in Asia Pacific.

***

If you're a business analyst who've been asked to explain the trends in your company's performance indicators, you already know this type of trend analysis is very hard work - you can't just sling things at the wall. Offering explanations for some cherry-picked data is not science; it's a mockery of the scientific method.

If this post makes you question the experts you hear night after night on the news, I'd have succeeded.

 

P.S. [2/22/2021] Right on cue, the New York Times put this article up today. To put it succinctly, the two statements can both be independently true: that the vaccines are helping, and that the cases have recently sharply declined. Two true statements do not a link establish. About the situation in the U.K., they say "Vaccines don’t explain it: Even though a quarter of the population has been vaccinated, only the earliest recipients had significant protection by Jan. 10, when cases there started to drop. Those early doses mostly went to health-care workers and elderly patients already in the hospital."

 

Notes on AstraZeneca-Oxford vaccine trial results 4

I'm not done with my notes on the Astrazeneca-Oxford vaccine trial results. Previous installments are 1, 2 and 3. This post and this post are also both related to it.

***

Trickle-down Biases

When I examined specific pieces of analyses in the Lancet paper, I've previously explained the biases that emerge when we can no longer assume that the groups being compared are randomized. While those details are tasty, we should also pay attention to biases that trickle down from the top.

Firstly, the AstraZeneca-Oxford (AO) trials targeted health and social care workers, which typically account for 60-90 percent of the analysis populations. This targeting makes sense because those front-line workers are at high risk of exposure to the virus, and so we'd like to have good data on the vaccine's effect on them. However, this decision also makes it harder to generalize the results to the rest of the population - simply because health and social care workers as a class is not "interchangeable" with the rest of the population. There are three major differences: a) their workplaces are much more strict about measures such as masking and cleaning; b) they are healthier than the general population; and c) they probably have more frequent testing. Remember this when digesting the information about risk of severe illness, hospitalization and death.

Secondly, the AO trials adopted a methodological sequencing of trial enrollment, roughly speaking, from healthier to less healthy. Younger participants (under 56) were accepted in June and July while older participants (56-69) came in later, followed by the oldest (70 and over). Other subgroups like HIV-positive people entered even later. This means that we must not assume that participants from one part of the trial can be directly compared to another part of the trial. In a typical clinical trial, we assume that the characteristics of early enrollees are similar to those of late enrollees. In this case, beware of comparisons in which one subgroup contains more early enrollees than another subgroup.

The sequencing problem is not limited to the AO trial. You may recall that in some of the American trials, the pharmas announced that they were targeting more minorities toward the end of the trials. This creates a bias as well.

Thirdly, the U.K. and Brazil AO trials are both extremely heavily biased toward white participants (90%, 70%), and so it's tough to judge whether the vaccine would work for minorities. Recall that subsequent to the Lancet paper, it was reported that the AO vaccine was far less effective in the South Africa trial. The underperformance has widely been attributed to the South African variant. You know what, the South African trial was 70% blacks. How do we know race isn't a primary factor?

***

The LD/SD Debacle

I have left the juiciest part till the end because I don't want this section to overshadow the good work that these researchers did. I also do not have any insider knowledge, and so will stick to what was disclosed in their official documents - which represents undoubtedly the scientists as seen through the eyes of AstraZeneca lawyers.

In AO's first press release about the trial, they highlighted a 90% vaccine efficacy finding in a group that received a half dose as the first dose of the vaccine. Some smart reporters picked up on the anomaly - as they didn't recall that the AO trial includes a treatment arm that has a half-dose first shot. AstraZeneca management then explained that this arm was accidentally created when they discovered that some of the participants were mistakenly given half doses.

Alarm bells ring off if you're statistically trained. It really doesn't help that AstraZeneca did not address this issue upfront. It's like a furniture dealer delivering me a new sofa set, forgetting to let me know that a cushion is missing - are they hoping that I'd not notice?

Statisticians also know that such a mistake can really upset a well-designed clinical trial. Is there any assurance that the accident fell on the study population "at random"? The subsequent explanation from Astrazeneca did not address this important question. The situation got even muddier when the Oxford researchers denied the "accident" explanation, claiming that it was part of the plan.

So, I was very eager to read the official disclosure in the Lancet paper, as surely they must address it head on. According to the paper, the original design of the AO trial certainly did not contain an LD/SD (low dose, standard dose) treatment arm. In late July, they amended the trial protocol after deciding that the early enrollees were given a half dose instead of a full dose. To complicate matters more, at that time, the trial design stipulated a single dose only. (In other words, if the trial were stopped at that stage, no one in the trial would have been given the SD single shot as originally planned.) So, they actually made two major alterations to the pre-specified plan: adding a second dose for all participants; and creating a new treatment arm that would be the originally-planned SD/SD, while retaining the existing treatment arm that would become LD/SD.

In my view, a more objective way to describe the LD/SD cohort is the "early enrollees" (June to July 10) and the SD/SD cohort is the "late enrollees" (July 9-August). This descriptor does not require any adjudication. We currently refer to them as LD/SD and SD/SD because the AO team told us to. In the Lancet paper, they spent paragraphs on why this interpretation is justified.

With the AO interpretation, one of the trial findings is that people who received LD/SD treatment enjoyed a 90% vaccine efficacy compared to those who got SD/SD. A different way of describing the same analysis is that people who enrolled before July 10 enjoyed a 90% vaccine efficacy compared to those who enrolled later. Regardless of which description, this comparison is marred by built-in biases. As described above, later participants in the U.K. trial are older because of sequential enrollment. The condition of the pandemic was also evolving. Because of the accident, they don't have randomized comparison groups. A proper experiment that investigates dosage should have both groups run simultaneously, and participants randomly assigned to either half or full doses.

The AO interpretation depends entirely on accepting their explanation of how they discovered that the early shots were half doses, and how they knew which participants were given half doses, after the shots went into arms. They devoted many paragraphs to this but since the story is so complicated, I'll address it in the next post.

 

 

 

How to tell if someone is a serious data analyst

The unexpected turns in the pandemic have exposed which data analysts are serious, and who aren't. At the current moment, anyone who proclaims that vaccines are the cause of the dramatic receding of the coronavirus has proven to be not serious. They are analysts who embrace every nugget of confirming data while ignoring inconvenient observations.

What are some of these inconvenient data points?

• The number of cases has come down drastically in many places, including many that have not yet rolled out their vaccination campaigns. (e.g. India)
  
  
• The level of decline in many places far exceeds what is a reasonable expectation from the data observed in the vaccine trials. In almost every place, the vast majority of the population have not been vaccinated.
  
  
• Some organizations have issued deceptive analyses which the media should have flagged as potential misinformation. The much-reported analysis by the largest healthcare provider in Israel compared what they defined as effectiveness measured from second shot (2D) to 2D+7 days to the VE number from the Pfizer trial, which counted cases from 2D+7 days to a median two months later. In other words, the case-counting window of those two numbers do not overlap for even a single day! Talk to us in a month or so.
  
  
• If these providers are tracking all those they have vaccinated over time, they should publish all the data (from the first shot) as opposed to selecting a few days. That way, we can access how different the real-world experience is from what happened during the trial. (To be fair, I expect those two to differ because we're not measuring the same thing. The idea that those two should match perfectly is bonkers.)

It may be hard to imagine how complicated it is to get an estimate of real-world vaccine effectiveness. In a clinical trial, we have access to a cohort of people who are given two placebo shots and are being followed up alongside the vaccinated cohort. We don't have a placebo cohort in real life.

An unvaccinated person is not the same as someone who's been given saline shots. It's long been known that the placebo effect is real - you can give people sugar pills and some of them will get better. This means any real-world measurement is over-optimistic.

**Now I'm grabbing you with me while I jump into the deep end

We've identified a group of people who are unvaccinated. We even go one step further and match these people's characteristics to those who have been vaccinated. This means the age-group distribution is statistically the same across both vaccinated and unvaccinated groups, and so on. Remember the Israeli healthcare provider counted cases between 2D (second shot) and 2D+7.

Here's the catch. When is 2D in the unvaccinated group? You know what, we cannot define 2D because none of these people got any shots! In fact, only those who are symptomatic and get tested will show up at a facility - the rest are just people in databases with a "vaccination = No" flag. Take the unvaccinated person who just tested positive. Where in this timeline did the person test positive? How do we know the timing of this case is within that 2D to 2D+7 window?

The answer is we don't. It is not even possible to define this. Depending on what we define as Day 0, this person can be counted in any number of analyses. This is yet another reason why we need to look at longer time windows. Any analysis based on short time windows is junk.

***

One way to get around this issue is to match the timing. Take each vaccinated individual in the analysis population, find a "double" via demographic matching in the unvaccinated group, exclude anyone who has reported symptoms by Day 0 as defined by the vaccinated group, and then, mark off the corresponding 2D to 2D+7 window for each matched individual. As per protocol, if the unvaccinated double gets infected before 2D, that case cannot count. (If you don't narrow the case-counting window for the unvaccinated, then the analysis is hopelessly favorable to the vaccinated group!)

The key lesson is if you don't have experimental data, the data analysis gets a lot more complex (more stimulating, but also fraught with risk). The analyst must release full details of the analysis; otherwise, it's impossible to evaluate its merit.

 

 

 

Help me solve this puzzle

I've been trying to reconcile a graph shown in the Lancet paper covering the Astrazeneca-Oxford (AO) Phase 3 trials, and a table in Appendix 1 of the same publication. This gets at the heart of the illogical way in which the investigators of vaccine trials (not limited to AO) are counting "cases".

As I have described on this blog, all so-called primary analyses of any vaccine trial start counting cases X days after the second shot, where X ranges from 7 to 21 days depending on the developer. This counting rule is justified by the claim that the vaccines take 7 to 21 days after the second shot to induce enough antibodies to fight the virus.

However, most vaccine developers have also been using the same trial data to argue that the vaccine offers partial protection after the first dose - there are even scientists who claim that the first dose by itself is sufficient. Sharp-eyed observers will realize that such an analysis would be impossible if they start counting cases from 2D + X days! Shouldn't there be zero cases from 1D + 1 day up to 2D + X days?

Apparently not, according to this graph from the Lancet paper. Notice the x-axis is labeled "days since first standard dose". Apologies for cutting out the y-axis labels. It's the cumulative case rate, and each tick is 1% of the analysis population.

Notice the "exclusion period." Instead of counting all cases after the first dose, they start counting at 1D + 22 days.

***

Now note the size of the analysis population (indicated below the graph), 6307 receiving the vaccine and 6297 taking the placebo. Appendix 1 offers more details on how they determined the analysis population, starting with everybody that were enrolled into the trials.

The relevant TableS3 looks like this:

The two columns COV002 and COV003 are the U.K. and Brazil trials respectively. For the key AO analyses, the two datasets are combined ignoring the underlying differences. In each trial, about 60% of the enrolled participants make it to the analysis population.

What puzzles me is the 8th exclusion reason described as "PCR+ test <= 14 days post-second dose". (By the way, these are not reassuring numbers: in the U.K., the vaccine arm shows twice the cases as the placebo arm and in Brazil, it's about the same.) I just don't understand why something that happens post second dose should determine whether or not someone should be included in an analysis that starts at 21 days after the first dose.

Because the size of the analysis population aligns with what's shown in the graph, I can confirm that those cases are indeed removed from the analysis.

It is possible that they have an unedited typo: should the line have read "<= 21 days post-first dose"? I have no idea.

If it's not a typo, then it appears that someone who got sick between 1D and 1D+21 days would be included in the analysis population but those cases would not count. Then, those who got sick between 1D+21 days and 2D would be counted as both cases and participants (thus contributing to the graph shown above). Then, those who got sick between 2D+1 and 2D+14 days would be excluded completely. And they restart the tally from 2D+15 on. That's a highly convoluted, illogical way of including and excluding cases based on when they occurred in the timeline.

***

DETOUR: Going back to Table S3 for a minute. The primary reasons for being dropped from analysis differ markedly between the two trials. For Brazil, it's because the trial started later, and many have not reached 1D + 21 days at the time of analysis. For the U.K., it's because the analysts decided to drop the low-dose cohort (LD/SD) from this analysis. I've already written elsewhere about the many built-in biases to the AO results so I won't repeat that here.

DETOUR 2: The graph shown above is not an analysis of a single-dose treatment. There is no exclusion for not yet receiving the second shot so it is a mixture of people who has had both doses and people who has only had the first dose. In addition, that second shot was not well targeted to a particular day (or week) of the trial. If you get the drift here, there is not a single analysis in the AO paper that can be interpreted without examining underlying biases.

***

What is doubly puzzling is that the numbers shown in TableS3 differ from those found in TableS2, which computes the analysis population for the primary analysis, that which starts counting from 14 days after the 2nd dose.

In the U.K. trial, there are three times as many exclusions in the primary analysis than the secondary analysis while in Brazil, there are slightly fewer exclusions in the primary than the secondary analysis. The two trials again show their differences. I'd guess that the number of exclusions due to testing positive before 2D+14 should be higher for secondary analysis as that analysis includes more participants by definition.

Someone smarter than I must be able to make sense of the above. I'm stumped.

 

 

 

 

 

It's Tversky-Kahneman time: the law of small numbers

Amos Tversky and Daniel Kahneman coined the phrase "the law of small numbers" (1971) to describe the fallacy of "asserting that the law of large numbers applies to small numbers as well." Statistical thinking is based on the belief that a random sample is sufficiently representative of the underlying population, which is justified only when the sample size is large enough.

***

In recent weeks, some researchers and their media allies have touted the ability of vaccines to combat "severe disease". This is based on statistics from the vaccine trials for which they have "small numbers" indeed- literally, fewer than a handful of severe cases in the vaccinated arms in each trial.

None of those trials were sized for measuring severe disease. To measure rare events requires huge samples, for the simple fact that if the number of participants is not high enough, the most likely scenario is zero severe cases. When you've observed no events, it's very hard to give a precise estimate of the probability of the event occurring.

We already have evidence that the claim of 100% protection against severe disease is shaky. Of the trial results that I have studied in detail (Pfizer, Moderna and Astrazeneca), Pfizer's boasted the largest sample size of 42,000 participants across treatment and vaccine arms. In the Pfizer trial, the vaccine efficacy when counting only severe cases was 75% (1 case among vaccinated versus 4 in the placebo group). On the surface, this is already worse than what Astrazeneca is touting (100%).

What is the range estimate around that 75% in the Pfizer trial? This is not for the faint of heart, so only keep reading if you feel strong.

With 42,000 participants, we have enough data to arrive at a precision of... -153% to 100%. You read that correctly, the data are consistent with any VE between negative 153% to 100%. Simply put, the estimate of 75% is useless, meaningless, junk science.

Astrazeneca? Their analysis is based on 11,000 participants - pooled across two separate trials. This is one quarter of the sample size of the Pfizer trial. A back-of-the-envelope calculation shows that if eventually, we observed 1 vs 4 cases in the Astrazeneca-Oxford trial (same as for Pfizer), the width of the range estimate would double Pfizer's because of the smaller sample size. If the Pfizer number is junk science, then what is the right word for the Astrazeneca number?

***

Now that the vaccine journey has moved from clinical trials to roll-out, the nature of the data has shifted from carefully collected experimental data to conveniently scavengered observational data, and as a consequence, the media have been flooded with bad takes. I'll have a few more posts on this problem soon.

 

A day in the life of a data scientist who rescues the null clan from cancellation

I haven't written a despatch from the desk of the data scientist in a while but I had quite the stereoptypical data science moment yesterday so I'm sharing with you how I spent three hours of my life.

For a consulting project, I'm using a list of last names compiled by the government. As all data science stories begin, I must jump through a series of hurdles. The hurdles are set up in a dark tunnel; I can't see the next gate until I've jumped past the previous one.

The file of names comes in Excel and csv (text) formats. I spend some time investigating the data in Excel. Then, I upload the csv file to Python, where I do some heavy lifting. Once satisifed, I convert the data table back to csv which I intend to send to my client together with the code for the meat of the project. Processing the file of names is but a tiny step in a longer journey.

In the main code, the first step imports the modified csv file. Now, I'm merging this dataset with other files. To give you a flavor for this step, imagine the following made-up task. A third party has built a model that predicts someone's birthplace based on one's family name. For each name on my list, I want to attach the likely birthplace. This task is similar to a VLOOKUP in Excel.

Merging datasets is rarely uneventful. You need a match key, a column that exists on both datasets, and in the same format. If one column is in small letters and the other column is capital letters, nothing will match, and the resulting matched dataset is empty. If one dataset has compound last names (say, Smith Johnson) while the other uses hyphens (Smith-Johnson), those entries will not connect. These issues are easily spotted and fixed as you realize the data have disappeared after the merge.

What I'm battling is more wicked. Perhaps once in hundreds of entries, I find some wild results. Values are attached to other things that do not seem possible given the code I've written. Combing through the code reveals no obvious bugs. The results are just too outlandish.

From past experience with Python, I try rewriting the code fragment using a simpler data container (from data frames to lists, for those who care). A different error pops up. I feel like another block is pulled from my tower of Jenga. Please not let it collapse! The new error is a missing value in the list of names, in Python syntax, a "nan".

Ordinarily, blank data is an easy error to fix. Here, I'm mystified because the original source, the Excel sheet, does not contain any blanks. Perhaps I've erred when dealing with a catch-all entry called "All Other Names" (which is a bundle of the most infrequent names, obscured for privacy reason.) So I trace the data all the way back to the import step. I almost wish I erred... but that is not to be.

An hour into this work, I am downtrodden. I have a list of non-repeating names, e.g. [ADAMS, SMITH, WILLIAMS, ....], and when I turn this list into a "set", a missing value appears, e.g. (nan, ALSTON, APPLETON, ... ). The set operator should have just changed the data container but should not have inserted a missing value!

Let's take stock of my situation. I discover some wild values in my dataset after merging a file of names with some other dataset. While investigating it, I learn that somewhere in my code, one name has been converted to missing. I have yet to figure out how that missing value has crept into the data. But I know that merging datasets with missing values can create wayward output.

I plunge into the rabbit hole yet again. I pinpoint the location of the empty value inside my list of names. The list has come from a data frame. I determine the offending row, which shows a blank in the name column. The data frame is a copy of the modified csv file, which leads back to the original csv from the government site. On that original file, I strike gold.

The gold isn't the blank that I've been chasing.

Pulling out that particular row, I extract the value of the "name" field to be present!! Its value is the family name "NULL". Light bulb flashing. Some procedure in my code must have interpreted "NULL" as signifying an absent value. It's automation gone astray. In most cases, NULL is a placeholder for missing but in a file of last names, "NULL" is valid data.

Which procedure is my nemesis? It has to be the file import function that reads the csv file into a Python data frame. Consulting the manual, I learn that by default, the system "handles" potential missing values by assuming NULL, and other values such as NA, N/A, and #N/A as placeholders for missing. Users can turn off this default. I immediately, excitedly edit the code to circumvent the automated conversion. 

The blank is still there! I shift the order of the parameters. Nope, still there. I try editing other parameters that may affect how missing values are processed. The blank entry stubbornly refuses to go away. Perhaps, like a few other Python functions I've encountered, some of the functionality have not been implemented. What now?

I've found the culprit. I just can't fix it. I've spent way more time than expected on this little step. So I bow to reality and "hard-code" the fix. This means I add a line that says on line X, overwrite the "name" field with the value "NULL". This is la brutta figura.

***

The next morning, I open up the modified csv file using Excel. I am shocked to find the blank entry there as well. So, it isn't the file import procedure that has converted the name "NULL" to blank. That's why manipulating the import procedure has not fixed the problem.

I recall that the imported csv file isn't the original csv file from the government's site. It's been modified using some Python code. Finally, I solve the puzzle. The file import procedure is the villain but the null-to-missing conversion has happened on the first use of it, not the subsequent step.

Half the life of a data scientist is expensed on little, exasperating puzzles. Many of these riddles are side effects of previous automation, frequently regarded as "smart". At the end of these three hours, I want to berate myself for obsessing over an issue of such world-conquering significance as the rehabilitation of the NULL clan from the ignomity of being cancelled by software, a group numbering fewer than 8,000 in a country of 330 million. For my sanity, I am able to say the mangled data merge salvages the worthiness of this little issue.