Big Data, Plainly Spoken (aka Numbers Rule Your World)

Contaminated science

 

 

In one sense, the E.coli outbreak attributed to McDonald's has ended, as reported cases have dwindled to a drip. In another sense, it hasn't ended, because the number of cases is still growing.

This is the challenge of foodborne disease outbreak investigations. It is still possible that the theory that onions caused the outbreak is a false lead. In the most successful investigations, scientists were able to isolate samples of contaminated foods which contain the specific strain of the bacteria causing illnesses. An example of a very successful case is the spinach investigation featured in Numbers Rule Your World (link).

***

Late last week, CDC put out a new press release (link), updating the public about the current investigation. But it manages to unnerve me with terrible messaging.

The communication on November 13, 2024 carries the following conclusion:

"Epidemiologic and traceback data show that fresh, slivered onions served at McDonald’s were the likely source of this outbreak. Affected onions served at McDonald's locations were recalled and are past shelf life."

The sentences point to two types of evidence: epidemiological data come from questionnaires sent to reported cases asking for information such as what foods were consumed prior to getting sick from E.coli; and traceback data require salvaging potentially contaminated ingredients at source and testing them in labs for the specific strain of E.coli associated with the current outbreak.

We already knew about the epi data. What's new in this press release is the traceback data. Epidemiological data always come first, and were used to point the finger at McDonald's, then Quarter Pounders at McDonald's, then fresh, slivered onions in Quarter Pounders at McDonald's.

Scroll to the bottom of the press release, and this is the entire paragraph published about traceback:

FDA tested recalled onions and one sample tested positive for non-O157:H7 Shiga toxin-producing E. coli. The E.coli found in onions was not the outbreak strain and no human illnesses have been linked to this strain. Additional testing results will be released when available by FDA.

It's a word salad worthy of U.S. presidential candidates.

Translation: the investigators did not find the specific strain of E.coli associated with this outbreak in the onion samples they collected from the farms. We don't know how many samples they tested but only one of them tested positive for any kind of E.coli, and unfortunately for the investigation, it wasn't the right strain. Thus, this positive result does not strengthen the evidence against onions from those farms, and their customer, McDonald's.

And yet, whoever wrote the summary paragraph inserted the false claim that epi and traceback data were consistent, and complementary.

As a scientist, I find that this type of communciations undermines my confidence in our public health organizations. One can say this is an instance of "story time". They lull us in with some data, and then spin a story ostensibly supported by the data but not really.

Indeed, most media outlets have swept the uncertainty under the rug, referring to this outbreak as the McDonald's outbreak, or the onions outbreak. The CDC press release actually only used the term the "likely" source.

***

For giggles, I asked ChatGPT whether the lab test results added to the evidence that onions were the cause of the outbreak, and here is its answer:

No, the lab testing did not directly add to the evidence that the onions were the cause of the E. coli outbreak. The epidemiological evidence was already strong, with 99% of the interviewed individuals reporting eating at McDonald's, and 84% specifically eating menu items containing fresh slivered onions. The lab testing of recalled onions found a non-O157:H7 strain of E. coli, which was not linked to human illnesses and did not match the outbreak strain. Therefore, while the testing confirmed that the onions were contaminated with E. coli, it did not provide direct confirmation that the specific strain responsible for the outbreak was linked to the onions. The epidemiological data (such as the high proportion of sick individuals who ate the onions) was the primary evidence supporting the conclusion that the onions were the source.

I like this answer!

***

Note: It's possible that the lab testers knew something else that links those tested onions to the outbreak but nothing about the lab test results as disclosed in the press release supports its summary paragraph.

Note 2: In the meantime, the economic damage of food recalls is real. McDonald's is reportedly spending $100 million to recover from the fallout (link).

 

Sloppy language misleads

One of the most frustrating aspects of reading Covid-19 related publications is reading the conclusions. A summary of results is supposed to provide readers with the minimally necessary information to understand the full contents of the paper - but what we encounter is "story time": almost always, these conclusions are stated in the most general sense, ignoring all the caveats, the exclusions, the imperfections of the methodologies and analyses. They are "stories" that go far beyond the limited evidence presented.

I've been reading the Clalit study that was a major force behind the approval of the first booster shots back at the end of 2021/start of 2022. This paper made the following one-line conclusion:

Participants who received a booster at least 5 months after a second dose of BNT162b2 [i.e. Pfizer] had 90% lower mortality due to Covid-19 than participants who did not receive a booster.

Is it possible to misread this sentence? Yes, if you literally read the sentence.

***

If we know something about vaccine efficacy/effectiveness, we immediately recognize that these authors are selling the same story - that the vaccine has VE 90%, the only difference here being the outcomes are deaths attributed to Covid-19, rather than reported cases. As such, this VE is driven by the relative ratio of the death rates of the booster group vs the no booster group. They didn't use the term VE as it typically connotes cases not deaths.

A death rate is a fraction composed of two parts: people who are counted as part of the comparison group, and people within that group who died during the study period. In this post, I'll pay attention to both these parts.

From my last post, we already know one issue with the above statement. If we take any participant at the start of the study and ask whether this person is counted as part of the booster group or the no-booster group during the study, the answer, for 90% of the participants, is both. 

Since at the start of the study, 100 percent of the participants had not received the booster, and by the end of the study period (a mere 54 days), 90 percent of them got the booster, the researchers could not have divided up the study population into two distinct groups of people - each person (except the 10% who didn't get the booster during the study period) first appeared in the no-booster group, and then transitioned to the booster group.

But the concluding statement reads like the study enrolled two distinct groups of participants. This misreading is even more likely for knowledgable readers because ordinary studies, such as well-designed clinical trials, use distinct groups.

Note that I'm not criticizing the methodology up to now, I'm describing what it is, and criticizing how the language in the summary statements sends readers the wrong message.

***

The above detail may sound trivial but it has major implications. Think about the regression adjustments. As with all observational studies, researchers claim that by adding standard covariates to their regression models (age, gender, ethnicity, prior health conditions, etc.), they have corrected substantively all biases. For example, if older participants are more likely than younger ones to want the booster, rather than being assigned a treatment at random as in a randomized clinical trial, then it's possible that the age distribution in the booster group does not mirror that in the no booster group, i.e. an age bias exists.

Look at the setup here: Ninety percent of the participants appear once in the treated group and once in the not treated group. By the design of this study, there is little variation across the comparison groups in the distribution of everything, including age. If we check "covariate balance", it's balanced because the two groups are by and large identical. So this type of studies underestimates the variability of everything - which leads to p-values that are too small, and regression adjustments that are meaningless.

***

In the paper's conclusion, the phrase "participants who did not receive a booster" is incorrect in more ways. We already know almost all of them ended up getting boosters before the end of the study period, which was fewer than two months long. I have previously said that a participant transitions into the booster group when s/he takes the booster shot.

This is in fact not true. That's because this study also imposes a "case counting window," under the claim that the booster shot takes 7 days to become effective. In other words, if someone takes a booster shot, and falls ill with Covid-19 within the first 7 days, this case is not counted as a case in the booster group.

Negating this case is one thing - but this study (and many others) took another step: the case is not negated but it is counted as a case in the no-booster group!

How do they pull this off? The study design says not to transition the participants to the booster group until the 8th day after the booster shot.

So, let's review this. Each participant starts off in the no-booster group, then the person took the booster shot; this person is kept in the "no-booster" group for an additional 7 days before being moved into the "booster" group. During these 7 days, if the person gets infected or died from Covid-19, the case or death is counted as "no booster".

Remember that we can only talk about exposure time for this type of study since almost everyone is eventually exposed to both conditions. The exposure time of the "booster" group therefore excludes the first seven days after the booster while the exposure time of the "no booster" group includes the first seven days after the booster.

This is an instance of the additional bias that Lataster described in his comment on our Three Biases paper.

***

As the story goes, even the above description is inaccurate. We haven't dealt with the following puzzling disclosure:

participants who received the booster and had a confirmed case of Covid-19 within 3 days before the effective-booster date (defined as 7 days after the booster was administered) were excluded

The analysts further subdivided the seven days after the booster shot is administered into two parts: Days 1-4 and Days 5-7. Let's follow a participiant who did not have the booster at the start of the study. During the study period, this person took the booster shot. If the person got sick during Days 1-4, the person is counted as a member of the no-booster group with infection as outcome. However, if the person got sick during Days 5-7, this person is excluded from the study, i.e. not counted as part of the no-booster group, and not counted as a case. Finally, if the person did not get sick by Day 7, the person transitions to the booster group.

There is a subtle but important difference. The Days 1-4 treatment is applied to every participant, regardless of outcome. If they got the booster, they continued to be called "no booster" for seven additional days. The Days 5-7 treatment is an exclusion based on the outcome. If the person got sick, the person is removed while if the person did not get sick, the person is retained. For me, it is never acceptable to re-define treatment groups after observing the outcomes. And yet, this paper (and many others) engages in this practice.

To understand the impact of these data processing procedures, you can use the same device we used in the Three Biases paper: imagine a vaccine that is ineffective, apply the analysis, does it result in no effect, or something else?

This is a great example of why researchers must disclose the code used to process the data. It's hard to believe these special adjustments did not get the attention of reviewers. But I can think of no other way to interpret those words. (In fact, the waterfall chart in the paper confirms the exclusions.)

***

Now that you have heard how they measure outcomes and define "no booster", it's time to revisit the conclusion:

Participants who received a booster at least 5 months after a second dose of BNT162b2 [i.e. Pfizer] had 90% lower mortality due to Covid-19 than participants who did not receive a booster.

Does this sentence correctly summarize the study?

 

 

Story time: vaccinated people cannot spread Covid19

If you ask me for advice, I say get the vaccine, but do not let your guard down afterwards. Sure, go out more, and re-connect with friends but continue to wear masks and maintain distance in crowded places.

But some of you say, the CDC tells us if one is fully vaccinated, one can treat the pandemic as over. Throw all (pre)caution to the wind.

Let's dissect that recommendation. If you are fully vaccinated, you can still get Covid-19 but they claim you will not get severely sick. They encourage you to return to pre-pandemic living. So they must believe that fully vaccinated people with mild disease cannot spread the virus. They must also believe that fully vaccinated people with asymptomatic infections cannot spread the virus either.

These are important points because half the U.S. population is now fully vaccinated, and if fully vaccinated cases can spread the virus, then potentially lots of people are spreading the virus, invisibly. While we don't get severely ill, the more sick people roam around, the more the virus replicates, and the higher the chance that a rare mutation surfaces that causes another lethal wave of Covid-19.

Starting around April, self-described experts began making claims that "fully vaccinated people cannot spread Covid-19". For example, in this Johns Hopkins piece, an expert said "I think the preponderance of the evidence supports the fact that vaccinated individuals are not able to spread the virus."

***

A month or two ago, I gave a talk to University of Denver students, with the key message being "don't judge a research study by its abstract; learn what research methodology it used to draw its conclusions." See this post for more.

At this stage of the pandemic, three research methods dominate Covid-19 research: clinical trials, real-world studies, and lab experiments.

If we want to prove that vaccinated individuals are unable to spread the virus, what kind of study do we need?

We can immediately rule out the lab experiment. Testing blood samples for antibodies will never yield data on clinical outcomes such as symptoms, infections, transmission or spread.

Spread is a very hard outcome to assess. You need to first identify infected people, and then do contact tracing to figure out the transmission chains.

Can clinical trials yield such data? Absolute not, if we use the vaccine trials as they were designed. Those trials only tested participants for infection once every month or two, and most cases were self-reported, meaning that they only identified a fraction of the infections. Moreover, no contact tracing was attempted so we have no idea who transmitted the virus to whom. Therefore, these trials yielded no data about spread.

What about real-world studies? While we have big sample sizes for these studies, they also contain nothing about transmission chains. The aforementioned Johns Hopkins article cites this CDC study. I fail to see how the data proved that vaccinated people cannot spread the virus. The paper shows that vaccinated people are less likely to be infected (on this, I have concerns about inadequate debiasing of the data), and then they stretch that result to claim that the vaccine stops transmission.

Nevertheless, the study made no attempt to figure out who transmitted the virus to whom. For example, how do we know that one of the vaccinated and Covid-positive participants did not infect a dozen unvaccinated co-workers? I don't know the answer, nor do I think that is the most likely scenario but I fail to find any evidence supporting the claim that vaccinated individuals do not spread Covid19 in that study.

So what we have is "story time". Readers of the study are lulled to sleep by the nice dataset that was collected to measure overall vaccine effectiveness, and as they doze off, the researchers feed them the story of preventing spread, which has little if anything to do with the data.

 

P.S. [7/14/2021] In addition to story time, we have contradictionary evidence that gets ignored or dismissed. For example, the Las Vegas Review Journal just reported that 10 vaccinated health-care workers got Covid-19, possibly all during a pool party. Eight of these are fully vaccinated. Only 1 unvaccinated attendee got sick. So 10 out of 11 cases from that party were vaccinated. 

They only know about these cases because the health care workers decided to get tested. There are probably more cases because many attendees may not have gotten tested. Note, however, that fully vaccinated people are more likely to ignore mild symptoms and not get tested.

If fully vaccinated, PCR-positive people cannot spread Covid19, each of these 10 healthcare workers were infected by an unvaccinated, infectious person or persons. Even in this small example, we don't know who spread the virus to whom.

Regrettably, the Review-Journal printed a fallacy that I have pointed out from day the first clinical trial result was published. It said "In laboratory tests, the Pfizer and Moderna vaccines were about 95 percent effective in preventing symptomatic cases of COVID-19. That means that about 1 in 20 people could become sick after being vaccinated, though their infections would be less likely to be severe." In these trials, the case rate in the placebo group - people who got saline shots - was only around 1 percent. If their interpretation were correct, then the vaccine bumped the chance of getting infected from 1 percent to 5 percent - 5 times worse! See this post for how to interpret vaccine efficacy.

Science in the Age of Covid-19, Part 3

Here is Part 3 of my talk on research methods used in Covid-19 science. In Part 2, I point out the key differences between randomized clinical trials (RCTs) and real-world studies. In particular, real-world studies are subject to a variety of real-world biases. The quality of such studies hinges on whether the researchers make judicious decisions about how to correct for such biases.

In Part 3, I explain why even RCT results merit careful interpretation. RCTs are not perfect either, and these vaccine studies offer great materials to learn about the unintended shortcomings. The key topics covered are interim analyses, case-counting windows, staged enrollment, week-to-week variability, placebo effects, immortal time bias (again), conditionality, and story time.

Towards the end of these slides, I discuss the trend of "re-analysis" or "post-hoc analysis". This is one of the more worrisome trends in Covid-19 research. Post-hoc data mining is an about-face to prespecifying an analysis plan, and is the mother of spurious results.

Having run well beyond an hour of materials, I then very quickly describe another research method - using models and simulations. I may come back with a Part 4 to give this section justice. In the meantime, you can review the set of posts I wrote last year about the Oxford model (here).

Here are the links to Part 1, Part 2, and Part 3 of the talk.

Curve watching during the pandemic

The pandemic has revealed the importance of a most exciting area of statistics - causal inference from observational data. We've been inundated with all kinds of reports about the effect of [vaccines, masking, social distancing, school closures, outdoor dinings...] on the pandemic. Most of these "studies" focus on either one factor or one region or one demographic segment. Then, the headlines and PR coming out of these studies apply the specific results to the general public. Vaccines work! Masks don't work! Closing schools doesn't make sense!

The quality of the causal inference is tested when we apply the findings of these studies to other settings. So in today's post, I explore the claim that vaccination campaigns are responsible for the decline in cases seen in many countries.

In order to shut down our biases, I am presenting the graphs first without country labels. In each chart, the time period is from November 2020, before any vaccinations, to April 2021. I used data from OurWorldinData.org. The red line shows the new cases per million while the blue line shows the proportion of the population who have been vaccinated (defined as those who received at least one shot if it's a two-shot vaccine).

If we only look at individual countries, here are the stories one can tell:

Panel A presents a clean picture. Cases have tumbled from a peak in mid January to a level below that of November by end of April. The decline begun soon after initiating the vaccination campaign, which has reached about half the population. So vaccines worked!

In Panel B, this country's highly successful vaccination campaign has covered over 60 percent of the population by end of April. That appeared to be a critical mass, at which point cases dropped below the level of November. One can build a story about herd immunity.

The herd immunity story is shaky though. The 60-percent threshold does not apply to Panel A. Using the same logic, one would argue that the country reached critical mass even at 30% vaccinated. But at 30% vaccinated, the country in Panel B was approaching peak cases!

In Panel C, the sharp drop in cases happened as the proportion vaccinated rose from 20 to 40 percent while cases grew during the first two months of the campaign.

In Panel D, cases plunged within weeks of the start of the vaccination campaign, and reached a low plateau with fewer than 10% of the population vaccinated.

Then, you have Panels E, F and G. If one only analyzes those countries, then one comes to the opposite conclusion. In these places, the cases apparently have not been slowed by vaccinations. Over 60% of the people in Panel E have taken a shot but no sharp decline is evident yet. For Panel F, there have been a spurt in cases in April with the vaccination proportion nears 60%.

The picture is very confusing. That's because the vaccine is not the only factor affecting the rise and fall of cases. Any model will fit some of the data some of the time; a good causal model must work across multiple countries.

Now, I reveal the countries. These are the top 11 countries ranked by the proportion vaccinated by end of April. I excluded countries with low overall Covid-19 incidence. Seven of these countries have experienced a nice drop in cases per million.

***

It's also a good idea to take a peek at the other end of the spectrum. What's going on in countries that have barely begun their vaccination campaigns.

Here are 12 countries with very low vaccination rates but with case incidence in a similar range to the 11 countries shown above.

None of these countries have seen spikes in cases, almost all of them are seeing downward trends in April, many of them exhibit up-and-down cycles over these months, and a few have marked plunges such as Georgia in December and Montenegro since March.

***

Simple one-factor models aren't going to work to explain trends across countries and time. A good causal model should include a baseline trendline, a vaccine factor, plus lockdowns and other mitigation measures.

Another challenge for data analysts: the data post-vaccination will be hard to compare to past data. There will be fewer Covid tests among the fully vaccinated. Should cause of death be assigned differently if the patient is known to have been vaccinated? Will we see more deaths from co-morbidities? These are open questions - when I learn more, I'll report back.

 

 

How to rise above the cacophony of sometimes phony data analyses

Data analyses are flooding the airwaves. They paint a confusing, contradictory picture of all aspects of the pandemic. Sadly, most of the work are hastily put together, and low quality. Here are some things to keep in mind when you look at these studies.

What happened in an experiment will not happen again

If a vaccine trial returns a 90% vaccine efficacy, we can say for sure the vaccine’s efficacy is precisely 90% for the 30,000 or so participants in that specific vaccine trial. We are not empowered to conclude that the VE for billions of people who will eventually get the shots will be 90% - no matter how many epidemiologists repeat this falsehood on TV.

That’s because the trial involved a small sample of people, and there is a margin of error around the 90% number. If the range estimate is 70% to 98%, then we say “with 95% confidence” that the VE for the general population is above 70%. That’s still an impressive number, and it has the advantage of respecting scientific principles.

Imagine repeating the same clinical trial with different choices of 30,000 participants repeatedly. The margin of error says 95% of the series of VE estimates from these trials will fall inside the 70% to 98% range. The chance of any of these trials repeating the exact 90% VE is practically zero.

Credit: Jeremy Schultz/Flickr

 

The randomized clinical trial (RCT) is the gold standard for establishing cause--effect. The lack of randomization in “real-world” studies opens a can of worms.

In a vaccine trial, we compare the case rates of those who are vaccinated to those who aren’t, which are the primary ingredients of the vaccine efficacy formula. We are currently being served weekly new studies that also compare vaccinated people to unvaccinated people. The media are telling us that these new studies are better because (a) they are more recent (b) they have much larger sample sizes and (c) they constitute “real-world” evidence. A popular theme is that these studies fill in the gaps left by the vaccine trials.

The noise you’re hearing are the chuckles from incredulous statisticians. Since RCTs are regarded as the gold standard for causal inference, no “real world” evidence should be modifying, and certainly not correcting, findings from a scientific experiment. Doing so is like hiring a C student to tutor an A student, and siding with the C student when they disagree.

There is one critically important property missing from all real-world studies: the randomization of treatment. (See, however, so-called natural experiments.) In an RCT, a coin flip determines who gets the vaccine but in the real world, who gets the shots is anything but random. Most countries have priority lists and specific types of people are getting the vaccines earlier than others. In any “real world” study, the unvaccinated group is different from the vaccinated group, not just by vaccination status but also by many other factors, both known and unknown. Any of those other factors could contribute – majorly or minorly – to the observed difference between the two groups. Randomizing treatment ensures that on average, these other factors will not bias the finding in an RCT, a condition that does not exist in real-world studies.

A partial solution is to define a better control group. We’d take a subset of the unvaccinated people who look like those who are vaccinated. This is an effort to manufacture artificially the randomization condition. This matching process is highly subjective, and we can only match people using measurable and influential factors. There is no law that dictates that every important factor is measurable. As an illustrative example, perhaps people with higher-speed Internet connections are more likely to land vaccination slots. Since medical researchers do not have individual data on people’s Internet speeds, we cannot control for this effect. Even the most obvious adjustments have problems. For example, young people can be excluded because they don’t qualify for vaccination yet but young people who are front-line workers are getting shots.

Another common issue is incurable imbalance: if all care home residents have been vaccinated, we won't find care home residents in the unvaccinated group. The biggest problem with matching studies is lack of transparency. The typical disclosure is vague and confusing – I can’t even figure out what they did. None of these studies publish their data or code.

So, a real-world study that corrects for selection bias is better than one that doesn’t but in no case is an observational study superior to an RCT. Larger sample sizes produce more precise estimates (with lower margins of error) but they do not auto-correct biases. More precise estimates derived from biased data pose even greater risk precisely because they feel more solid when the biases are ignored.

The randomized clinical trial is a victim of abuse.

In reviewing the torrent of studies that have come out of the vaccine trials, I can’t help but notice that analysts are mercilessly abusing the RCT framework. I’m going to echo Andrew Gelman’s criticism of the standards of research in psychology. We’re concerned about the high probability of false-positive findings.

The VE numbers from Pfizer, Moderna, Astrazeneca, etc. are not comparable to each other. That’s because each team measures case rates differently. Pfizer drops all cases prior to 7 days after the second shot; Moderna and Astrazeneca drop cases prior to 14 days after the second shot. Then, you have people reanalyzing the data, who use their own case-counting windows. The U.K. government published a post-hoc analysis of the Pfizer data, counting only cases between 15 and 28 days after the first dose; this is echoed in a Canadian re-analysis of the same data, in which they set the window of 14 days to 20 days. A more recent “real-world” study out of Scotland counted from 28 to 34 days after the first dose. This is no small matter as VE is an improvement metric relative to a baseline, and we can’t keep our heads straight which the baseline applies.

Many scientists justified the choice of when they count cases, on record no less, by pointing to the cumulative case curve, arguing for counting when the vaccine’s curve starts flattening because that’s exactly how long it takes for the vaccine to do its magic. This type of post-hoc thinking is what teachers warn students not to do. These clinical trials were sized to measure the endpoint, and not sized to identify a turning point on a case curve. Imagine repeating this trial many times over. Assuming that the VE after 90 days rises to roughly 90 percent in each trial, it is likely that the turning point on the curve is not precisely at day 14 (or whenever it was in the first trial).

Analysts appear to have looked at their individual datasets, identified the period in which the vaccine showed the best performance against the placebo, defined their VE metric to hone in on that time window, and justified the definition with some sciency blather. This is a recipe for over-estimating the true performance.

The proof is the absence of new studies that go through similar post-hoc thinking to adjust down the findings of the respective RCT. In fact, the re-analyses of the Pfizer data, which moved the estimated VE of the first dose from around 50% to 90% is a nice example of a C student correcting an A student.

To prevent the danger of post-hoc theorizing, scientists agree to pre-register their studies so they have to define how they would count cases ahead of time. Unfortunately, those protocols are written loosely to allow a wide range of options, such as 7 days, 14 days, 21 days, etc. as possible evaluation metrics. This maybe exposes how little investigators know about the behavior of their inventions.

If only the start of the case-counting window were the only “researchers’ degree of freedom”. The researchers also follow their noses about (a) the duration of the case-counting window (b) whether a positive test is required to confirm cases (c) which test is allowed for confirmation and the parameters for running the test (d) the maximum length of time between reporting symptoms and testing positive (e) what symptoms are included and excluded as qualifying (f) how many symptoms are qualifying. As I showed in this post, VE drops fast with just a few more cases on the vaccine arm so any of these decisions matters.

The RCT framework does not protect us against these potential abuses. RCTs are the gold standard, but they aren’t fool-proof.

Story time after RCT

The potential problems outlined in the previous section relate mostly to trial design decisions. The vaccine RCTs are also being suffocated by post-hoc hacking. I already devoted an entire post to this phenomenon of “story time”. Not all results obtained from analyzing data from RCTs have the crucial randomization property. That’s because analysts can destroy the property by cherry-picking the data. We’re lulled into thinking these lesser findings have the same status as a proper RCT finding.

The most infamous example of such analysis is the 90% VE attributed to the so-called low-dose, standard-dose subgroup in the Astrazeneca-Oxford trial (link). This VE is indeed based on comparing case rates of a vaccine arm to that of a placebo arm. In a true RCT, those two arms are statistically the same except for what was placed in the two shots. Nevertheless, the low-dose subgroup happened as an accident, and it subsequently emerged that this subgroup contained only younger participants, have earlier enrollment dates, have longer intervals between doses, etc. And yet, on the front page of the Lancet paper summarizing this trial, the investigators stated “In participants who received two standard doses, vaccine efficacy was 62.1%... and in participants who received a low dose followed by a standard dose, efficacy was 90.0%...” By this point, there has been no disclosure about the “accident,” and anyone reading this sentence is likely to assume that there were two randomized dosage schedules used in this trial.

Likewise, much ink has been spilled on dose intervals. Many analysts have compared subgroups that took their second shots later and earlier. Even though the underlying data were collected in an RCT, dose intervals were not randomized. And yet, these results were published and publicized using methods for analyzing RCTs and not methods for analyzing observational studies. In some cases, the researchers even disclosed that these subgroups differed on numerous important factors, and still they plodded on.

**

In this era, even results coming out of RCTs must be scrutinized. Early analyses, post-hoc analyses, deep-dive analyses, and side analyses typically discard the crucial randomization property, and should be treated as observational studies.

Of course, results from observational ("real-world") studies should be scrutinized even more. In a future post, I’ll outline how to review analyses of observational data.

Story time in vaccine trial reporting

Some years ago, I started using the term "story time" to describe a common practice in research studies when they lull us with some data analysis, and then feed us conclusions that are unsupported by the data while we're falling sleep!

I'm starting to see "story time" in reporting on vaccine trial results. I suspect that much of this is unintentional, arising from a shallow understanding of the analyses. Here is a recent example from an Independent article about care homes in the U.K. seeing high levels of infection after a single dose of the Pfizer vaccine. The reporter stated:

Research suggests the Oxford vaccine does give some partial immunity after a delay of between two to three months but trials for the Pfizer jab did not compare different dose intervals.

This statement is partially true in the sense that the Oxford study provided an analysis of dose intervals while the Pfizer study did not (see my earlier post here). The story time is the moment when we infer that the dose interval was studied in a well-designed clinical trial.

***

The Oxford study was not originally designed to study dose intervals. In fact, the original design of the Oxford study was a single dose, which means there isn't such a thing as dose interval. In the middle of the study, the researchers added a second dose. By that time, a large proportion of the participants in the U.K. were already 30 to 40 days since receiving their first shot. This fact biases the study because it does not have much data on shorter dose intervals.

Like Pfizer (and every other study), the revision of the Oxford protocol stipulated a target dose interval (4-6 weeks). (Note that they later drew conclusions about 8 to 12 weeks.) There is a statistical reason for controlling the dose interval - we don't want to add new sources of variation to the outcomes, and dose interval is considered a relatively minor factor.

Somehow, the AstraZeneca staff could not control the dose intervals, ending up with some participants getting their second dose in "fewer than 6 weeks" and others getting it in "more than 12 weeks." (They never disclosed the entire range of possibilities.)  Therefore, they have accidentally attained variation in dose intervals. While this allows them to conduct an analysis of dose intervals, it has the side effect of muddying the headline efficacy analysis.

From there, they did an analysis of the effect of dose intervals on efficacy. Wait a minute, the dose intervals were not randomly assigned to the treatment population. In a proper randomized controlled experiment, several dose intervals will be pre-selected, and participants will be randomly assigned a dose interval. Since this is no longer a randomized controlled experiment, we cannot interpret the analysis as if the dose intervals were randomized.

When the dose intervals were not randomized, we cannot assume "all else is equal". The observed difference in efficacy may be caused by factors other than dose interval. It's not clear how they got people back for their second shots, and why they couldn't give people the second shot within their target window for the latter enrollees. Further, I am not convinced that you can compare vaccine and placebo groups with similar dose intervals without investigating selection bias.

The "story time" is to lull us in with a randomized controlled experiment and as we fall asleep, feed us less reliable conclusions that come from an embedded observational study.

 

[P.S. I have an ongoing series that digs into the details of the Astrazeneca-Oxford Phase 2/3 trial report. So far, the posts are 1 on overall efficacy and asymptomatic cases, 2 on protocol alterations, 3 on researcher degrees of freedom.]

One-dose vaccine strategy elevates PR over science

I fear that the U.K. policy of one-dose vaccines will backfire, and cause the pandemic to continue longer than necessary. Here are several reasons why.

Partial protection provides a convenient excuse for vaccinated people to do away with inconvenient mitigation measures

Assuming that the trial results hold, people receiving two vaccine shots have a very low chance of getting infected, and thus, it is acceptable that they return to normal living after vaccination. (I myself would not, since I think two doses are closer to 70 percent effective in reality, and we don't know that the vaccine stops asymptomatic spread, and so continuing to reduce contacts is advisable. Yet, under this scenario, one should not begrudge people who decide to return to normalcy.)

If people are given partial protection through just the first shot, it is irresponsible to return to normal living. Even the vaccinated must continue to wear masks, maintain social distancing, and stay at home as much as possible for the foreseeable future. Even now, it has proven impossible to enforce mitigation measures - at least not in the U.S. where I live. Vaccination presents a convenient justification for non-compliance.

This behavioral problem alone should have doomed the one-dose lobby.

One-dose treatment solves a problem that does not exist

The first weeks of vaccination has shown that the problem is demand not catching up to supply, not the other way round. We have more doses shipped than doses that are going into arms. If people don't like your cakes, you can't solve this problem by selling half-cakes!

The high efficacy of single doses is wishful thinking

Anyone who believes that the efficacy of the first dose is 90% lives in a fantasy world. Such a conclusion requires cherry-picking the best-case scenario, ignoring results that don't fit the theory, ignoring the margin of error, assuming that the second shot is essentially worthless, and assuming that the first shot is effective for the pertinent period of time without any boost from the second shot. The last two assumptions expose the one-dose strategy as a tautology - a conclusion that stems directly from one's assumption.

In a prior post, I explained how any guess of single-dose efficacy requires an assumption of (a) the efficacy of the second dose and (b) the duration of protection of the first dose, for neither of which do we have direct evidence coming from the vaccine trials.

It's a jigsaw puzzle, not a Barbie doll

People who propose the one-dose strategy are treating the clinical trial results like a Barbie doll. They are switching the green dress with the red skirt, or the high heels with flats. In fact, the toy is a jigsaw puzzle, in which each piece has its exact place, and cannot be haphazardly substituted.

For example, the headline vaccine efficacy numbers of 95% - recall the decimal obsession by Moderna - is based on a case definition in which they only count cases that occur at least 7 to 14 days after the second shot. You simply can't talk about that number in the same breath as efficacy for the first shot which requires counting cases after the first shot and before the second shot. They are not comparable numbers.

Use the scientific method properly

What supporters of the one-dose strategy should do is to lobby the FDA and/or pharmas to immediately start new clinical trials to test the one-dose strategy against the two-dose treatment.

From a scientific perspective, the biggest problem is post-hoc theorizing. Let's use stock market performance as an example. Do you feel like the business press are having it both ways? If stock prices went down yesterday, it's because the unemployment numbers were terrible. But if stock prices went up yesterday, it's because the terrible unemployment numbers were not as bad as imagined, or because businesses expected the job market to start improving, or because executives shrugged off the terrible unemployment numbers. They end up fitting a story to the observed data, as opposed to using the observed data to confirm a hypothesis.

Given the setup of the vaccine trials, before seeing these results, no one would have argued for one dose. The argument emerges after one sees the particular outcome from the trial. Trouble is, if the trial were run many times, would we have seen the same efficacy curve? (The variance across different trials already tells us the answer is NO.)

If the front part of the curve looks less favorable (say 30%), these same people would say we can't draw inferences about one dose (as opposed to one dose isn't going to work). In other words, this analysis contains an optimism bias, as does any theorizing after peering at the data.

***

Despite what I said above, I rate it as likely that certain countries will eventually follow the U.K. example and allow one-dose treatment. (As of now, the FDA has sensibly decided not to go down this path.)

Politicians score easy PR points by claiming they are helping "more people" when in reality, such a policy will lengthen the public-health crisis. Previously, I already noted that the PR agency of the UK government should be given an award.

 

P.S. Sadly, it appears that the incoming Biden administration is intent on elevating PR over science, as news has emerged that they will "release all available vaccines" and "not withhold second doses". This is euphemism for giving out one dose to everyone (as opposed to reserving the second doses for those who have gotten the first dose). For those "scientists" who are jumping behind this policy, I'd request that they answer the questions listed on my previous post. I can be persuaded that the one-dose strategy will lead to a better outcome but I'm having trouble understanding what set of assumptions lead to such a policy.

Added to the problems listed above, I think this latest news may persuade some of us to become vaccine skeptics. Those of us who believe the data show the vaccine needs two doses to work, and who do not get assurance that the second dose will be available in the right time frame, may decide to opt out of the vaccine.

 

P.P.S. In justifying this misguided policy, people are going to say "don't let the perfect be the enemy of the good". This is the cue that they are about to ignore the science. No one ever said the vaccine trial results are "perfect"; in fact, many of us believe they are less than perfect. I have written frequently about the shortcuts that were taken. We are not letting the perfect be the enemy of the good; we are opposing letting the unknown replace the good enough.

Other than some handwaving, I have not heard any description of what "the good" entails. If you can't quantify it, it's not science. If we are not going to argue about the science, then we must debate the assumptions, and evidence supporting such assumptions. Let's go!

Excess deaths, reporting delays, statistical adjustments, misleading headline

The New York Times published a very nice presentation on "excess deaths" in the United States from March through July. (link)

The following chart shows how public health policies utterly failed in many Northeastern states in the April and May.

In Connecticut, for example, in the middle of April, the number of deaths exceeded expected deaths by 2.6 times. In Massachusetts, around the same time, actual deaths exceeded expectation by 2.3 times.

A lot of work went into generating these charts - starting with meticulous data collection from each state, and as much as possible, reconciling how metrics are defined differently across states. Then, there is a bit of statistical modeling.

Modeling is required because "excess deaths" is - like most statistical quantities - not directly observable. Excess deaths is the difference between actual deaths (from all causes) in the presence of the Covid-19 pandemic and expected deaths (also from all causes) in its absence. The former is directly measurable but the latter is not.

To establish how many people could have died during a specific week in Connecticut if the novel coronavirus did not exist, statisticians look at deaths during the same week in that state in past years. Over a number of years, one should find some statistical consistency in the death rate. This illustrates the power of averaging.

If 2020 were a normal year, the chart should look something like this:

The above line is for Hawaii, which, because of its remote location, has not been affected much thus far by Covid-19. What you see is the actual weekly deaths hovering around the baseline (corresponding to the historical average), sometimes rising above, and sometimes dipping below.

The line for Connecticut, though, looks nothing like that for Hawaii. The most important feature is that the entire line sits above the baseline, meaning that there have been excess deaths for every week since March.

***

Ready for even more modeling?

For most states, the time line reaches the last week of July, which may surprise you. The first part of excess deaths - the observable part - counts all causes, obviating the need to verify coronavirus test results. The measure relies on death certificates. But it may take weeks to months to receive data on death certificates so how does NYT have estimates of excess deaths up to the week prior?

Does NYT have a crystal ball? In a sense, yes. It uses a statistical model to "top up" the incomplete deaths data in the most recent weeks. It turns out that actual deaths are not completely observable always. There is a reporting delay which means that the data for more recent weeks are incomplete. The data age like fine wine - the older the data, the more complete.

The NYT utilized a similar strategy as the model for projecting expected deaths - assuming historical patterns hold. The CDC keeps track of the amount of reporting delay. It might know that typically, 20 percent of the data showed up with one weekrepo's delay, 50 percent with two weeks' delay, and so on.

There is a crucial difference betweeen these two statistical models, though. The first model projects expected deaths in the absence of the novel coronavirus. The second model tops up the actual deaths reported at the time of analysis, essentially projecting the number of deaths already occurred but not yet reported. These actual deaths are in the presence of the coronavirus. This is the crucial difference. This second model has a further assumption - that reporting delays have not been affected by the pandemic.

The NYT team disclosed this assumption clearly in the text. This is what they mean by the following sentences:

Even with this adjustment, it's possible there could be an underestimate of the complete death toll if increased mortality is causing states to lag more than they have in the past or if states have changed their reporting systems.

It's quite likely that the volume of deaths, the budget pressure, and political interference have changed the pattern of the reporting delay.

The reporter then argued that the adjusted estimates are still better than the unadjusted ones. One can't refute that argument. Of course, this discussion assumes that the chart must show a time line leading to the present. The designer can choose to show only those weeks with virtually complete data.

Or, we can bring in a third model. This one looks at the early weeks of the pandemic, for which we believe almost all of the death-certificate data have been recorded. We can then estimate the error of the second model that serves to top up the actual deaths.

***

NYT's excess-death analysis has found over 200,000 more deaths (from all causes) in the U.S. between March and July than expected based on historical patterns. Given the U.S. counted about 140,000 deaths due to Covid-19 during this period, there have been about 60,000 unexplained deaths so far.

I find the NYT headline, "The true coronavirus toll in the US has already surpassed 200,000," alarmist. The analysis did not support this conclusion - unless they make the further assumption that the only factor causing excess deaths during those months is Covid-19. That is plausible but not fact-based. A more correct headline should be "Deaths due to Covid-19 may have been undercounted by up to 40 percent." That's also shocking and sobering.

How to act like a data scientist 11: do not assume random

Credit: ep_jhu/Flickr

 

You might find it curious, that someone who pointed out how testing drives the case numbers from my very first Covid-19 post (here), and who called for broad, non-targeted testing in this Wired column also from early in the pandemic (also here), has been silent on testing statistics. Specifically, I have not - until now - commented on the media's two favorite testing metrics: the number of tests per capita, and the "positivity" ratio, i.e. the proportion of tests that came back positive.

The reason? I quote from my post earlier in the week:

In the short history of the Covid pandemic, people started with case statistics. Then, they claimed that death statistics are less manipulated than case statistics, when they learned about testing. Then, they claimed that testing statistics are less manipulated, until they realized governments determined who got tested. Some governments counted tests shipped, not test results. Then, they claimed hospitalization statistics are less manipulated, until they learned that hospitals sent sick patients to nursing homes.

Commentators citing and interpreting these statistics behave as if they aren't manipulated. The President already contradicted them on that count. Manipulation is both intentional and unintentional. (I use unintentional intentionally to include intended unintentional.)

In my Wired column, I pointed out that one huge concern with non-random testing is it produces biased data that prevent us from understanding the pandemic. What makes this worse is when analysts speak about the statistics as if they came from randomized testing.

The positivity rate going up only tells you that of the people who were recently tested, more people have tested positive. If people who got tested were a random selection of the entire population, then yes, it indicates that the coronavirus would have spread more widely. A higher proportion of the population would have been infected.

This is not a given when the sample of people tested is non-random. Another reason why positivity rate goes up is more infected people are getting tested than before. (This is a broader statement than saying more people are getting infected.) Randomized testing almost never happens on its own - it must be organized. If the sample were truly random, you'd expect that someone who is not infected is equally likely to present themselves for testing as someone who is infected. Are you kidding?

Greater spread is one of several reasons for positivity rate to go up. Unless the analyst has examined the other reasons and found them lacking, then the conclusion is based on assuming greater spread, and assuming the other reasons do not apply. It is a tautology.

So, anyone saying positivity rate going up implies Covid-19 is spreading is doing some or all of: (a) assuming that Covid-19 is spreading, (b) assuming that the tested population is randomly selected, (c) assuming that the tested population has always been randomly selected [demonstrably wrong], (d) assuming that uninfected people are comprising a growing proportion of those being tested, (e) assuming testing data are comprehensive and accurate, (f) assuming no one is manipulating the testing data.

***

Regarding manipulation, we have indirect evidence, such as the U.S. President musing about slowing down testing. We also know that the UK government counted "tests shipped" rather than "tests results returned". This is like automakers counting "sell through" rather than "sell out". Sell through tallies include cars sitting in lots of dealers, and is part and parcel of the "channel stuffing" strategy to inflate short-term revenues. Sell out happens when drivers take ownership of the cars. Recently, the UK government stopped publishing testing numbers. In the U.S., I previously reported that California's testing numbers were fishy. I'm suspicious of testing data in the U.S. given the variety of public and private entities conducting tests (while being appreciative of those who put in effort to gather such data.)

The fact that few governments have implemented random testing, even to a small subpopulation, makes it clear that politics is interfering with the public health imperative. The side effect is suboptimal policies misled by low-quality data.

Recall what I said here: don't blame low-quality data for misguided policies when politicians have the wherewithal to improve the data quality.

***

For data scientists, this is yet another instance of garbage in garbage out. In an age in which the provenance of data is frequently unclear, it's worth our time to investigate and interrogate the data - and never assume random when we simply don't know.