Big Data, Plainly Spoken (aka Numbers Rule Your World)

Keeping up with the two books at CDC

In May, the CDC started to practice a form of "off the books accounting" (Here's a page describing creative accounting principles). In the U.S., we now have two sets of books when it comes to counting Covid-19 cases.

On May 1, the CDC introduced a new definition for "breakthrough cases." Breakthrough cases are cases that are found among the vaccinated people. Prior to May 1, all PCR-positive cases, including mild and asymptomatic cases, were counted as breakthrough if they occurred at least 14 days after the single J&J shot or the second mRNA shot. After May 1, only PCR-positive cases that result in hospitalizations and deaths within that case-counting window are counted as "breakthrough cases".

Because of the revised definition, the U.S. now has two tallies of cases among the vaccinated: one count includes mild and asymptomatic cases while the other count excludes them. (No such distinction is made for unvaccinated Americans.)

So if you ask how many Covid-19 cases there are in the U.S., there are two possible answers. The reality is even messier because each state may be reporting one or both counts to the national database.

That statement from the CDC implies that one of the two counts (the one including mild and asymptomatic cases for vaccinated Americans) is incomplete.

***

Consider media statements such as "95% of Covid-19 cases in the U.S. are now in the unvaccinated."

This statement is meaningless unless you're told which set of books was consulted. Were mild and asympotomatic cases in fully vaccinated Americans included in the total, or only CDC-approved breakthrough cases? (A separate but still pertinent issue is whether sufficient fully vaccinated people are tested to measure mild and asymptomatic infections.)

It appears likely that a data analyst computes the number of cases in the unvaccinated population by subtracting the number of breakthrough cases from the total number of reported cases. The total number of reported cases is typically the number of positive PCR test results, and therefore it should include all cases, including mild and asymptomatic, and regardless of vaccination status.

Now, pay attention to the following types of cases:
a) Mild and asymptomatic cases among the vaccinated that were reported at least 2 weeks after the last prescribed vaccine shot
b) Mild and asymptomatic cases among the vaccinated that were reported prior to 2 weeks after the last shot
c) Hospitalizations and death among the vaccinated that were reported prior to 2 weeks after the last shot.

CDC does not qualify these cases as "breakthrough cases" but all of these produced PCR-positive test results. The first statement means these cases are not counted as vaccinated cases while the second statement says they are counted as unvaccinated cases - if the analyst subtracts the breakthrough cases from the total cases.

Imagine that! Someone who is hospitalized two days after receiving the second Pfizer shot is counted as an unvaccinated case in this math. Someone who has mild disease a month after receiving the second Pfizer shot is also counted as an unvaccinated case.

What happens if the data analyst recognizes this problem, and wants to fix the math? You'd have to create a third bucket to hold all those cases - which are neither cases in the fully vaccinated nor cases in the unvaccinated. In the business world, this is known as "off the books accounting". You simply disappear these cases into the nether world.

In a follow-up post, I will estimate the scale of this off-the-books accounting using real-world data. Stay tuned.

Every headline produced by real world studies has been retracted and rewritten

Back in March, I reviewed several early "real-world studies" of vaccine effectiveness (here, here, here, here), including the Dagan et. al. study from Israel that eventually became one of the most frequently cited studies on the unreasonable effectiveness of mRNA vaccines.

The researchers of these early studies acknowledged the difference between observational datasets, and data coming from randomized clinical trials; specifically, the problem of selection bias inherent in the former but obviated by the design of the latter. Matching methods were deployed to reduce the effect of selection before generating a vaccine effectiveness number that mirrors vaccine efficacy in an RCT.

I also later reviewed several studies (Denmark, CDC, CDC 2) that utilized regression adjustments. Regression adjustment is an exercise in wishful thinking: for example, an age-adjusted estimate of vaccine effectiveness is the value of VE after removing the effect of age. Mathematically, the regression adjustment imposes a common age distribution on the vaccinated and unvaccinated subgroups, so age becomes a non-factor.

However, the age-adjusted regression estimate is relevant only if
(a) the estimate of VE for any given age range is reliable, and
(b) the specific age distribution imposed on both subgroups is realistic.

In the present situation, it is hard to believe either of those assumptions. For vaccination campaigns everywhere prioritize high-risk subgroups, particularly older people with or without comorbidities.

The headlines coming out of those early real-world studies were loud and clear: real-world evidence confirmed that the vaccines performed as well as and maybe even better than the clinical trials indicated. The media highlighted VE estimates of over 90% on all reported cases of Covid-19.

As a reminder, this was a CNN headline from February 2021.

The first sentence of the article cites the 90% number: "Pfizer-BioNTech's Covid-19 vaccine appears to reduce symptomatic coronavirus infections by more than 90% in the real world, Israeli researchers said Sunday."

If you want the other view, Fox News said the same thing.

***

By end of May, amidst reports of breakthrough infections, the story started to evolve. The new headlines now focused on VE estimates of close to 100% on reported severe cases, hospitalizations, or deaths. Readers were admonished for ever believing that vaccines should prevent disease, even though the primary endpoints for all clinical trials and early real-world studies were prevention of symptomatic, PCR-positive Covid-19 cases in the selected case-counting window.

Since June, the U.K., despite having one of the world's fastest vaccination programmes, began to suffer another surge in cases. The new headlines now drew attention to hospitalizations rather than cases.

The report cites a Public Health England study, saying "although the Delta variant reduces the effectiveness of vaccines against symptomatic infection, two doses of COVID-19 vaccine still protect against severe disease."

Israel, the early star in vaccinations, faced a similar scenario, and their reporters followed the same path, delivering headlines such as this:

It's not about infections anymore; only hospitalizations matter.

It's not just the headlines that changed. This new batch of studies employs different methods of analysis. Matching and regression adjustments were both sent to the scrap pile. The new analyses now wantonly ignore selection and other kinds of biases, and treat the observational data as if they were collected from randomized clinical trials. They assume the vaccinated and unvaccinated groups are balanced on all variables, an assumption not remotely plausible.

Here is an example of such a calculation, lifted from Matan Holzer's twitter account (see title of bottom chart):

In these type of studies, vaccine "efficiency" for hospitalizations is based on the ratio of the odds of hospitalizations for vaccinated vs unvaccinated to the odds of vaccination. The same ratio is obtained by dividing the hospitalization rate of the vaccinated by the hospitalization rate of the unvaccinated. This is the analysis that would have been performed to evaluate VE in a clinical trial - although those trials did not contain enough participants to attain statistical significance on the hospitalization metric.

Such analyses gave rise to headlines that proclaim that only unvaccinated people are sick enough to go to hospitals.

What are the data supporting the above headline (link)? It's the share of hospitalizations that are attributed to the unvaccinated people.

In the meantime, the following chart from BBC shows how the new wave of infections is  growing even faster than in the last surge, which happened before vaccinations.

***

By mid July, hospitals in the U.K. and Israel are admitting a rising number of fully vaccinated patients. So, the story is shifting again.

The previous metric comparing vaccinated and unvaccinated groups as if they were randomly assigned is abandoned. Experts now pretend that they did not introduce this analysis to the public, and criticize those looking at this number as a probability illiterate - failing to grasp Bayes' Rule. (here) Nate Silver chimed in:

I agree that the efficiency metric is poorly conceived, a fact that was as obvious in May as it is in July. I don't like the "story-first" mentality adopted by many scientists, which led to all those headlines about almost no vaccinated people getting hospitalized.

As the hospitalization story falls apart, the media is lining up behind the VE metric on deaths. We are being told that the vaccines have broken the link between infections and deaths, which reminds me of when they said the vaccines severed the link between infections and hospitalizations.

The math has been clear on this from the start: hospitalizations lag cases, and deaths lag both. We are going to find out in the coming weeks whether this latest narrative sticks

We need a post-mortem on the real-world studies, a honest evaluation on why these findings have low predictive value.

 

 

P.S. [7/23/2021] The same story is unfolding in Israel. Weeks ago, they happily cited naive estimates of vaccine effectiveness - first on infections, later on severe disease. Now that those metrics don't show what they used to show, suddenly those metrics are worthless. This Times of Israel article is highly informative.

Real-world vaccine studies through the lens of vaccine trials

Real-world studies are hard because observational data are packed with known and unknown biases. One thing is for sure: any analysis that doesn't correct for known biases is clearly flawed.

When discussing real-world studies of Covid-19 vaccine effectiveness, I have often lamented the ill effect of "case-counting windows". Recall that none of the vaccine studies count all cases from the day the participants enter the trial and get their first shot. In the case of Pfizer, all Covid-19 cases that occurred prior to the 7th day after the second dose were removed, in effect, nullified.

This procedure is feasible in the vaccine trial, when the same case-counting window can be applied to both vaccine and placebo arms. In real-world studies, the unvaccinated people do not take placebo (saline) shots, and therefore, it is impossible to locate 7 days after the second shot. As a result, the case-counting window removes cases only from the vaccinated group and not from the unvaccinated group.

tldr; Scroll to the end of the post

***

I'll show you the effect visually. We start with the cumulative case curve that was displayed in the Pfizer FDA Briefing document.

This chart is straightforward. As time passes, the number of cases grew on both vaccine and placebo arms. At the time of the interim analysis (which led to the emergency use authorization), the cumulative case rate for the vaccine arm was 0.25%, or 25 out of 10,000 was found to be sick while that for the placebo arm was 1.4% or 140 out of 10,000. Because of the trial setup, we can assume all else being equal, and so the vaccine is said to have cut the case rate by just over 80%. The vaccine efficacy is 80%.

But Pfizer said VE was 95%, not 80%. This is because of the case counting window. All cases on either arm that occurred prior to 2D+7 days are nullified. Like this:

This case-counting strategy knocks down the case rates for both placebo and vaccine arms. The placebo group now has a cumulative case rate of 0.8% at interim analysis, while the vaccine group has a cumulative case rate of 0.04%. The vaccine is now said to have cut the case rate by 95%. (But this number is not comparable to the 80% number above. The 95% efficacy applies only to people who did not get sick prior to 2D+7 while the 80% number applies to everyone in the vaccine trial.)

What happened is purely arithmetic. The second calculation is based on a subset of the data used in the first calculation.

***

Next, we do a thought experiment. We pretend that the data came from a real-world study instead of a clinical trial. We will therefore relabel the placebo group as "unvaccinated" and the vaccine arm as "vaccinated". We will now compute a vaccine effectiveness using the methods from real-world vaccine studies.

Here is the picture:

The cumulative case rate for the unvaccinated group is 1.4% while that for the vaccinated group is 0.04%. Thus, the vaccine apparently cut the case rate by 97% - almost perfect protection, even better than what was seen in the vaccine trial.

What just happened is that I took the orange line (for unvaccinated) from the first chart and the blue line (for vaccinated) from the second chart. The blue line is uncontroversial: given the dates of vaccinations of the vaccinated people, the analyst can compute how many days after the first shot someone got sick with Covid-19, and therefore apply the case-counting window, removing all cases occurring prior to 2D+7. This mimics the formula used in the vaccine trial.

When it comes to the unvaccinated group, the analyst could not apply a case-counting window. That's because the unvaccinated group did not take any injections - unlike the placebo participants in the vaccine trial who took saline shots. As a result, the analyst just counts all the cases in the unvaccinated group during the entire study period. This case rate is then compared to the case rate of the vaccinated - but restricted to various case-counting windows.

The beauty of this thought experiment is that it allows us to estimate how much pro-vaccine bias has been introduced by this case-counting window. On the second chart, we know that if case-counting could be applied to the orange line, the case rate would have been 0.8%. On the third chart, we know that the case rate was treated as 1.4% when case-counting became infeasible. Thus, the baseline case rate is inflated by 70% due to this asymmetric application of case counting.

Without a doubt, these real-world studies produce vastly inflated estimates of the true vaccine efficacy. (And here, I have only covered one of many biases in these studies.)

 

TLDR;

Real-world data are messy because unlike a clinical trial, there is no design to constrain biases. Real-world studies are evalulated on whether they have sufficiently corrected for known biases. One useful trick is to run the real-world study methodology through an unbiased dataset (e.g. data coming from a clinical trial or randomized experiment.) One hopes that the RW methodology yields the same answer as when one analyzes the clinical trial using techniques for analyzing trials.

The analysis here shows that the most popular real-world study methodology over-estimates the effectiveness of vaccines. The reason is that the case-counting window is applied symmetrically in the trial analysis but only to the vaccinated subgroup in a real-world study. This asymmetry causes the baseline case rate to be inflated by 70% (in the case of Pfizer), and vaccine effectiveness is the relative improvement over that baseline.

The case-counting window for Pfizer starts at 7 days after the second dose, meaning that all reported infections happening before that day are nullified when analyzing the clinical trial. In the real-world studies, these earlier infections are nullified only for the vaccinated group but not for the control group. This is a pro-vaccine bias built into real-world data. The only remedy is a correction performed at analysis. A first-order correction is to divide the unvaccinated case rate by a factor of 1.7 before computing vaccine effectiveness. (This correction however does not adjust for many other biases found in real-world data.)

 

Story time: vaccinated people cannot spread Covid19

If you ask me for advice, I say get the vaccine, but do not let your guard down afterwards. Sure, go out more, and re-connect with friends but continue to wear masks and maintain distance in crowded places.

But some of you say, the CDC tells us if one is fully vaccinated, one can treat the pandemic as over. Throw all (pre)caution to the wind.

Let's dissect that recommendation. If you are fully vaccinated, you can still get Covid-19 but they claim you will not get severely sick. They encourage you to return to pre-pandemic living. So they must believe that fully vaccinated people with mild disease cannot spread the virus. They must also believe that fully vaccinated people with asymptomatic infections cannot spread the virus either.

These are important points because half the U.S. population is now fully vaccinated, and if fully vaccinated cases can spread the virus, then potentially lots of people are spreading the virus, invisibly. While we don't get severely ill, the more sick people roam around, the more the virus replicates, and the higher the chance that a rare mutation surfaces that causes another lethal wave of Covid-19.

Starting around April, self-described experts began making claims that "fully vaccinated people cannot spread Covid-19". For example, in this Johns Hopkins piece, an expert said "I think the preponderance of the evidence supports the fact that vaccinated individuals are not able to spread the virus."

***

A month or two ago, I gave a talk to University of Denver students, with the key message being "don't judge a research study by its abstract; learn what research methodology it used to draw its conclusions." See this post for more.

At this stage of the pandemic, three research methods dominate Covid-19 research: clinical trials, real-world studies, and lab experiments.

If we want to prove that vaccinated individuals are unable to spread the virus, what kind of study do we need?

We can immediately rule out the lab experiment. Testing blood samples for antibodies will never yield data on clinical outcomes such as symptoms, infections, transmission or spread.

Spread is a very hard outcome to assess. You need to first identify infected people, and then do contact tracing to figure out the transmission chains.

Can clinical trials yield such data? Absolute not, if we use the vaccine trials as they were designed. Those trials only tested participants for infection once every month or two, and most cases were self-reported, meaning that they only identified a fraction of the infections. Moreover, no contact tracing was attempted so we have no idea who transmitted the virus to whom. Therefore, these trials yielded no data about spread.

What about real-world studies? While we have big sample sizes for these studies, they also contain nothing about transmission chains. The aforementioned Johns Hopkins article cites this CDC study. I fail to see how the data proved that vaccinated people cannot spread the virus. The paper shows that vaccinated people are less likely to be infected (on this, I have concerns about inadequate debiasing of the data), and then they stretch that result to claim that the vaccine stops transmission.

Nevertheless, the study made no attempt to figure out who transmitted the virus to whom. For example, how do we know that one of the vaccinated and Covid-positive participants did not infect a dozen unvaccinated co-workers? I don't know the answer, nor do I think that is the most likely scenario but I fail to find any evidence supporting the claim that vaccinated individuals do not spread Covid19 in that study.

So what we have is "story time". Readers of the study are lulled to sleep by the nice dataset that was collected to measure overall vaccine effectiveness, and as they doze off, the researchers feed them the story of preventing spread, which has little if anything to do with the data.

 

P.S. [7/14/2021] In addition to story time, we have contradictionary evidence that gets ignored or dismissed. For example, the Las Vegas Review Journal just reported that 10 vaccinated health-care workers got Covid-19, possibly all during a pool party. Eight of these are fully vaccinated. Only 1 unvaccinated attendee got sick. So 10 out of 11 cases from that party were vaccinated. 

They only know about these cases because the health care workers decided to get tested. There are probably more cases because many attendees may not have gotten tested. Note, however, that fully vaccinated people are more likely to ignore mild symptoms and not get tested.

If fully vaccinated, PCR-positive people cannot spread Covid19, each of these 10 healthcare workers were infected by an unvaccinated, infectious person or persons. Even in this small example, we don't know who spread the virus to whom.

Regrettably, the Review-Journal printed a fallacy that I have pointed out from day the first clinical trial result was published. It said "In laboratory tests, the Pfizer and Moderna vaccines were about 95 percent effective in preventing symptomatic cases of COVID-19. That means that about 1 in 20 people could become sick after being vaccinated, though their infections would be less likely to be severe." In these trials, the case rate in the placebo group - people who got saline shots - was only around 1 percent. If their interpretation were correct, then the vaccine bumped the chance of getting infected from 1 percent to 5 percent - 5 times worse! See this post for how to interpret vaccine efficacy.