Big Data, Plainly Spoken (aka Numbers Rule Your World)

How the CDC study handled biases in observational data

I offered a few high-level comments on the widely publicized CDC study of real-world effectiveness of the mRNA vaccines in my previous post. Today, I take a deeper dive into the study.

The main value of this real-world study comes from the weekly swabs requested from each participant. Unlike other real-world studies based on "found data", this CDC study is an organized effort with enrolled participants who agreed to send in swabs so their infection status can be determined each week. Of the vaccine trials I reviewed, only the Astrazeneca included this feature (although only for the U.K. trial and possibly only a part of that). These swabs reveal asymptomatic cases. As a result, the measured infection rate is likely to exceed what was observed during the vaccine trials.

The study reported an admirable compliance rate on those swabs. The median participant submitted all requested swabs.

 

Similar to the Danish study, the analysis population consists of specific higher-risk people. The CDC study focuses on healthcare workers and essential workers. Like all real-world studies, we must be careful when generalizing study results. It's one thing to say the vaccine was found to be highly effective for healthcare & essential workers; it's a completely different thing to claim that the study showed the vaccine to be highly effective for everyone (which is what the media and many "experts" have been touting all day).

Unlike the Danish study (based on comprehensive, found data), the CDC study does not include every healthcare worker or essential worker. Because of the enrollment requirement, the study population is self-selected so the first question to ask is whether the analysis population is representative of all healthcare workers and essential workers. There is nothing in the paper to answer this question. Table 1 discloses that 7 out of 10 people in the study are under 49, over 80% are white and non-Hispanic, and 70% has zero chronic condition (left undefined in the paper). The analysis population is thus significantly younger and healthier than our average American.

 

There exists two validity questions. The above deals with the difference between the analysis population and the general population. The next question of validity concerns the differences between the vaccinated group and the unvaccinated group within the analysis population.

There are two sentences in the CDC study that perfectly describes the challenges of any real-world effectiveness study:

For example, the infection rate in Miami, Florida was 8.6%, which is 65% higher than the average of 5.2% in the study. Twelve percent of the unvaccinated group work in Miami, compared to 3.5% of the vaccinated group. The infection rate in Portland, Oregon was 1 percent, much lower than average, and 90% of the Portlanders included in the study was in the vaccinated group.

Another example is occupation. The infection rate of primary healthcare workers was 2%, and over 90% of them were in the vaccinated group. Meanwhile, the infection rate of first responders was 9% (over 4.5 times higher), and 40% of them were unvaccinated.

Those two sentences suffice to explain the field of real-world studies. The researchers conclude that VE is very high, that is to say, the unvaccinated group has a much higher infection rate than the vaccinated group. They suggest that the vaccine is wholly responsible for the observed difference in infection rates. But any of the following statements can also be true:

• Some or all of the difference is explained by the over-representation of higher-risk males in the unvaccinated group
• Some or all of the difference is explained by the over-representation of higher-risk Hispanics in the unvaccinated group
• Some or all of the difference is explained by the over-representation of higher-risk first responders (and under-representation of lower-risk healthcare personnel) in the unvaccinated group
• Some or all of the difference is explained by the over-representation of higher-risk people living in Arizona, Florida and Texas (and under-representation of lower-risk residents of Minnesota and Oregon)

The two groups being compared differ not only by their vaccination status but also by gender, race, occupation and state of residence. The unvaccinated group has an over-representation of higher-risk individuals of each category.

If you recall my posts about the studies by Mayo Clinic or Israel's Clalit, this is the imbalance problem they solved by the matching procedure. In the CDC study, they ignored all these biases except for study site.

For studies using regression models, the usual corrective mechanism is to include adjustment terms, similar to the calendar time bias adjustment used in the Danish study. For more details, see my previous post.

In the CDC study, the only factor they adjusted for is study-site bias.

In a side comment, the researchers said they considered regression models that adjusted for sex, age, ethnicity and occupation ("individually"), and the change to VE was "<3%." They appear to be saying that those factors are unimportant because the change to VE is small. And yet, in Table 2, we learn that the study-site adjustment changed the VE from 91% to 90%, which is a difference of 1%.

I'd have preferred to see a model that includes all variables that can explain the difference in infection rates between the vaccinated and unvaccinated groups. Even if the effects of some of these variables are not statistically significant, they need to be in the regression model to obtain a better estimate of the effect of vaccination status - this is because of the correlation between vaccination status and those demographic variables (which is due in part to self-selection).

 

Curiously, the CDC study did not do a calendar time bias adjustment, and therefore, the bias identified in the Danish study is present. This bias originated from the drastic decline in infection rates during the first quarter of 2021. Upon vaccination, an individual migrates from the unvaccinated group to the vaccinated group. If we tally up the person-hours for the unvaccinated, they will skew towards the start of the study compared to those for the vaccinated group.

The time bias in the CDC study might be mitigated by the unusual enrollment cadence. While the study ran from mid December (start of vaccinations) to mid March, 60 percent of participants got their first doses in December. That leaves 40 percent. Since 25 percent remains unvaccinated to the end, only 15 percent got their first shots between January and mid March. So, the vaccinations were concentrated heavily in the first two weeks of the study.

This sheds some light on the self-selection bias problem described above. It seems that the 25 percent who remain unvaccinated chose not to get inoculated as there was plenty of time for them to do so.

Further, because the case counting window applied to the vaccinated group only, infections in the first few weeks of this study period can only appear in the unvaccinated group, and we know the overall infection rate was higher in those weeks.

Adjusting for these remaining biases will not wipe out the vaccine effectiveness but will provide a more realistic and believable measurement.

 

I realize I still haven't gotten to the "partial vaccination" analysis. That will appear in a future post.

 

 

What the Danish study tells us about the CDC study on real-world effectiveness

In a prior post, I noted that in a Danish study of real-world effectiveness of the Pfizer vaccine among nursing home residents and healthcare workers, the researchers applied what they call a "calendar time adjustment". This adjustment is necessary because most unvaccinated people-hours were accumulated in the early weeks of the study and most vaccinated person-hours occurred in the later weeks -- while the overall rate of infection was dropping precipitously throughout the study period. As a result, even if the vaccine were 0% effective, a simple analysis falsely concludes that it is highly effective (I show how the VE can be computed as 67%.) This is a real-life example of a statistical paradox known as Simpson's paradox.

Does "calendar time adjustment" cure the bias? It's hard to say because the researchers did not publish their model. It's not clear how they incorporated the adjustment. Is it a simple linear term? Is it an interaction term? Is time measured in days, weeks, or months? There are many possibilities. Regardless, the calendar time adjustment shifted dramatically the effect of vaccination from 96% (counting only cases from 2D+7) to 64%. (Bottom row of the table excerpt shown below)

But how is the "adjusted" VE to be interpreted?

In reality, this number can't be interpreted without knowing the structure of the model they used. So, I will describe a common type of adjustment, found frequently in the typical social science journal. This adjustment involves adding a linear week-of-study main effect to the regression model. It's one of the simplest possible tweaks.

In the regression model with adjustment, we assume two factors affect the ratio of infection rates: vaccination status, and week-of-study. (By contrast, in the unadjusted model, infection ratios are affected only by vaccination status. This works for a clinical trial in which other factors are balanced by randomization. It assumes no other factors are biased.) These two factors are themselves correlated: the later the week, the more likely someone is vaccinated. Using this model, the researchers found that the VE is 64% on the first week of the study period. The regression model also outputs an effect of week-of-study (not disclosed in the paper). For argument's sake, let's say this is - 4% per week (expressed in VE units). This means on the second week of the study period, the VE is 60%, and then 56% on the third week, and so on.

Notice that the drop in VE is 4% between any two consecutive weeks of the study. This is obviously false based on the non-linear drop in overall infection rates from week to week. But if the analyst specifies a linear model, the result is a linear effect. Linearity is a consequence of the structural assumption, not of the data. The above discussion builds in several other structural assumptions*. The success of the adjustment depends on the correctness of these assumptions.

Not adjusting for a clear time bias is definitely bad - and leads to a vastly overstated VE. It's just not clear whether the adjustment is good enough, since the authors did not disclose the structure of their model.

* For example, the reference week does not have to be the first week of the study; it could be the last week, or any week of the study. The effect of week-of-study can vary depending on the effect of vaccination status.

 

The Burden of Case-Counting Windows

I know that the 64% VE estimate is still too high for a different reason. This brings me back to a topic you've seen on my blog a zillion times - the infamous case-counting window. Take a look at the results again:

Just like every other study, these researchers computes four different VEs based on four different case-counting windows. As discussed in this previous blog post, the VE increases as you run down the table because more and more confirmed cases are removed from the case count, effectively turning cases into non-cases.

[The red highlight is my own. This is a significant negative VE observed in the two weeks after the first dose of Pfizer, the period that is carved out of most studies. I see this as another real-world confirmation that one dose does not give sufficient immunity.]

Below is a new graphic showing what researchers are actually doing to the case curve as they apply different case counting windows. (I'm taking the definitions of case-counting windows used in the Danish study and superimposing them on the Johnson & Johnson case curve).

In the clinical trials, the same case-counting rules are applied to both vaccine and placebo groups. It's simple enough because the placebo group got two placebo shots.

In a real-world study, the unvaccinated group are those who reach the end of the study period without having taken any shots. Therefore, a case-counting window such as "14 days after 1st dose until 2nd dose" has no meaning for the unvaccinated group. As a result, in real-world studies, cases are nullified only for the vaccinated group. For each of the four calculations in the Danish report, the same number of cases was used to compute the infection rate of the unvaccinated group while fewer and fewer cases were counted in the vaccinated group as you run down the rows. This issue alone vastly overstates the VE when compared to the same case-counting rules applied from clinical trials. [This problem is avoided in matching studies in which the unvaccinated people are assigned a "day 0". Each methodology has both pros and cons.]

The use of case-counting windows means that there is zero chance a real-world study can replicate the findings of a clinical trial. Another problem looms for future studies because Johnson & Johnson is a one-dose vaccine, therefore the frequently-cited window of 2D+14 (14 days after the second dose) is also undefined. I'm sure J&J will cry foul if researchers counted all cases after 1D+14 for J&J and only cases after 2D+14 for Pfizer! Alas, the unvaccinated people are not represented by a pharma although as far as we know, they do pay their taxes.

Nevertheless, I don't believe that the primary purpose of real-world studies should be to replicate the findings of a clinical trial - as I said before, this is like asking a C student to grade the work of an A student. Instead, real-world studies should be used to fill in the gaps of our knowledge.

 

On to Today's Hot Potato

Yesterday, the media breathlessly recited the headlines from a new CDC study of healthcare workers and essential workers, saying that the mRNA vaccines are 90% effective, and 80% effective for "partial vaccinations".

It turns out that this CDC study uses a methodology very similar to the Danish study. So we can apply what we learned from the Danish study directly.

What do those numbers mean?

1. Let's start with the positive. Participants in the CDC study submitted weekly swabs for testing, and therefore this study measures infections, including asymptomatic cases. (This complicates any comparison with clinical trial results which do not count asymptomatic cases. We should expect the VE to be lower than in trials if we believe asymptomatic transmission to be an important factor.)
   
   
2. The 90% vaccine effectiveness is computed for a specific case-counting window that starts from 14 days after the 2nd dose, and adjusted for study-site bias.
   
   
3. In the VE computation, the case-counting window can only be applied to the vaccinated group. Therefore, the infection rate of the unvaccinated group counts all cases in the entire study period. This makes the estimate over-optimistic.
   
   
4. During the study period (mid December to mid March), the overall infection rate was dropping precipitously in the U.S. as it did in Denmark. However, the CDC analysis does not adjust for calendar time at all. This is another source of over-estimation.
   
   
5. The unvaccinated group was heavily biased toward several states with much worse infection rates than the vaccinated group. The published VE estimate incorporates a study-site adjustment. (This is similar in concept to the calendar-time adjustment described above.)
   
   
6. The unvaccinated group comprises people who are different from those in the vaccinated group on numerous demographic variables. While they adjusted for study site (i.e. residence) bias, they did not correct the gender, race, and occupation biases.

I will be back with a longer discussion of the CDC study soon. The "partial vaccination" analysis requires more space than I have here.

The obsession with 90% is perplexing because a vaccine does not need to be 90% effective to be a useful tool to fight the pandemic. This fixation is creating the wrong impression that anything less than 90% is unworthy.

Note on a Simpson's paradox in real-world vaccine effectiveness studies

You have heard it in the media: real-world studies of high-risk people such as nursing home residents or healthcare workers offer definitive proof that the vaccines' true efficacy is <insert a percentage above 90%>!!

These were the people who got the vaccines first, they are at greatest exposure to the virus, more of them are fully vaccinated (two shots), they have the highest risk of death, they are easier to track down. Ergo, the result is rock-solid, beyond criticism!

I wish someone had the foresight, and did a quick randomized trial while delivering vaccines to these groups, then we'll have a good dataset to draw some useful conclusions.

Alas, no one did (as far as we know) - which means all these studies are real-world observational studies. I discussed two examples of these studies (Mayo Clinic, Israel's Clalit health fund), both analyzing everyone in their customer database using the matching technique. Matching is used to create a synthetic control group (after the fact) in the hope of emulating a randomized control group as in a randomized, controlled trial (RCT). After matching, we apply methods commonly used to analyze RCTs, as if the conditions of RCT have been satisfied.

Today, I'm examining a Danish study of the Pfizer vaccine in two high-risk groups, nursing home residents, and healthcare workers. (link). This is an example of those real-world studies of high-risk groups that have so excited the punditry.

Counting People No Longer Works

Let's start with a simple calculation, and I'll explain why it fails. The study period is from December 27, 2020 to February 18, 2021, just under 2 months starting from the day Denmark initiated vaccinations. Based on the Figure 1 at the back of the paper, I estimated that during this period, there were 1,675 confirmed cases in Denmark among the nursing home population. The researchers said 382 cases were discovered among the vaccinated nursing-home residents, so there were 1,294 cases confirmed for unvaccinated residents. (The actual numbers used in their calculations are slightly lower because of exclusions but this doesn't matter for what I'm about to say.)

During the study period, 37,172 nursing home residents got at least one Pfizer shot so the case rate for vaccinated is 382/37,172 = 1%. For the unvaccinated group, I use the total number of nursing home residents who have not been infected as of December 27, which was 39,050. So the case rate for unvaccinated is 1,294/39,050 = 3.3%. The ratio of those case rates results in a vaccine effectiveness (VE) of 70%.

This calculation sounds vanilla but has a serious problem. Each nursing home resident who is counted as vaccinated is also counted as unvaccinated - because on December 27, 2020, the first day of vaccinations, everyone is unvaccinated. Then, over the course of time, those who have not been infected are eligible to get vaccinated. Upon vaccination, the nursing home resident transitions from being counted as unvaccinated to being counted as vaccinated. By the end of the study period, only about 5% of the residents are still unvaccinated. So 95 percent of this population are counted twice, and in each case, the persons spend the first part of the timeline as an unvaccinated and then the second part of the timeline as vaccinated.

Moreover, the proportion of time spent as unvaccinated varies depending on the day of vaccination. To deal with this issue, we change the denominator of the case rate. Instead of cases per person, we use cases per person-week. Each person is still counted twice, but in any given week, a specific resident is counted just once, either on the vaccinated arm or the unvaccinated arm but not both.

There were roughly 148K person-weeks accumulated on the vaccinated arm, and 164K on the unvaccinated arm, leading to new case rate estimates of 0.26% and 0.79% (interpreted as per 100 person-weeks). This leads to VE of 67%.

(In the second half of this post, I'll show why this 67% number is compatible with a completely useless vaccine. Stay tuned.)

Matching No Longer Works

For any real-world study, we ask why we can believe that the group labeled unvaccinated and the group labeled vaccinated are identical except for their vaccination status. In prior studies I featured on this blog, the researchers rely on the matching technique to assert this claim. Matching is infeasible here because the entire study is centered on two high-risk groups, for whom the progress of vaccination was rapid. Among the nursing home residents, about 5 percent were vaccinated during the first week, reaching almost 95% by the end of the study period.

The Danish researchers treated this issue differently from the Clalit researchers.

In the Israeli study, the investigators matched each vaccinated person with an unvaccinated one on a rolling basis through the study period, and they ended follow-up on both members of each pair when the unvaccinated person received the first shot. This matching and censoring scheme creates many undesirable side effects (for details, see my prior post):

• the follow-up time is severely limited because most matched, unvaccinated patients quickly became vaccinated, stopping the count on both members
   
• cases on the vaccinated arm are not counted because of the early stopping described above
   
• the number available for matching shrinks rapidly throughout the study period, leaving the majority of high-risk subgroups unmatched, and dropped from the study.

By contrast, in the Danish study, few are dropped from the study (only those who have been infected prior to the first day of vaccination). Each person is followed through the entire study period (until they get infected, died or become untrackable). For anyone who eventually got vaccinated, the person contributes to unvaccinated person-weeks until the day of vaccination, after which the person contributes to vaccinated person-weeks. They don't have extra early-stopping rules triggered by something that happens to an unvaccinated person so every case will be counted (except when they play around with case-counting windows, which I'll cover in a future post).

But there is one big problem with this design, which the researchers recognized and tried to resolve.

Vaccination Timing Bias

Because of the design described above, cases counted as unvaccinated typically occur earlier in the study period than cases counted as vaccinated. If you think about any given individual, the time progression is always from unvaccinated to vaccinated. So earlier infection dates are correlated with being unvaccinated, and later dates with being vaccinated.

This is particularly problematic because vaccinations were happening in the backdrop of a general decline in cases post-holiday, at least partially related to a national lockdown that started in Denmark a week before vaccinations began. To deal with this bias, the Danish study added a "calendar-time adjustment" to their regression model.

Unfortunately, such an adjustment is better than nothing but isn't sufficient. In the rest of this post, I'll walk you through a numerical example to show how this timing bias is a real-world example of what statisticians call a "Simpson's paradox". Then, in a next post, I discuss what a regression adjustment does, why it doesn't get rid of the problem, and also address other residual biases.

Simpson's Paradox

For this demo, I start with an assumption that the vaccine is useless (VE = 0). Then, I show that using the usual way that we compute vaccine effectiveness, we will conclude that the vaccine is extremely effective. This "paradox" is of the Simpson's type, well known to statistics students.

As mentioned already, the Danish study attempts to correct for this issue. This demo explains why they must do something about it. It also provides an estimate of the magnitude of this bias (huge).

As background, we start with some real data taken from Figure 1 of the paper: the progress of vaccinations during the study period, and the weekly trend of cases in nursing homes.

Note in particular that the case rate (gray) dropped markedly over the course of the study.

Assuming VE = 0 means each week, the split of cases between the vaccinated group and the unvaccinated group is equal to the split of people between the two groups. This assumption makes the group-level case rates equal each week. The VE in any week is zero since VE is based on the ratio of group-level case rates. Like this:

So far, so good. Now, we compute a cumulative VE using the standard method. We sum the cases and the person-weeks over all weeks, and then divide the two to get an overall VE for the entire study period. This is exactly the calculation I did in a previous section, which led to a VE of 67%.

I hope your head is where mine is: WTF!

We forced VE to be 0% every week of the study period (by assumption), and yet the cumulative VE for the entire period turns into 67%. This is known as a "Simpson's paradox". The aggregated statistic leads to a startlingly different conclusion than the disaggregated (weekly) statistics.

This is the reason why the Danish researchers introduced a "calendar time adjustment" to their analysis. They realized that they must do something to overcome this paradox.

What's Lurking Behind the Scene

We expected the cumulative VE to be 0% since it's just the average of a bunch of zeroes. The problem is it's not a simple average but a weighted average. The weights reflect the progression of vaccination through the study period.

The unvaccinated group is front-loaded, and shrinks over time while the vaccinated group is back-loaded, and expands. So the aggregate case rate for the unvaccinated group is weighted toward the first few weeks while the aggregate case rate for the vaccinated group is weighted toward the last few weeks. Remember that the overall weekly case rate was high in the first two weeks, and then started declining. This explains why the unvaccinated group with a much higher cumulative case rate than the vaccinated group despite our assumption that the vaccine is useless.

In the above diagram, you see that for the unvaccinated group (pink), the largest weights coincide with the several weeks with very high case rates while for the vaccinated group (blue), the largest weights fall on the most recent weeks with close to zero case rates. Simpson's paradox is a situation in which aggregating the data removes too much information, and distorts the interpretation.

The magnitude of this distortion is extreme! What should have been 0% shows up as 67%.

(In the preprint, I only have the proportion vaccinated at the start and end of the study. In the demo, I drew a straight line, which means I assumed an even pace through those weeks. The reality is that the earlier weeks were probably slower than the later weeks, which makes the distortion even worse.)

***

The situation of rapid vaccination and declining overall case rate are prevalent everywhere so every real-world study, particularly ones focused on high-risk groups, faces this problem.

This analysis does not show the vaccine is useless. It shows that if the researcher does not correct for this timing bias, the estimate of vaccine effectiveness is extremely optimistic.

***

My second book, Numbersense, opens with an example of Simpson's paradox, and then it keeps getting better. Get your copy here.

In the next post, I explore how the Danish researchers attempted to fix this bias.

 

 

 

 

 

 

 

 

 

 

Probabilities of Covid-19

 

Astrazeneca Clears the Clot

In yet another badly-kept secret, the blood-clot investigation by the EU found the Astrazeneca vaccine still safe and effective for the general population based on a cost-benefit analysis. But they also said that the vaccine may be linked to a rare type of blood clot with low blood platelets, in general but also in younger people (under 55) in particular. They will update the product information to mention this possible risk. This is exactly what I expected.

The EMA did not provide any details on the general analysis but using Spiegelhalter's calculation (referenced in my last post on this subject), it appears that about several hundred cases of blood clots should have been observed since vaccinations began in Europe, compared to 30-40 that have been documented. In the rare blood clot analysis, they disclosed two calculations, "less than 1" vs 5, and "1.35" vs 12 (expected vs observed). Both calculations ignored cases that happened in the 14 days after vaccination (presumably the first shot although the press release did not specify), and weirdly, the age group for the analysis is "under 50" while it is "under 55" in sentences that do not cite data. The number of cases among the subgroup under 50 is smaller or equal to the number of cases among under 55. Those incidence ratios are huge, even if the counts are small.

The press release has another inconsistency. Up top, it says "most of [the rare type of blood clots] occurred in people under 55 and the majority were women" and in the middle, it says "the reported cases were almost all women under 55", and then later, it says "the majority of reports involved women under 55". Logic requires that if almost all were women and under 55, then it's not just the "majority" were women but "most" were women. I hope they clarify the record.

tl;dr For the vast majority of people, blood clots caused by vaccination is not a concern (based on data up to now) but a few unlucky people under 55 may develop a rare type of blood clot after vaccination that has led to deaths. Based on clinical trial results, the vaccine would have prevented many more deaths through Covid-19 by interpolating the VE, and so they continue to recommend its usage.

 

A Different Way to Think about Vaccine Efficacy

A loyal reader, who's a doctor out there saving lives, has plodded me many times to do a back-of-the-envelope calculation. I finally did, so I'll describe it here.

In the Johnson & Johnson trial, about 2 percent of the placebo group got infected in roughly two months. (This means 98 out of 100 participants who got saline shots in the trial have not been infected within the first two months, most of whom probably haven't been exposed to the virus yet. In earlier trials, the number is even higher.) Let's round the J&J vaccine efficacy to 60% for ease of calculation. That means the infection rate of vaccinated people is expected to be cut by 60%, from 2 percent to 0.8 percent over 2 months.

It's a little easier to think in terms of 12 months so we multiply the rates by 6. The baseline rate of infection over a year is 12 percent, and vaccination brings it down to 5 percent. If we (fully) inoculate 100 people, the expected number of cases should go from 12 to 5, so there are 7 fewer cases per 100 vaccinated. Flip that number around. What it means is for each averted case, we need to vaccinate 100/7 = 14 people. 

If your glass is half full, you'd say this is actually not bad because there are many common medicines out there with far worse effectiveness. According to Stat News, 60 people must take statins for one of them to avoid a heart attack.

If your glass is half empty, you'd notice that of the 14 who got inoculated, we expect just 1 will get a real benefit. The other 13 would have the same fate after a year, whether they got the shot or not.

The catch is we can't predict which of the 14 will benefit. We know that if those 14 did not get vaccinated, they will - with certainty - not benefit. If they take the shot, they've purchased a lottery ticket with 1/14 chance of winning and 13/14 chance of breaking even.

I just portrayed that too favorably. Some of the 13 will not benefit but will get harmed by the side effects. Buying that lottery ticket brings a small chance of making your health worse. To learn more about this, read my 2010 post on Avandia, the diabetes drug, here.

 

Bonus Time

If you're still reading, here's a bit more.

The computation above is called "number needed to treat" (NNT). This number depends on the baseline rate of event.

If you live in a country with a much lower rate of cases, then the NNT is much higher. For example, if the baseline infection rate is 10 times lower, 1.2% per year instead of 12% per year, then the expected infection rate for the vaccinated is also divided by 10, thus 0.5%. So for every 1,000 people vaccinated, we would have 12 - 5 = 7 fewer cases. So the NNT is now 1000/7 = 143. This country has to inoculate 143 people to reduce the case count by 1.

The media love to talk about the vaccine's impact on the death rate. The Covid-19 death rate is even lower, more like 0.2%. Let's bump up the efficacy to 90% so the death rate among the vaccinated is 0.02%. Out of 10,000 vaccinated people, we expect 20 - 2 = 18 averted deaths. The NNT for deaths is 10000/18 = 1250 556.

 

 

 

 

 

Blood clots: is there a there

 

A number of mostly European countries have suspended administering the Astrazeneca-Oxford vaccine in order to investigate whether it has led to blood clots. These decisions have aroused angry reactions from the media. Here's an opinion column by Prof. Spiegelhalter that captures the mainstream view.

Many reporters have mis-spoken when they say the countries have banned the vaccine. They are pausing new inoculations so that the health authorities can assess the situation.

Spiegelhalter's argument - which was also that of Astrazeneca - relies on comparing the incidence rate of blood clots among those who have been vaccinated (millions) and the incidence rate of blood clots in the general worldwide population. He estimates that about 100 people in any given week experience blood clots, while so far only 30 cases have been reported post-vaccination (across many weeks).

As with any statistical argument, we should outline the unspoken assumptions under which this argument will prevail.

First, we must assume that the system used to collect information about blood clots is reliable and mostly complete. Spiegelhalter says the "yellow card" system used in the U.K. for reporting side effects has produced far fewer side effects than during the clinical trials. He interprets this as evidence that the vaccine is even safer than advertised. Another possibility is that the side effects are being under-counted. The hard part is to figure out by how much.

Second, we must assume that the people who have been vaccinated so far is a random sample of the entire worldwide population. On second thought, it's not obvious that the rate of blood clots should be the same in every country, or every age group. There may also be seasonality in the condition - the data on the vaccine has come only from the spring season while the general data are averaged over the year.

Third, we must assume that the characteristics of the people who have reported blood clots are similar to the average person who gets blood clots in an average year. It's worth noting there are reports of unusual blood clots in young vaccinated people. The problem may be blood clots in young people, not all people.

Fourth, we must assume that all blood clots are alike. Are there different types of blood clots? Are the reported blood clots similar to the average blood clot? Depending on the answer, we may need an incidence rate for specific types of blood clots (and also for specific age groups).

Fifth, we must assume the outcomes of those who reported blood clots post vaccination are similar to those in any given year. In particular, several deaths have been reported. Is the rate of death among those who experienced blood clots abnormal?

Sixth, I'd like to tally the incidence of blood clots in the unvaccinated populations as well. According to Spiegelhalter's back-of-the-envelope calculation, the incidence of blood clots is actually far fewer than expected (only 30 cases in several weeks vs 100 weekly), and so we should expect the data to reveal that the chance of experiencing blood clots in the last few weeks was much higher among the unvaccinated than among the vaccinated - assuming those weeks were normal weeks. (If this proves true, it raises the prospect that vaccination is protective against blood clots!)

Seventh, we must also assume that there is no one-time anomalous event that is causing a spike in blood clots. Some reports claim that there might be a bad batch of vaccine. It is possible for both of the following to be true: that the Astrazeneca-Oxford vaccine does not cause excessive blood clots, and that a particular batch of the vaccine caused an excess of blood clots among those received those doses.

Eighth, we must assume that the several weeks of data collected about the vaccine are fully representative of ongoing data. There is a precautionary tale here. At any moment in this pandemic so far, if we took the cumulative data and annualized them, we would have produced horrible forecasts of the future.

***

That's a lot of assumptions and the list is not necessarily complete. What we are dealing with is another observational (i.e. real world) study. Without the randomization of randomized clinical trials, it's just really hard to connect an effect to a cause.

Making assumptions is not fatal in statistical analyses - on the contrary, every statistical argument contains some set of assumptions. If I were conducting this investigation, I'd focus on collecting data to check the above assumptions. I believe the mainstream view may prove correct; this would imply most of the above assumptions hold.

It is also true that the current known incidence rate is small, benefits outweigh the side effects, etc. Notice that this is a different line of argument. The statistical argument claims the blood clots are normal and thus there is no problem at all. This second argument accepts the blood clots as abnormal but an acceptable price to pay; it is unnecessary if one agrees with the statistical argument.

***

I don't think it's wrong for European politicians to take a cautious approach. I wish the experts were not all over the media slamming the politicians and citizens about their alleged failure to understand science - because such ferocious rhetoric is drawing unnecessary attention to what should be a routine process. Just say you support a quick investigation, Astrazeneca should cooperate and provide any necessary data, and you expect the vaccinations will resume in short order because this is likely a false alarm.

 

P.S. [3/17/2021] Just listened to the hour-long press conference by the European Medicines Agency outlining the process of the investigation. They clearly state that they don't believe the extent of thie problem would ever outweight the benefits of the vaccine, but they also say the specific circumstances of those cases require examination. They expect to come to a quick conclusion, by Thursday afternoon.

Many of the issues mentioned in this post were brought up, such as specific types of blood clots, batches of vaccines (or manufacturers), and establishing base rates for specific time frames. The committee of experts will be looking into those and other factors. One reporter asked an important question - whether there will be a biological investigation to go along with the statistical analysis.

The scientist representing EMA mentioned the back-of-the-envelope calculation that would indicate that people who were vaccinated were far less likely than normal to have blood clots. I don't think this is their strongest argument; unless the vaccine somehow prevents blood clots, you'd have expected the frequency to be roughly normal - and if it's not, you now have to go down the rabbit hole of explaining the biases in the vaccinated population, which I can assure you opens up a can of worms.

Also worth noting (which I did in a comment below) is that both sides could be right. It may be that the vaccine causes excessive, potentially fatal blood clots of a specific type in a specific subpopulation. That means the vaccine is still safe for almost everyone. This bolsters the cost-benefit argument. The end result may be just a warning about a rare side effect for people with specific conditions.

 

Take 2 on the Johnson & Johnson vaccine trial results

In the previous post, I characterized the vaccine efficacy (VE) of Johnson & Johnson's vaccine (J&J) as 67%, which is the headline number in the FDA report.

The VE Constellation

If J&J had offered a single estimate of vaccine efficacy, we'd have been shocked.

They centered our attention on 67%, which is described as the VE for "centrally-confirmed moderate to severe/critical Covid-19 cases" with a case-counting window starting 14 days after the first shot (1D+14).

The following list is a partial run-down of other VE estimates that also appeared in the report, and how they relate to the headline number.

If the case-counting window is shifted forward to 1D+28, the VE becomes 66% (lower by mere decimals). This estimate throws out another two weeks of cases, and was added to the clinical trial protocol midway through the trial. I can't think of a reason to justify this metric, other than to compare it directly to Pfizer's headline number (2D+7 = 1D+28).

If that's the case, we also need J&J's VE starting the count at 1D + 35 to compare to Moderna's headline VE at 2D+14. This is not directly cited in the report but it's still 67%.

The VE is 55% if case counting starts the day after the first shot. This metric is not referenced in the text of the report but can be obtained from the cumulative incidence curve I featured in the last post.

The numbers above aren't quite the same as Pfizer's or Moderna's because J&J removes "mild cases." However, J&J reported only 4 "mild cases with onset >= 14 days post-vaccination". This makes the definition of "mild" pointless (unless there are many mild cases prior to Day 14, which we do not know). The VE including mild cases is still 67%.

The headline VE also does not contain all known cases by the time of data cutoff. That's because J&J requires "central confirmation" of positive cases. About 30% of cases have not been confirmed as of the FDA meeting. These cases were excluded from the analysis - which has the same effect as assuming they are false positives. The report said that if all the unconfirmed cases came back confirmed, the VE will be 66%, decimals fewer than the headline number but on the other side of 0.5.

J&J employs adjudicators to decide if a Covid-19 case is "severe." Therefore, they report another VE that ignores moderate cases. This secondary endpoint was added midway through the trial in an amendment to the trial protocol. When moderate cases are removed, the VE goes up to 77% at 1D+14, and 85% at 1D+28. The definition of "severity" includes ICU admission and deaths but neither is necessary, and the adjudicators have the final say. We do not know how many cases they adjudicated. It is not clear how clinicians can utilize such a finding unless "severity" is predictive of hospitalization, intubation, death, etc.

In describing the above result, the report says "Efficacy against severe disease appears to be greater when cases that occurred before 28 days are excluded." Mathematically, the severe cases on the vaccine arm are more front-loaded, and if the earlier cases were excluded, the VE rises. This exposes the very odd enterprise of reducing case counts through exclusions, which almost always leads to increased VE. This increased VE has a more restrictive interpretation (only severe cases, only cases occurring after 28 days, etc.) so nothing practical is gained. The vaccine is less effective, not ineffective, in those excluded areas.

Next, there is a VE for hospitalizations, which was 82% at 1D+14 and 100% at 1D+28. We are now getting into the territory of rare events, and the error bars on those numbers span 80%. Not surprisingly, this analysis is post-hoc, not originally specified in the trial protocol. There are several inscrutable sentences about what counts as a "hospitalization" on p. 33 of the FDA report, and if interested, I'd direct you there (link to PDF).

There is one other VE estimate that should interest my American readers. The J&J trial has participants from the U.S. (~50%), Brazil (~20%), South Africa (~10%) and other South America. For U.S. participants, the VE was 74% at 1D+14, and 72% at 1D+28. These estimates included the unconfirmed cases, which means it is only comparable to the number that includes those (66%). The size of the trial is not large enough to conclude that VE is higher for Americans (the confidence intervals completely overlap).

Double Zero

Recently, the media started describing the primary purpose of vaccines as preventing deaths, even though none of the clinical trials have been sized to measure the death rate with the required precision. In several trials, including J&J, there have been zero deaths attributed to Covid-19 in the vaccine arm, and people then erroneously concluded that the vaccine provides 100% protection against death.

Most lottery players have never won a lottery. However, all lottery players do not conclude that their chance of winning is 0%.

J&J reported 7 deaths on the placebo arm that have been adjudicated as Covid-19 related. All seven occurred in South Africa. So, in the U.S. section of the trial, the verdict thus far is 0 vs 0. If you're an optimist, that's 100% protection. If you're a fatalist, that's 0%. If you're a statistician, you'd wait for more data.

Double Testing

The central confirmation of positive cases is, as far as I know, unique to the J&J trial. This procedure is effectively a guard against false positives but since negative tests are not checked, the double testing always reduces the number of detected cases. The FDA report discloses that typically about 90% of the positive results are confirmed by the central lab.

The central lab uses the Abbott Alinity RT-PCR test, the result of which over-rides local test results. It appears that this type of tests has a false-negative rate of at least 20% (link). The other trials do not re-confirm positives.

Let's assume 10% of all positive test results are thrown out by the central lab. The report tells us nothing more. It is not at all clear that the confirmation rate is even across all cases. Is the probability of confirmation the same between placebo and vaccine cases? Is the rate of confirmation the same regardless of the date of onset of cases? Any such imbalance affects how we interpret the data.

Double Blind

The J&J trial is double-blind while Pfizer/Biontech, Moderna and Astrazeneca-Oxford were all single-blind. This rules out a potential source of bias, and raises the question of why the other pharmas did not conduct double-blind trials.

Blinding reduces the chance that participants know their vaccination status and thus behave differently. Despite being double-blind, there are hints that some people in the J&J trial may have inferred whether they got the vaccine or the placebo. On Table 4, we learned that there was double the number of people who withdrew from the study on the placebo arm (72) than the vaccine arm (32) - this difference is highly significant, even though the rate of dropout is small.

Double Jeopardy

As with all trials, J&J tests each participant on the day of the first shot before innoculating him or her, and throws out anyone who tests positive. Because of this reason, J&J removed about 4,000 people, which seems quite high in a trial with 40,000 enrollees.

While these participants are no longer part of the trial, they presumably took the shot before their test results were available, and were thus followed up. Surprisingly, the vaccine appeared to be ineffective in this subgroup at both 1D+14 (3 vs 4) and 1D+28 (1 vs 2), and also ineffective against "severe" disease (1 vs 0).

Double Alibi

In a side analysis about "asymptomatic cases," J&J disclosed 3 cases that were originally deemed asymptomatic but later re-classified as symptomatic. The investigators said they analyzed symptomatic cases first since it's the primary endpoint, at which time those cases were deemed asymptomatic and thus excluded. When it came time to look at asymptomatic cases, the cases have been re-classified, and thus were excluded from the secondary analysis. In other words, those cases fell through the cracks, neither here nor there.

Perhaps only statisticians find this little episode amusing. We're talking about a few cases. This shows the high stakes in the vaccine competition among the pharmas; every case matters.

 

The third worst-kept secret

It's the third time, and we've noticed a pattern. The FDA can't keep a secret. In late February, they approved the emergency use of the Johnson & Johnson (J&J) vaccine. On the day the Advisory Committee met to deliberate in public, there was zero suspense since the entire process was choreographed weeks in advance.

The headline result from the J&J trial are these:

J&J reported a vaccine efficacy of 67% during the trial. This result from a single multi-national clinical trial implies that the vaccine efficacy is above 60% in general. Emphatically, this statistic should not be interpreted as 6 out of 10 vaccinated people being protected (and 4 left unprotected).The baseline case rate after two months was about 2 percent during the J&J trial so 98 out of 100 participants did not get ill. Upon vaccination, everyone's chance of getting infected is cut by 60%. In addition, J&J also documented few side effects of concern. Given the FDA's stated intention of approving any vaccine that has over 30% efficacy (on par with the flu vaccine), the outcome of the meeting was never in doubt.

As you must know by now, the J&J VE number cannot be directly compared to VEs from other trials because the FDA allows each pharma to decide what it counts as a case, and which time window is used to count cases. J&J counts cases starting 14 days after the inoculation, and uses a more stringent set of rules to qualify a "case".

Since all vaccine trials are sized to deliver a preliminary verdict as fast as possible, none of these trials have sufficient data to draw strong statements about subgroups - especially not at the interim analysis. This is true also of J&J. In addition, it is premature to talk about the probability of rare events, such as "severe" cases, hospitalizations and deaths. This has not stopped the media from trumpeting "exploratory" and sometimes "post-hoc" findings in the same breath as the overall VE estimate, which is a more solid result.

(Notably, none of the post-hoc analyses comes with corrections for multiple comparisons, which is a basic tool to reduce the chance of spurious findings; the more things you look for, the more things you'll find, the greater the chance that your discovery is not replicable. This comment applies generally to all vaccine trials.)

The J&J vaccine is an adenovirus vector, so it's like the Astrazeneca-Oxford technology (link), and different from the mRNA technology used by Pfizer/Biontech (link) and Moderna (link). The adenovirus technology is also relatively new, having only had one prior drug approved for ebola by the EU while the mRNA has never had any (not counting the two EUAs granted for the novel coronavirus).

The J&J vaccine can be stored in standard refrigerators, which simplifies storage and transportation. However, the shots are still packaged in groups of 5 doses; once unpacked, the entire lot must be consumed within 2-6 hours.

The J&J vaccine treatment evaluated in this trial is a single shot. However, the pharma has an ongoing two-shot experiment that began in November (more later).

Exploring the case curve

Like the other pharmas, J&J provided a cumulative case curve starting from Day 0 (the day of the shot).

Let's review how this curve should be interpreted. In theory, the vaccine efficacy is eyeballed from the chart. If you pick a time, say Day 56, the cumulative incidence rate of the placebo group is roughly 2 percent (red line) while the cumulative infection rate of the vaccine group is just under 1% (blue line). So the vaccinated group experienced half of the incidence rate of the placebo group, which translates to 50% improvement, or VE of 50%. In short, the blue line is at half the level of the red line. (To be precise, the VE was 55% by Day 56.)

As with every trial, what's shown on the cumulative case curve does not lead to the headline VE number (which is 67% in the case of J&J). The headline number nullifies cases in the first 14 days. This is the same as resetting the count of cases on Day 14 to zero (while using the at-risk population on Day 0). I explained this procedure in greater detail in this blog post. Using the same technique as above, we see that the blue line is now at a third of the level of the red line. This two-thirds improvement corresponds to VE of 67%.

Since the primary success metric is VE, we should plot VE directly, which leads to the following chart:

I'm only drawing up to Day 56 because by that time, the analytically-ready population has already been cut by half - as most of the trial participants have not yet reached that amount of follow-up time. Remember that the FDA allows analysis when only half the participants have reached two months after treatment (currently interpreted as 56 days rather than 60 days). The data beyond that period is based on a small sample, which is also biased because of staged enrollment of lower-risk people first.

It is instructive to show J&J's curve on the same chart as Pfizer's.

Firstly, the Pfizer curve is much smoother throughout the first 56 days, with the difference particularly stark during the first 14 days. J&J's curve is a step function: VE is basically zero for two weeks, then VE magically jumps to 45 percent, and grows slowly thereafter.

On the one hand, someone can look at the J&J curve and say this justifies the presumption that VE = 0 in the first two weeks. The step function embodies the idea that the vaccine is useless for two weeks. On the other hand, someone can say a biological process affecting a large population over time should be smooth rather than a step function. My view is the latter so I wonder what happened with case counting in the first two weeks. Unfortunately, because J&J discounts cases in the first two weeks, the study report has very limited data for that period, e.g., it provides no information about how many "mild cases" were detected during the first 14 days.

Secondly, Pfizer's treatment has two shots, the second one administered around Day 21 while J&J's is one shot on Day 0. Interestingly, the two VE curves are closest together at Day 21 and then the Pfizer curve rises faster than J&J's. If I worked for Pfizer, I might see this as evidence that we timed the second dose just right. (Of course, there can be other reasons for the rate divergence.) When J&J has results for its two-shot trial, we'll have confirmation about this hypothesis.

Next, I look at weekly data, rather than cumulative.

The chart shows VSC (the vaccine's share of cases for a given week), and the timeline starts at 14 days after the first shot. The lower the VSC, the higher the VE. This view reveals the vaccine efficacy to be quite stable from week to week. The first data point for J&J (red) is the first week that J&J considers official, and has the lowest VSC (or peak VE), although we can't tell if these are just random fluctuations. The cumulative chart hides this information because the most recent week is weighed down by the data from previous weeks.

***

A good question to ask is whether every person requires the same level of protection from a vaccine. I believe the answer is no. People who have lower risk of exposure, people who are younger and fitter, people who take other mitigation measures more seriously, people who have fewer daily contacts with others, people who intend to wear masks after getting the vaccine, etc. have a lower baseline risk of infection, and so they may benefit from a one-shot vaccine.

I have quite a few pages of notes left, and cover some other interesting observations in the next post.

Saying it again: timing traps 1 2 3 4...

Previously, I posted some notes on the study of vaccine effectiveness by Israel's largest health fund (insurer). I mentioned that these researchers are dealt a tough hand. The fast pace of vaccinations makes it very hard to find unvaccinated people who are similar to the vaccinated individuals. This formidable condition does not just bias the analysis set but also affects the counting of cases.

After reading many studies, I continue to doubt the ability to nail down the timing of control infections, which is the basis for estimating VE of any interim case-counting window. In this post, I will describe several issues that caught my attention - these are not unique to the Clalit study. Without the dataset, it's not easy to know how much these issues matter but these questions are worth asking.

Timing Trap 1

Due to the fast progress of the vaccination campaign in Israel, the researchers adopted a “rolling” mode of processing in which matches are found for newly vaccinated people, one day at a time. This process allows a vaccinated individual to be used as an unvaccinated control prior to his or her vaccination. If the Clalit team had followed the Mayo Clinic’s process, in which an unvaccinated individual is someone who did not get any shots throughout the study period, I believe they would suffer from excessive drop-off.

But the rolling mode of processing has an undesirable side effect. Consider someone who is vaccinated on January 1. This same person may be matched to a vaccinated person on December 25, while he or she was unvaccinated. Then, six days later, the vaccination takes place, destroying the prior match. Therefore, the unvaccinated control has a follow-up life-time of only 6 days.

The Clalit protocol requires that the vaccinated person of the matched pair be censored as well, meaning that person is also only tracked till January 1, therefore the follow-up life-time for the vaccinated person has been curtailed to 6 days. As you no doubt know by now, all research teams pick VE metrics that start counting cases sometimes from 14 days after the first dose, sometimes from 21 days after the second dose, and when a follow-up period is snipped at 6 days, no cases could ever count for that pair.

This births an awkward scenario. If the vaccinated person subsequently get sick, even if the case appears 14 days after the second dose, it would not count (technically because of censoring). This is a form of asymmetric treatment: the vaccinated person’s follow-up can be stopped by an action of the unvaccinated but the unvaccinated person’s follow-up cannot be ended by an action of the vaccinated. Shortening follow-up always leads to lower case counts.

Toward the end of the paper, the authors acknowledge that a large contingent of unvaccinated controls quickly flipped over to the vaccinated group “often within a few days of matching.” As a result, the average length of follow-up in the Clalit study is only 15 days. The main result they reported on the front page of the paper is for the case-counting window of Days 14 to 20. Many matched pairs have stopped being tracked by this time. The other most commonly cited VE estimate is for a case-counting window starting 7 days after the second dose (around Day 28). Fewer than 25% of the analysis set have been followed for over 25 days.

Timing Trap 2

A common exclusion criterion is positive PCR test prior to the vaccination date (for vaccinated cases) or prior to study enrollment date (for unvaccinated controls), defined as the vaccination date of the matched vaccinated individual. As I will now show, this exclusion rule seems to create a bias in favor of the vaccinated group.

Consider an infection that occurred on January 1. If the individual is subsequently vaccinated, then this case is excluded from the analysis, preempting matching. If the individual remains unvaccinated during the study period, then s/he can only successfully match with someone who received his or her first shot prior to January 1. If matched, the case accrues to the unvaccinated group.

Now, if we classify all January 1 cases by subsequent vaccination status, those cases affecting the vaccinated group vanish from the analysis while some of the cases on the unvaccinated group remain in the analysis set. This asymmetric treatment may produce a bias in favor of the vaccinated group.

Clalit's exclusion criterion removes people who tested positive prior to the study start date, which is December 20. The policy in Israel is not to vaccinate people who previously had Covid-19. But see also the next section. The Mayo Clinic's study excludes people with positive tests prior to getting their vaccinations.

Timing Trap 3

The authors of the NEJM paper explains how they compute vaccine effectiveness for a case-counting window that does not start the day after the first shot. Their favored window is Day 12 to 20 after the first shot. You already know this means nullifying any cases that are reported during the first 11 days.

These researchers went one step further. For each matched pair, if one individual get sick prior to Day 12, the other member of the pair is censored, that is to say, the other member cannot be counted as a case whether or not s/he gets sick after Day 12. The researchers simply stop tracking that individual. This rule limits the ability to measure a vaccine's ability to delay the onset of disease.

We don't have the data so we can't know the size of the impact. We'd like to see how many cases occurred during the case-counting window but did not count because of censoring due to events that happen on the other group.

Timing Trap 4

I already mentioned another timing-related problem in the discussion of the Mayo Clinic paper. The matching process creates subgroups of individuals that are considered "similar" on account of their age, sex, and other matching factors. Within any such subgroup, effectively, the observational study treats them as interchangeable.

But they are no longer interchangeable when we assign cases to a time-line! Because the timing of cases depends on what day is Day 0. Day 0 for an unvaccinated control is determined by the matched vaccinated individual. Therefore, a case detected on a given calendar day can be interpreted as a Day 1 case or a Day 30 case or any Day X case, depending on who among the subgroup the unvaccinated person get paired with.

***

As you can see, real-world studies are complicated enough without these timing traps. The exclusions and matching process already requires many discretionary decisions. The insistence on creating arbitrary case-counting windows makes these real-world studies much harder to interpret, and prone to mistakes. 

 

The confusing pictures in Israel and in the Israel study

Today and tomorrow, I’m taking you through another real-world study which has become the favorite of mainstream media, the study by Israel’s largest health fund (insurer), that confirms the effectiveness of the Pfizer and Moderna vaccines. The report has appeared in the prestigious New England Journal of Medicine (NEJM) here (with supplements).

It has come to my attention that my readership is split between those who see my commentary on scientific papers as an anti-science provocation, and those who value reading my notes and have requested more. The goal of my blogs has always been to promote critical thinking, to encourage chewing before you swallow information. My target readers are people who want to stay for the entire movie – even to visit the sets – after watching the trailer. If you’re here, that probably describes you.

What is a Real-World Study?

The Israel study by Clalit is an example of a real-world observational study. I covered the basics of such studies here, and also described a similar study by the Mayo Clinic – nference team here.

In brief, real-world studies analyze data that are collected by healthcare systems, which are much more challenging than experimental data tracked during randomized clinical trials (RCTs). In real-world studies, the difference between the vaccinated and unvaccinated individuals is not just caused by vaccination, but it may be explained by a myriad of other factors, such as age bias, geographical bias, and socio-economic bias. In an RCT, because we use coin tosses to determine someone’s vaccine eligibility, all other factors are equalized, avoiding this mess.

The Clalit study adopts the same high-level strategy as the Mayo Clinic’s, manufacturing a synthetic control group that mimics the placebo group in a RCT. All vaccinated individuals during the study period are considered. Each vaccinated customer (who survives the exclusions) meets a matching algorithm that attempts to locate an unvaccinated individual who is similar on a list of factors, such as age and sex. Unmatched people are dropped from the analysis set. The vaccinated and unvaccinated subgroups can now be treated as if randomized. The researchers then compute vaccine effectiveness (VE) using various case-counting windows, and outcome metrics. They select one of these analyses to highlight on the front page.

For Clalit, they conclude that the real-world vaccine effectiveness was 57% during the case-counting window of 14 to 20 days after the first dose (1D+14 to 1D+20), under the criterion of “symptomatic Covid-19” which is closest to what the vaccine trials were measuring. Surprisingly, they did not mention until later the effectiveness from 7 days after the second dose, which was 94%. This number looks like the VE reported by Pfizer during the vaccine trial, even though the study populations are different.

Footnote: Yet another real-world study contradicts the lobby who’s been pushing for single doses or delaying second doses after “re-analyzing” the Pfizer case curve to argue that its vaccine yielded 90% efficacy after a single dose. For what it’s worth, the Mayo Clinic study reported 70% effectiveness in the comparable time frame as Clalit’s; neither number is close to 90%.

In both studies, the day of the first shot determines Day 0 for a vaccinated person, and sets the day of “study enrollment” (“recruitment”) for the matched unvaccinated individual. As I explained before, fixing a “Day 0” for the unvaccinated is a necessary hassle in these observational studies, a problem that does not arise in a randomized clinical trial in which people in the control group are given two placebo injections. As with the Mayo Clinic study, the date of infection of a matched control depends on the identity (and thus Day 0) of the vaccinated individual, when the algorithm selects one out of multiple possible pairings.

Matching Mechanics

The details of the matching process vary between the Clalit and Mayo Clinic studies. The Israeli insurer requires a stricter form of matching (“exact”) while the Mayo Clinic uses the propensity scoring approach. The propensity score distills a set of matching variables into a single dimension, so that the matches are approximate. For example, all matched pairs in the Clalit study must be the same sex. In propensity scoring models, it’s permissible to pair a male with a female – so long as the matched pair is sufficiently similar on other factors.

Stricter matching offers better assurance that the matched pairs are sufficiently similar to each other. However, statistical adjustments come with trade-offs. The yield of Clalit’s stricter matching process is lower than for the Mayo Clinic. As far as I can tell, every vaccinated individual at Mayo (excepting exclusions) is successfully matched. By contrast, only four out of ten vaccinated individuals remain in Clalit’s analysis set after the exclusion and matching process. The high drop-off rate presents formidable challenges to generalizing the study’s findings.

“All persons who were newly vaccinated during the period from December 20, 2020 to February 1, 2021, were matched to unvaccinated controls in a 1:1 ratio according to demographic and clinical characteristics.”

I sighed when I read this sentence which initiated the second paragraph of the NEJM paper.

If I am harsh, I’d say the quoted sentence is false. But such careless language routinely appears in journals in this field, including the Lancet, NEJM, BMJ, etc. The sentence implies (wrongly) that the findings of the real-world study applies to all Clalit clients who were vaccinated before February 1, 2021. The media have taken the liberty of extending this result to all vaccinated Israeli since half of Israel are members of Clalit.

The key data are to be found in the Supplement (Table S2), the top part of which is shown below.

'

The first column concerns the “full” vaccinated population while the third column describes the “matched” vaccinated population. In the table header, we learn of a drastic drop-off from 1.5 million to 597K. This 60% deletion is in large part due to the incurable imbalance problem I mentioned before (see here). Only two out of every five vaccinated individuals remain in the analysis set.

When reading real-world studies, we must always guard against our natural instinct to assume random (see this prior post). Many of the assumptions we make about randomized clinical trials (RCTs) cannot apply. If the analysis set were a random sample of 40 percent of the “full” data, the study’s findings can be generalized to the full dataset. But in an observational study, the analysis subset is decidedly not a random sample of the full population – remember, the point of matching cases to controls is to get rid of unwanted biases that exist in the full population! Therefore, this study’s finding concerns the matchable sub-population (the third column), and not all vaccinated Israelis.

Eventful Exclusions

The research team behind the Clalit study faces an insurmountable problem. The pace of vaccination in Israel is too fast for this type of studies. They quickly run out of unvaccinated individuals for matching, especially people in groups prioritized for vaccinations. These include health care workers, adults over 60 years old, and adults with coexisting conditions.

Look back at Table S2. Nine percent of the vaccinated people are 80 years or older, and this age group got trimmed down to only 4% in the analysis set. Meanwhile, a quarter of those vaccinated were under 40 (which is surprising given the priority rules), and this age group was over-weighted to 36% after the matching process. The matching process has introduced a strong youth bias into the real-world study. This must inform our understanding of the VE estimates.

Exclusions also cause the analysis set to deviate from the full set of vaccinated customers. Figure 1 in the NEJM paper provides partial information. Some highly consequential exclusions include people living in nursing homes, people confined to their homes, and health care workers. Because these high-risk segments have been omitted, the estimates of vaccine effectiveness are exaggerated if extrapolated to the general population. These exclusions also make any findings on hospitalizations, severe cases and deaths hard to stomach.

Of note, the original study protocol did not contain those specific exclusions. I’m not convinced by the justification of “high internal variability in the probability of exposure or the outcomes.” Another influential exclusion that was subsequently appended removes people who “had a health care interaction within 3 days before the vaccination date”. This one exclusion rule knocks out 13% of the vaccinated individuals (202K), or one-fifth of all exclusions. The justification for this rule is that those interactions “may indicate the start of symptomatic disease and may preclude vaccination.” I don’t know what to make of this since the rule was applied to the vaccinated group (Figure 1), and it appears that the excluded individuals have a high chance of getting infected after vaccination. We don’t know how they define a “health care interaction” or decide on 3 days, which was not pre-specified.

The full list of exclusions are: under 16, not a customer of Clalit, not a “continuous” customer of Clalit, a positive PCR test prior to December 20, 2020, residents of nursing homes, people confined to the home, health care workers, missing home address, missing BMI, missing smoking status, people who had a health care interaction within 3 days before vaccination.

The matching factors include: age, sex, sector (ethnicity), neighborhood, number of influenza vaccines in past 5 years, pregnancy, and number of coexisting conditions.

I’m curious about matching coexisting conditions on the count, which ignores severity or type of condition. This is an "exact" matching criterion that is inexact. An obese individual with healthy lungs can be paired with an individual with normal weight and chronic pulmonary disease, if both have just one coexisting condition (actually one or two but let’s not dwell on that here). I can’t comment further without access to the data; I just want to point this out for any methodology nerds out there.

***

So far, we've learned that the exclusion and matching process used by the Clalit team generated an analysis set that appears to bias younger, lower-risk individuals, thus we expect the vaccine effectiveness estimates to be over-stated. The precise impact of such bias is hard to gauge in real-world studies, which is why researchers prefer randomized trials. The team in Israel faces the unusual challenge of the fast pace of vaccination, leaving them with incurable imbalance. In the next post, I address how they attempted to work around this problem, and the limits to what scientists can do about it. (It's a look behind the movie sets that I promised you.)

These studies demonstrate that the vaccines are effective enough (and also safe enough). As with all real-world observational studies, we must be careful about generalizing the results to larger populations. The point of matching is to define two comparable groups by deleting individuals without appropriate comparators but in doing so, neither group represents the universe from which it is drawn.

Finally, it's important for anyone reading real-world studies to pull up the latest real-world Covid-19 statistics on Israel. Here are the cases and deaths charts from OurWorldinData.org.

 

The decline in cases has stalled for two weeks, and the daily total is currently 6 times higher than at the start of the current wave. Deaths may also be stalling, and are currently almost three times higher than at the start of the current wave.

Of course, one must not jump to conclusions. The curves are not purely driven by vaccinations.

 

Real-world studies: limits of knowledge

In the last post, I featured a real-world study of vaccine effectiveness sponsored by the Mayo Clinic. The preprint is available here. Having read several similar studies, I think this is one of the better ones.

Real-world studies apply to data that have been observed, and not collected from a designed experiment. As a result, the vaccinated group and the unvaccinated group are not twins except for vaccination status. Thus, the analyst must correct for any biases that may explain part, even all, of the observed group difference.

I'll be making further observations about this research that reveals the limitation to what we can do in a real-world study.

How much covariate balance was achieved?

A key goal of statistical adjustment is to achieve covariate balance, which enables us to treat the two (adjusted) groups as if randomized. Matching by propensity score is not exact because the propensity score is a single number that summarizes all the matched variables (age, sex, race, ethnicity and number of prior PCR tests in the case of Mayo Clinic). Table 1 seems to confirm that the two groups post-matching have essentially the same age distribution.

Matching on age is justified because this coronavirus affects older people worse than younger people. The unvaccinated population, if not adjusted, is younger and so a simple analysis will understate the effect of the vaccine.

The true picture is messier, which can be found in Figure S1(B):

Focus on the columns, each of which is an age group spanning 5 years. The blue columns represent vaccinated people for whom they sought unvaccinated twins (shown in orange). The matched unvaccinated group contains an underweight of people aged 80-90. This gap is bridged by adding people over 90, and people aged 70-80. The excess of people over 90 in the control group is potentially an issue. This difference in distribution is obscured by the aggregate statistic shown in Table 1.

The counterfactual

Figure S1(B) is also revealing when placed side by side with Figure S1 (A), which displays the age distributions of the vaccinated and unvaccinated groups prior to matching.

The orange columns before (A) and after (B) matching look completely different. This means the matched unvaccinated group is far from representative of all unvaccinated people. This is expected because the statistical adjustment is used to carve out a subset that mirrors the vaccinated group.

The blue columns appear to be identical on both charts. This confirms that the analysts were able to find a match for every vaccinated individual. That is frequently impossible. But remember that upfront exclusions (such as zip codes with fewer than 25 vaccinated people, and people without prior PCR test results) already make the analysis set different from the set of all vaccinated people.

So what is the point of an observational study? It's to give a good answer to the counterfactual question - what would have been the case rate of the vaccinated group in the analysis set if they had not been vaccinated?

The other uses of real-world studies

Food for thought. What if we reverse the matching process - finding twins for the unvaccinated group? This analysis answers the reverse question: what would be the case rate of the unvaccinated group if they were vaccinated? I hope you can see that the resulting matches will be different.

In fact, this is the more interesting question. We expect most people to eventually get the shots so this result can be used to project the imminent future.

I don't get why all the real-world studies aim to replicate the finding of the randomized clinical trial (RCT). If they find a high VE, we learn yet again why the RCT is the gold standard. If their result is worse, we'll choose to believe the RCT because it is the gold standard. I just can't imagine a scenario in which the pharmas will conclude that the VE is lower than what was measured in the RCT, based on a real-world study.

Real-world studies have other uses. For example, it can provide additional color on subgroups. As we know, the RCTs did not enroll enough minorities to provide estimates with reasonable margins of error. They also did not have enough people to analyze geographical differences. Unfortunately, the studies so far have not focused on these aspects that can bring new knowledge.

The one quantity that can't be matched

I now address the biggest concern I have about these real-world studies. They claim an ability to trace the timeline of vaccine efficacy from Day 0 (first shot) through Day 7, 14, 21, ... etc. I'm going to explain why these studies don't yield accurate interim estimates. They should focus on total counts inside a wide case-counting window.

In the vaccine trial, the control group takes two placebo shots. Therefore, each unvaccinated participant has a well-defined Day 0, so that subsequent positive tests can be positioned relative to Day 0.

In a real-world study, there are no placebo shots. As explained in the last post, the Mayo Clinic-nference team defines Day 0 for an unvaccinated individual as the day of the first shot of the paired vaccinated twin. The problem is that there are many possible unvaccinated people to match to each vaccinated person.

Consider an unvaccinated person who gets infected on February 1. The study period for the Mayo Clinic research was from December 1 to Febraury 8. This person - based on zip code, sex, race, ethnicity and number of prior tests - is eligible to be matched to several vaccinated individuals. In each matched pair, Day 0 is fixed by the vaccinated person. Different vaccinated persons may have different Day 0. Therefore, the case on February 1 can end up on any part of the cumulative case curve, depending on which vaccinated individual the matching algorithm pairs this person with!

If the unvaccinated person is matched to someone who got vaccinated on January 31, then the infection on February 1 is counted as a Day 1 infection. If s/he is matched to someone who got vaccinated on January 1, then the same infection on February 1 is counted as a Day 32 infection.

Smarter minds may figure out how this issue can be solved. However, I know it can be avoided by not analyzing the shape of the curve - focus on the total count recorded from Day 0 to the end of the study period. This recommended analysis is independent of the timing of the cases.

Which part of the cumulative case curve is most reliable?

Lastly, I want to draw your attention to Figure S2(A).

This chart brought me so much joy! When reviewing the RCTs and the real-world studies, I kept wanting to see this chart but other research teams failed to sate my hunger.

The chart shows the number of days each vaccinated person in the analysis set has been followed since Day 0 (first shot). This chart is pivotal to our understanding of the VE calculations in an interim analysis.

It seems like an eon ago when I called for the FDA to require two months of follow-up for every trial participant, and not the median trial participant. In the above chart, we can roughly place the median follow-up time at around Day 30. Half the people have more than 30 days of follow-up and half have fewer.

Imagine if this analysis were performed after all people have at least 60 days of observation. We should see a single column at Day 60.

Now look at the very first column on the left side sitting at Day 1. These are people who got their first shot the day before the study period ends so that the analysts only have one follow-up day for them.

Take a quick look at the famous Pfizer cumulative case curve:

I don't have Pfizer's version of Figure S2(A) but imagine it. The first column on that chart were people from the trial that have had a single day of follow-up. This means they can only impact the leftmost edge of the cumulative case curve.

By contrast, the right side of the cumulative case curve consists only of participants who have reached a long follow-up period.

In other words, the part of the curve that has the greatest "support" is the left side. The further to the right of the curve, the fewer people are contributing to the counts, the less reliable is the data. If they showed margins of error, the margins will grow quite drastically moving left to right. What have most analysts been doing with the Pfizer curve? They chop off the left side (link).

This timing issue only affects interim analyses. Let's say we conduct a "full" analysis when all participants have reached 112 days of follow-up (the right edge of the curve above). If we had a Figure S2(A), we would see a single column at Day 112. In that case, every participant is affecting the entire length of the curve. (Technically, we have censoring due to testing positive, death or dropout but not due to interim analysis.)

***

With that, I finish my review of the Mayo Clinic paper. I will turn my attention to other real-world studies. There is considerable variability between these studies, as, you might have noticed, there is quite a bit of art in this science.