Big Data, Plainly Spoken (aka Numbers Rule Your World)

Curve watching during the pandemic

The pandemic has revealed the importance of a most exciting area of statistics - causal inference from observational data. We've been inundated with all kinds of reports about the effect of [vaccines, masking, social distancing, school closures, outdoor dinings...] on the pandemic. Most of these "studies" focus on either one factor or one region or one demographic segment. Then, the headlines and PR coming out of these studies apply the specific results to the general public. Vaccines work! Masks don't work! Closing schools doesn't make sense!

The quality of the causal inference is tested when we apply the findings of these studies to other settings. So in today's post, I explore the claim that vaccination campaigns are responsible for the decline in cases seen in many countries.

In order to shut down our biases, I am presenting the graphs first without country labels. In each chart, the time period is from November 2020, before any vaccinations, to April 2021. I used data from OurWorldinData.org. The red line shows the new cases per million while the blue line shows the proportion of the population who have been vaccinated (defined as those who received at least one shot if it's a two-shot vaccine).

If we only look at individual countries, here are the stories one can tell:

Panel A presents a clean picture. Cases have tumbled from a peak in mid January to a level below that of November by end of April. The decline begun soon after initiating the vaccination campaign, which has reached about half the population. So vaccines worked!

In Panel B, this country's highly successful vaccination campaign has covered over 60 percent of the population by end of April. That appeared to be a critical mass, at which point cases dropped below the level of November. One can build a story about herd immunity.

The herd immunity story is shaky though. The 60-percent threshold does not apply to Panel A. Using the same logic, one would argue that the country reached critical mass even at 30% vaccinated. But at 30% vaccinated, the country in Panel B was approaching peak cases!

In Panel C, the sharp drop in cases happened as the proportion vaccinated rose from 20 to 40 percent while cases grew during the first two months of the campaign.

In Panel D, cases plunged within weeks of the start of the vaccination campaign, and reached a low plateau with fewer than 10% of the population vaccinated.

Then, you have Panels E, F and G. If one only analyzes those countries, then one comes to the opposite conclusion. In these places, the cases apparently have not been slowed by vaccinations. Over 60% of the people in Panel E have taken a shot but no sharp decline is evident yet. For Panel F, there have been a spurt in cases in April with the vaccination proportion nears 60%.

The picture is very confusing. That's because the vaccine is not the only factor affecting the rise and fall of cases. Any model will fit some of the data some of the time; a good causal model must work across multiple countries.

Now, I reveal the countries. These are the top 11 countries ranked by the proportion vaccinated by end of April. I excluded countries with low overall Covid-19 incidence. Seven of these countries have experienced a nice drop in cases per million.

***

It's also a good idea to take a peek at the other end of the spectrum. What's going on in countries that have barely begun their vaccination campaigns.

Here are 12 countries with very low vaccination rates but with case incidence in a similar range to the 11 countries shown above.

None of these countries have seen spikes in cases, almost all of them are seeing downward trends in April, many of them exhibit up-and-down cycles over these months, and a few have marked plunges such as Georgia in December and Montenegro since March.

***

Simple one-factor models aren't going to work to explain trends across countries and time. A good causal model should include a baseline trendline, a vaccine factor, plus lockdowns and other mitigation measures.

Another challenge for data analysts: the data post-vaccination will be hard to compare to past data. There will be fewer Covid tests among the fully vaccinated. Should cause of death be assigned differently if the patient is known to have been vaccinated? Will we see more deaths from co-morbidities? These are open questions - when I learn more, I'll report back.

 

 

The miracle of randomization in AB testing and RCTs

I'm in the midst of preparing my talk this Friday at a University of Denver conference (more info here). The talk will focus on the range of research methods that have been utilized in pandemic-related scientific research, why research methods matter, and how they are related to research quality.

I have some materials that probably will get edited out of the final presentation but I don't want to just throw them away. So I am posting some of them here. Today, I like to use a practical example to illustrate the miracle of randomization, which is an important feature of controlled scientific experiments.

It's hard to imagine running an A/B test or a clinical trial without randomization but this idea is not that old - the British statistician RA Fisher is often credited as having invented it in the 1920s. Randomization means the participants are randomly assigned (via a coin flip) to A or B groups (or test or control groups).

The consequence of randomizing treatment is profound. It leads to "covariate balance". If 50% of the research population is female, we expect 50% of the test group as well as 50% of the control group to be female. Depending on the total sample size, the margins of error around those percentages vary.

The one act of randomization confers balance between the test and control groups on all covariates of interest - not just gender but also race, income groups, comorbidities, employment status, anything that is relevant to your study.

What's more, you get balance on other covariates for free! Any covariate, including things you've never dreamed about.

It is this last point that has enormous practical implications.

Let me explain using a real-life A/B test. Let say Amazon wants to test sending customers an email on Christmas eve, and measure whether this email increases spending in the next week. The data scientist draws up the list of targeted customers, then randomly select half of this list to receive the email, and the other half to be excluded from the mailing.

If everything goes as planned, at the end of the week, the data scientist can pull the data, tabulate the spending on both groups of customers, and any difference in spending can be attributed to the email. Of course, the analyst knows that household income, gender, religion, ethnicity, education, prior spending, etc are all factors that affect holiday spending. Since treatment was randomized, all of those factors are well balanced between test and control groups, and therefore cannot explain any statistically significant change in spending.

So far so good.

In a meeting with the IT team, it was disclosed that a systems error caused all Yahoo.com emails to be dropped from mailing. Let's assume that people still using Yahoo.com emails are bigger spenders on Amazon (reasonable since these people are probably older). Has this error ruined our test?

The short answer is no. That's because randomization not only balances variables we know about, but also balances unknown variables. In this case, if the analyst pulls out the proportion of Yahoo.com email addresses from the test and control groups, it's highly likely that the difference is immaterial. Therefore, we can still interpret any difference in spending as caused by email.

It gets even better!

You might think the above scenario is concocted and unrealistic (though I can assure you it's not.) What makes it a tad unrealistic is the eventual discovery of the error. Most errors are committed and never revealed.

For example, people with Yahoo.com emails may not be dropped by our IT team; they may simply be victims of spam filters. Most email providers out there use one of several spam filters, which are black-box algorithms that change constantly. For any number of reasons, it's possible that Yahoo's spam filter algorithm dumped Amazon emails into the spam folder during the week of our email test, and so people with Yahoo.com emails mostly didn't see the emails.

This situation cannot be predicted by the data scientist, nor will it likely to discovered after the fact. However, our test analysis can proceed as usual because again, the test group will have about the same proportion of Yahoo.com emails as the control group. To the extent that a difference in spending is observed, it isn't due to the spam filter.

We can call this the unintended consequences of randomization. This is really really cool stuff.

***

Not long ago, I wrote about real-world studies of vaccine effectiveness that mimic randomized controlled trials by synthesizing test and control groups. The matching process involves explicitly balancing the two groups on a list of matching variables (age, gender, etc.).

This process does not have the magic of a randomized experiment. In practice, balance is enforced on those matching variables but there is no reason to believe the two groups are also balanced on other known covariates not used for matching. Further, it is even less likely that balance is achieved on unknown covariates.

This is one reason why experiments are considered the gold standard for causal studies.

 

 

The statistics behind the J&J blood-clot decision

Last Friday, the U.S. ended the pause of the Johnson & Johnson vaccine, which may be linked to rare blood clots in women under 50. As with the EMA's stance on the Astrazeneca vaccine, they recommend continuing to administer these vaccines based on a cost-benefit analysis.

In today's post, I summarize the key information in the documents released during Friday's meeting of experts about those blood clots. The media have failed to cover the statistical reasoning behind the decision.

In its presentation, J&J highlighted the number that is most beneficial to its commercial interest - the risk of rare blood clots in the general population regardless of gender and age. The media is also fixated on this statistic, which is the least useful of all the numbers in these documents. Out of 8 million people who have been given the J&J shot since March (remember that this vaccine only got approved at the end of February), they have reported 15 cases of rare blood clots coupled with low platelet counts. This means the risk is 2 cases per million.

Measuring the risk of this side effect is a great example of why no statistic is purely objective. The subjectivity lurks behind how the risk is defined. Risk is a ratio of two numbers: the number of cases, and the relevant population at risk. The 2 cases per million number bakes in two choices: on the denominator, J&J selected the entire population regardless of age and gender; on the numerator, J&J selected the co-occurrence of rare blood clots and low platelet counts. Those are not the choices I expected. 

The denominator should be women only, or younger women only, given that all 15 cases occurred in women under 60 years old. The choice of the denominator is not immaterial! The CDC ultimately used women aged 18-49 as the relevant population, and this change of denominator bumped the risk from 2 up to 7 per million. (Including men roughly doubles the denominator while adding zero cases to the numerator.)

The cutoff age of 50 is also a choice. Two cases were found in women between 50 and 60 years old. By setting the at-risk population as women under 50, those two cases are removed from the numerator. Each additional case increases the risk by roughly 0.5 times if the denominator is not changed appreciably.

The CDC also looked at smaller subgroups. Women in their thirties accounted for the most cases, and the risk in that age group was 12 per million (7 cases among 600K vaccinations). Meanwhile, 1.4 million women aged 50-64 took the J&J vaccine, with two reported cases of blood clots and low platelets, leading to a risk of < 2 per million. Scientists can choose the most restrictive analysis by focusing only on women in their thirties, or the most expansive analysis, on all women. The CDC's choice of 50 is reasonable while not the only option.

The numerator is also unusual, and different from what the EMA used when examining the Astrazeneca vaccine. J&J and the CDC decided to define the side effect as rare blood clots coupled with low platelet count. This is a more restrictive case definition, which reduces the number of cases under study. Interestingly, this definition neatly draws a line between the adenovirus platform and the mRNA platform (Pfizer & Moderna), as so far, the mRNA vaccines have seen some cases of blood clots but zero cases of blood clots plus low platelets.

***

The next issue is how bad is that risk. For this, we have to compare to other risks. 

One obvious comparison is the baseline rate of blood clots and low platelet count. This is where the choice of numerator presents a difficulty. The baseline risk is essentially zero: that combination of conditions almost never happens. One of the CDC documents compared the observed risk to the risk of blood clots in women 20-50 years, which is invalid. The risk of blood clots plus low platelets should be quite a bit lower than the risk of blood clots (with or without low platelets). In a different CDC document, the risk of blood clots and low platelets is estimated to be under 1 case per million. In the J&J document, they claim the baseline risk is 0.1 per million. Neither of these numbers can be directly compared to the 7 per million observed risk because the baseline risk is per million Americans, not per million American females aged 18-50.

Nevertheless, it is abundantly clear that the risk of getting blood clots and low platelet count among women under 50 who took the J&J shot is much higher than the baseline risk. The risk of dying from this side effect is also much higher than the baseline mortality risk. (About 25-30% of the known cases have already died.) On an absolute scale, the condition is very rare. On a relative scale, it is concerning.

All 15 cases were reported between 6 and 15 days after the vaccination. Notably, this is the period during which scientists insist the vaccine has zero benefits. 

***

Another relevant comparison is the risk of death from Covid-19. This is a key to the cost-benefit analysis. How should we compute this risk? We should focus on the period from March and April, the two months in which the J&J vaccine was administered. 

This CDC slide indicates that among females aged 18-50, the Covid-19 death rate was around 0.3 per 100,000, or 3 per million. This means the chance of dying from Covid-19 is about the same as from dying from blood clots+low platelets after taking the J&J vaccine.

Yes, that's right. It's also confirmed in the CDC simulation model. 

So, what does the CDC mean when it says the benefits outweigh the costs? The simulation model finds that slowing down the vaccinations causes a lot of excess Covid-19 deaths among men, and among females over 50, and therefore, across all age groups and genders, stopping J&J causes more harm than good.

***
Two other small details caught my attention. There is another fatal case of blood clots and low platelets that was excluded - seemingly because the situation is too complex. There are also other cases being investigated including cases affecting men. 

When the investigation first started, there were 6 reported cases. About 10 days later, the number of cases more than doubled to 15. The vaccine didn't become more risky suddenly. This phenomenon is caused by self-reporting of cases. Such self-reporting has always been a weakness of how we monitor side effects. The risk may be over-estimated because of under-reporting of mild cases.

 

 

 

 

 

 

 

 

What to think about vaccines today, including why vaccines protect you against blood clots

Georgette's comment to my other post prompted me to write a new post on current Covid-19 topics.

Vaccines protect you against blood clots

They didn't say it out loud but they imply it. Keep reading...

The FDA's pause on the Johnson & Johnson vaccine is a significant event. The situation parallels the controversy over many European countries halting the distribution of the Astrazeneca vaccine. My take is the same as before. The only unknown is whether the FDA will issue an additional warning about rare blood clots that affect certain subgroups of people. They are likely to keep recommending the vaccine to most of us.

The circumstantial evidence related to blood clots is accumulating. Both J&J and Astrazeneca vaccines use the adenovirus technology. These blood clots are happening mostly to young women, and are paired with other conditions (low platelets), which makes natural cause being rather unlikely. The diagnoses and demographics of the affected people are similar across both vaccines, and, though rare, the same condition has popped up in numerous countries.

The cost-benefit analysis for the young female demographic is completely different from the average person. The benefit of the vaccine is much lower, given the tiny mortality rate from Covid-19 (not counting the benefit accrued to others). Meanwhile, the cost of serious illness and death from rare blood clots is relatively higher because the baseline risk of this condition is minuscule for that age group. A preliminary estimate by the European Medicines Agency (EMA) found that for the younger population, the reported cases of those rare conditions were 5 - 12 times higher than expected.

Doctors are justifiably alarmed, given the "do no harm" principle. Side effects have always been the challenge of developing vaccines. Injecting otherwise healthy people with a substance and causing harm is not accepted, except in a pandemic. The risk to most people is still low but it doesn't then follow that the risk is low to all.

The statistic of the moment says the overall incidence of blood clots is far, far lower than expected among those who were vaccinated. Therefore, they say, the vaccine does not cause blood clots. The gaping hole in this argument is that the data present strong evidence that these vaccines protect people against blood clots! If the vaccines have nothing to do with blood clots, then we expect similar incidence among the vaccinated as among the unvaccinated, and both rates being similar to the expected value. Instead, they tell us blood clots are far less common than expected among those who got the Astrazeneca shots.

I support the FDA's pause decision. It builds confidence that the agency is keeping an eye on safety as there are plenty of examples of medicines that have side effects that are not found during a clinical trial. I hope they look beyond case counts, and review other information as well. Recall my post on the law of small numbers. 

Fully vaccinated people does not have a 5% chance of catching Covid19

Your chance of catching Covid-19 after getting fully vaccinated is not 5%. It cannot be that high. If you review the data from all the vaccine trials, the typical case rate on the placebo arms is 2-3 percent. That means, of those participants who were given placebo shots during the trials, 97-98 out of 100 did not get symptomatic, confirmed Covid-19. If the chance of getting infected after 2 vaccine shots was 5%, then the vaccine would have negative efficacy (worse than placebo)!

The source of this confusion reflects a misunderstanding of a vaccine trial. Vaccine trials are different from the usual clinical trials of drug treatments. In a cancer drug trial, every participant has cancer. In a vaccine trial, we don't know who has been exposed to the coronavirus. In a short trial, with only half the participants reaching 8 weeks of observation, most of the uninfected probably have not been exposed yet.

Giving a probability of getting infected without stating a time window is also useless. If the 5% number came from the trial results, then the right way to describe it is 5% within two months (which, if you think about it, is disturbingly high).

Vaccines and the cargo cult

There is a cargo cult developing around vaccine science. Vaccines have become the single and only explanation for positive trends while not enough vaccines is the single and only explanation for negative trends.

For example, the U.K. and Israeli governments credit their fast vaccine rollouts as the reason for plunging cases. To do proper science, we must take into account all other factors, and in particular, the impact of national lockdowns - which coincided with the vaccination campaign - and the reversal of the holiday spike.

The vaccines-only narrative lends support to lockdown skepticism, mask skepticism, and social-distancing skepticism. Those brands of skepticism also weaken our national fight against the coronavirus.

All vaccines are the same

Vaccines are not all the same. To believe that a VE 60% vaccine is the same as a VE 90% vaccine is to believe that the scale for measuring VE is severely flawed. To believe that vaccines using different technologies and formulations are all the same is to believe that formulations and technologies are window-dressing. Even if all vaccines were identical, people still have preferences: some go for a black car, others a red car - even when it's the same car.

 

 

How the CDC study handled biases in observational data

I offered a few high-level comments on the widely publicized CDC study of real-world effectiveness of the mRNA vaccines in my previous post. Today, I take a deeper dive into the study.

The main value of this real-world study comes from the weekly swabs requested from each participant. Unlike other real-world studies based on "found data", this CDC study is an organized effort with enrolled participants who agreed to send in swabs so their infection status can be determined each week. Of the vaccine trials I reviewed, only the Astrazeneca included this feature (although only for the U.K. trial and possibly only a part of that). These swabs reveal asymptomatic cases. As a result, the measured infection rate is likely to exceed what was observed during the vaccine trials.

The study reported an admirable compliance rate on those swabs. The median participant submitted all requested swabs.

 

Similar to the Danish study, the analysis population consists of specific higher-risk people. The CDC study focuses on healthcare workers and essential workers. Like all real-world studies, we must be careful when generalizing study results. It's one thing to say the vaccine was found to be highly effective for healthcare & essential workers; it's a completely different thing to claim that the study showed the vaccine to be highly effective for everyone (which is what the media and many "experts" have been touting all day).

Unlike the Danish study (based on comprehensive, found data), the CDC study does not include every healthcare worker or essential worker. Because of the enrollment requirement, the study population is self-selected so the first question to ask is whether the analysis population is representative of all healthcare workers and essential workers. There is nothing in the paper to answer this question. Table 1 discloses that 7 out of 10 people in the study are under 49, over 80% are white and non-Hispanic, and 70% has zero chronic condition (left undefined in the paper). The analysis population is thus significantly younger and healthier than our average American.

 

There exists two validity questions. The above deals with the difference between the analysis population and the general population. The next question of validity concerns the differences between the vaccinated group and the unvaccinated group within the analysis population.

There are two sentences in the CDC study that perfectly describes the challenges of any real-world effectiveness study:

For example, the infection rate in Miami, Florida was 8.6%, which is 65% higher than the average of 5.2% in the study. Twelve percent of the unvaccinated group work in Miami, compared to 3.5% of the vaccinated group. The infection rate in Portland, Oregon was 1 percent, much lower than average, and 90% of the Portlanders included in the study was in the vaccinated group.

Another example is occupation. The infection rate of primary healthcare workers was 2%, and over 90% of them were in the vaccinated group. Meanwhile, the infection rate of first responders was 9% (over 4.5 times higher), and 40% of them were unvaccinated.

Those two sentences suffice to explain the field of real-world studies. The researchers conclude that VE is very high, that is to say, the unvaccinated group has a much higher infection rate than the vaccinated group. They suggest that the vaccine is wholly responsible for the observed difference in infection rates. But any of the following statements can also be true:

• Some or all of the difference is explained by the over-representation of higher-risk males in the unvaccinated group
• Some or all of the difference is explained by the over-representation of higher-risk Hispanics in the unvaccinated group
• Some or all of the difference is explained by the over-representation of higher-risk first responders (and under-representation of lower-risk healthcare personnel) in the unvaccinated group
• Some or all of the difference is explained by the over-representation of higher-risk people living in Arizona, Florida and Texas (and under-representation of lower-risk residents of Minnesota and Oregon)

The two groups being compared differ not only by their vaccination status but also by gender, race, occupation and state of residence. The unvaccinated group has an over-representation of higher-risk individuals of each category.

If you recall my posts about the studies by Mayo Clinic or Israel's Clalit, this is the imbalance problem they solved by the matching procedure. In the CDC study, they ignored all these biases except for study site.

For studies using regression models, the usual corrective mechanism is to include adjustment terms, similar to the calendar time bias adjustment used in the Danish study. For more details, see my previous post.

In the CDC study, the only factor they adjusted for is study-site bias.

In a side comment, the researchers said they considered regression models that adjusted for sex, age, ethnicity and occupation ("individually"), and the change to VE was "<3%." They appear to be saying that those factors are unimportant because the change to VE is small. And yet, in Table 2, we learn that the study-site adjustment changed the VE from 91% to 90%, which is a difference of 1%.

I'd have preferred to see a model that includes all variables that can explain the difference in infection rates between the vaccinated and unvaccinated groups. Even if the effects of some of these variables are not statistically significant, they need to be in the regression model to obtain a better estimate of the effect of vaccination status - this is because of the correlation between vaccination status and those demographic variables (which is due in part to self-selection).

 

Curiously, the CDC study did not do a calendar time bias adjustment, and therefore, the bias identified in the Danish study is present. This bias originated from the drastic decline in infection rates during the first quarter of 2021. Upon vaccination, an individual migrates from the unvaccinated group to the vaccinated group. If we tally up the person-hours for the unvaccinated, they will skew towards the start of the study compared to those for the vaccinated group.

The time bias in the CDC study might be mitigated by the unusual enrollment cadence. While the study ran from mid December (start of vaccinations) to mid March, 60 percent of participants got their first doses in December. That leaves 40 percent. Since 25 percent remains unvaccinated to the end, only 15 percent got their first shots between January and mid March. So, the vaccinations were concentrated heavily in the first two weeks of the study.

This sheds some light on the self-selection bias problem described above. It seems that the 25 percent who remain unvaccinated chose not to get inoculated as there was plenty of time for them to do so.

Further, because the case counting window applied to the vaccinated group only, infections in the first few weeks of this study period can only appear in the unvaccinated group, and we know the overall infection rate was higher in those weeks.

Adjusting for these remaining biases will not wipe out the vaccine effectiveness but will provide a more realistic and believable measurement.

 

I realize I still haven't gotten to the "partial vaccination" analysis. That will appear in a future post.

 

 

What the Danish study tells us about the CDC study on real-world effectiveness

In a prior post, I noted that in a Danish study of real-world effectiveness of the Pfizer vaccine among nursing home residents and healthcare workers, the researchers applied what they call a "calendar time adjustment". This adjustment is necessary because most unvaccinated people-hours were accumulated in the early weeks of the study and most vaccinated person-hours occurred in the later weeks -- while the overall rate of infection was dropping precipitously throughout the study period. As a result, even if the vaccine were 0% effective, a simple analysis falsely concludes that it is highly effective (I show how the VE can be computed as 67%.) This is a real-life example of a statistical paradox known as Simpson's paradox.

Does "calendar time adjustment" cure the bias? It's hard to say because the researchers did not publish their model. It's not clear how they incorporated the adjustment. Is it a simple linear term? Is it an interaction term? Is time measured in days, weeks, or months? There are many possibilities. Regardless, the calendar time adjustment shifted dramatically the effect of vaccination from 96% (counting only cases from 2D+7) to 64%. (Bottom row of the table excerpt shown below)

But how is the "adjusted" VE to be interpreted?

In reality, this number can't be interpreted without knowing the structure of the model they used. So, I will describe a common type of adjustment, found frequently in the typical social science journal. This adjustment involves adding a linear week-of-study main effect to the regression model. It's one of the simplest possible tweaks.

In the regression model with adjustment, we assume two factors affect the ratio of infection rates: vaccination status, and week-of-study. (By contrast, in the unadjusted model, infection ratios are affected only by vaccination status. This works for a clinical trial in which other factors are balanced by randomization. It assumes no other factors are biased.) These two factors are themselves correlated: the later the week, the more likely someone is vaccinated. Using this model, the researchers found that the VE is 64% on the first week of the study period. The regression model also outputs an effect of week-of-study (not disclosed in the paper). For argument's sake, let's say this is - 4% per week (expressed in VE units). This means on the second week of the study period, the VE is 60%, and then 56% on the third week, and so on.

Notice that the drop in VE is 4% between any two consecutive weeks of the study. This is obviously false based on the non-linear drop in overall infection rates from week to week. But if the analyst specifies a linear model, the result is a linear effect. Linearity is a consequence of the structural assumption, not of the data. The above discussion builds in several other structural assumptions*. The success of the adjustment depends on the correctness of these assumptions.

Not adjusting for a clear time bias is definitely bad - and leads to a vastly overstated VE. It's just not clear whether the adjustment is good enough, since the authors did not disclose the structure of their model.

* For example, the reference week does not have to be the first week of the study; it could be the last week, or any week of the study. The effect of week-of-study can vary depending on the effect of vaccination status.

 

The Burden of Case-Counting Windows

I know that the 64% VE estimate is still too high for a different reason. This brings me back to a topic you've seen on my blog a zillion times - the infamous case-counting window. Take a look at the results again:

Just like every other study, these researchers computes four different VEs based on four different case-counting windows. As discussed in this previous blog post, the VE increases as you run down the table because more and more confirmed cases are removed from the case count, effectively turning cases into non-cases.

[The red highlight is my own. This is a significant negative VE observed in the two weeks after the first dose of Pfizer, the period that is carved out of most studies. I see this as another real-world confirmation that one dose does not give sufficient immunity.]

Below is a new graphic showing what researchers are actually doing to the case curve as they apply different case counting windows. (I'm taking the definitions of case-counting windows used in the Danish study and superimposing them on the Johnson & Johnson case curve).

In the clinical trials, the same case-counting rules are applied to both vaccine and placebo groups. It's simple enough because the placebo group got two placebo shots.

In a real-world study, the unvaccinated group are those who reach the end of the study period without having taken any shots. Therefore, a case-counting window such as "14 days after 1st dose until 2nd dose" has no meaning for the unvaccinated group. As a result, in real-world studies, cases are nullified only for the vaccinated group. For each of the four calculations in the Danish report, the same number of cases was used to compute the infection rate of the unvaccinated group while fewer and fewer cases were counted in the vaccinated group as you run down the rows. This issue alone vastly overstates the VE when compared to the same case-counting rules applied from clinical trials. [This problem is avoided in matching studies in which the unvaccinated people are assigned a "day 0". Each methodology has both pros and cons.]

The use of case-counting windows means that there is zero chance a real-world study can replicate the findings of a clinical trial. Another problem looms for future studies because Johnson & Johnson is a one-dose vaccine, therefore the frequently-cited window of 2D+14 (14 days after the second dose) is also undefined. I'm sure J&J will cry foul if researchers counted all cases after 1D+14 for J&J and only cases after 2D+14 for Pfizer! Alas, the unvaccinated people are not represented by a pharma although as far as we know, they do pay their taxes.

Nevertheless, I don't believe that the primary purpose of real-world studies should be to replicate the findings of a clinical trial - as I said before, this is like asking a C student to grade the work of an A student. Instead, real-world studies should be used to fill in the gaps of our knowledge.

 

On to Today's Hot Potato

Yesterday, the media breathlessly recited the headlines from a new CDC study of healthcare workers and essential workers, saying that the mRNA vaccines are 90% effective, and 80% effective for "partial vaccinations".

It turns out that this CDC study uses a methodology very similar to the Danish study. So we can apply what we learned from the Danish study directly.

What do those numbers mean?

1. Let's start with the positive. Participants in the CDC study submitted weekly swabs for testing, and therefore this study measures infections, including asymptomatic cases. (This complicates any comparison with clinical trial results which do not count asymptomatic cases. We should expect the VE to be lower than in trials if we believe asymptomatic transmission to be an important factor.)
   
   
2. The 90% vaccine effectiveness is computed for a specific case-counting window that starts from 14 days after the 2nd dose, and adjusted for study-site bias.
   
   
3. In the VE computation, the case-counting window can only be applied to the vaccinated group. Therefore, the infection rate of the unvaccinated group counts all cases in the entire study period. This makes the estimate over-optimistic.
   
   
4. During the study period (mid December to mid March), the overall infection rate was dropping precipitously in the U.S. as it did in Denmark. However, the CDC analysis does not adjust for calendar time at all. This is another source of over-estimation.
   
   
5. The unvaccinated group was heavily biased toward several states with much worse infection rates than the vaccinated group. The published VE estimate incorporates a study-site adjustment. (This is similar in concept to the calendar-time adjustment described above.)
   
   
6. The unvaccinated group comprises people who are different from those in the vaccinated group on numerous demographic variables. While they adjusted for study site (i.e. residence) bias, they did not correct the gender, race, and occupation biases.

I will be back with a longer discussion of the CDC study soon. The "partial vaccination" analysis requires more space than I have here.

The obsession with 90% is perplexing because a vaccine does not need to be 90% effective to be a useful tool to fight the pandemic. This fixation is creating the wrong impression that anything less than 90% is unworthy.

Note on a Simpson's paradox in real-world vaccine effectiveness studies

You have heard it in the media: real-world studies of high-risk people such as nursing home residents or healthcare workers offer definitive proof that the vaccines' true efficacy is <insert a percentage above 90%>!!

These were the people who got the vaccines first, they are at greatest exposure to the virus, more of them are fully vaccinated (two shots), they have the highest risk of death, they are easier to track down. Ergo, the result is rock-solid, beyond criticism!

I wish someone had the foresight, and did a quick randomized trial while delivering vaccines to these groups, then we'll have a good dataset to draw some useful conclusions.

Alas, no one did (as far as we know) - which means all these studies are real-world observational studies. I discussed two examples of these studies (Mayo Clinic, Israel's Clalit health fund), both analyzing everyone in their customer database using the matching technique. Matching is used to create a synthetic control group (after the fact) in the hope of emulating a randomized control group as in a randomized, controlled trial (RCT). After matching, we apply methods commonly used to analyze RCTs, as if the conditions of RCT have been satisfied.

Today, I'm examining a Danish study of the Pfizer vaccine in two high-risk groups, nursing home residents, and healthcare workers. (link). This is an example of those real-world studies of high-risk groups that have so excited the punditry.

Counting People No Longer Works

Let's start with a simple calculation, and I'll explain why it fails. The study period is from December 27, 2020 to February 18, 2021, just under 2 months starting from the day Denmark initiated vaccinations. Based on the Figure 1 at the back of the paper, I estimated that during this period, there were 1,675 confirmed cases in Denmark among the nursing home population. The researchers said 382 cases were discovered among the vaccinated nursing-home residents, so there were 1,294 cases confirmed for unvaccinated residents. (The actual numbers used in their calculations are slightly lower because of exclusions but this doesn't matter for what I'm about to say.)

During the study period, 37,172 nursing home residents got at least one Pfizer shot so the case rate for vaccinated is 382/37,172 = 1%. For the unvaccinated group, I use the total number of nursing home residents who have not been infected as of December 27, which was 39,050. So the case rate for unvaccinated is 1,294/39,050 = 3.3%. The ratio of those case rates results in a vaccine effectiveness (VE) of 70%.

This calculation sounds vanilla but has a serious problem. Each nursing home resident who is counted as vaccinated is also counted as unvaccinated - because on December 27, 2020, the first day of vaccinations, everyone is unvaccinated. Then, over the course of time, those who have not been infected are eligible to get vaccinated. Upon vaccination, the nursing home resident transitions from being counted as unvaccinated to being counted as vaccinated. By the end of the study period, only about 5% of the residents are still unvaccinated. So 95 percent of this population are counted twice, and in each case, the persons spend the first part of the timeline as an unvaccinated and then the second part of the timeline as vaccinated.

Moreover, the proportion of time spent as unvaccinated varies depending on the day of vaccination. To deal with this issue, we change the denominator of the case rate. Instead of cases per person, we use cases per person-week. Each person is still counted twice, but in any given week, a specific resident is counted just once, either on the vaccinated arm or the unvaccinated arm but not both.

There were roughly 148K person-weeks accumulated on the vaccinated arm, and 164K on the unvaccinated arm, leading to new case rate estimates of 0.26% and 0.79% (interpreted as per 100 person-weeks). This leads to VE of 67%.

(In the second half of this post, I'll show why this 67% number is compatible with a completely useless vaccine. Stay tuned.)

Matching No Longer Works

For any real-world study, we ask why we can believe that the group labeled unvaccinated and the group labeled vaccinated are identical except for their vaccination status. In prior studies I featured on this blog, the researchers rely on the matching technique to assert this claim. Matching is infeasible here because the entire study is centered on two high-risk groups, for whom the progress of vaccination was rapid. Among the nursing home residents, about 5 percent were vaccinated during the first week, reaching almost 95% by the end of the study period.

The Danish researchers treated this issue differently from the Clalit researchers.

In the Israeli study, the investigators matched each vaccinated person with an unvaccinated one on a rolling basis through the study period, and they ended follow-up on both members of each pair when the unvaccinated person received the first shot. This matching and censoring scheme creates many undesirable side effects (for details, see my prior post):

• the follow-up time is severely limited because most matched, unvaccinated patients quickly became vaccinated, stopping the count on both members
   
• cases on the vaccinated arm are not counted because of the early stopping described above
   
• the number available for matching shrinks rapidly throughout the study period, leaving the majority of high-risk subgroups unmatched, and dropped from the study.

By contrast, in the Danish study, few are dropped from the study (only those who have been infected prior to the first day of vaccination). Each person is followed through the entire study period (until they get infected, died or become untrackable). For anyone who eventually got vaccinated, the person contributes to unvaccinated person-weeks until the day of vaccination, after which the person contributes to vaccinated person-weeks. They don't have extra early-stopping rules triggered by something that happens to an unvaccinated person so every case will be counted (except when they play around with case-counting windows, which I'll cover in a future post).

But there is one big problem with this design, which the researchers recognized and tried to resolve.

Vaccination Timing Bias

Because of the design described above, cases counted as unvaccinated typically occur earlier in the study period than cases counted as vaccinated. If you think about any given individual, the time progression is always from unvaccinated to vaccinated. So earlier infection dates are correlated with being unvaccinated, and later dates with being vaccinated.

This is particularly problematic because vaccinations were happening in the backdrop of a general decline in cases post-holiday, at least partially related to a national lockdown that started in Denmark a week before vaccinations began. To deal with this bias, the Danish study added a "calendar-time adjustment" to their regression model.

Unfortunately, such an adjustment is better than nothing but isn't sufficient. In the rest of this post, I'll walk you through a numerical example to show how this timing bias is a real-world example of what statisticians call a "Simpson's paradox". Then, in a next post, I discuss what a regression adjustment does, why it doesn't get rid of the problem, and also address other residual biases.

Simpson's Paradox

For this demo, I start with an assumption that the vaccine is useless (VE = 0). Then, I show that using the usual way that we compute vaccine effectiveness, we will conclude that the vaccine is extremely effective. This "paradox" is of the Simpson's type, well known to statistics students.

As mentioned already, the Danish study attempts to correct for this issue. This demo explains why they must do something about it. It also provides an estimate of the magnitude of this bias (huge).

As background, we start with some real data taken from Figure 1 of the paper: the progress of vaccinations during the study period, and the weekly trend of cases in nursing homes.

Note in particular that the case rate (gray) dropped markedly over the course of the study.

Assuming VE = 0 means each week, the split of cases between the vaccinated group and the unvaccinated group is equal to the split of people between the two groups. This assumption makes the group-level case rates equal each week. The VE in any week is zero since VE is based on the ratio of group-level case rates. Like this:

So far, so good. Now, we compute a cumulative VE using the standard method. We sum the cases and the person-weeks over all weeks, and then divide the two to get an overall VE for the entire study period. This is exactly the calculation I did in a previous section, which led to a VE of 67%.

I hope your head is where mine is: WTF!

We forced VE to be 0% every week of the study period (by assumption), and yet the cumulative VE for the entire period turns into 67%. This is known as a "Simpson's paradox". The aggregated statistic leads to a startlingly different conclusion than the disaggregated (weekly) statistics.

This is the reason why the Danish researchers introduced a "calendar time adjustment" to their analysis. They realized that they must do something to overcome this paradox.

What's Lurking Behind the Scene

We expected the cumulative VE to be 0% since it's just the average of a bunch of zeroes. The problem is it's not a simple average but a weighted average. The weights reflect the progression of vaccination through the study period.

The unvaccinated group is front-loaded, and shrinks over time while the vaccinated group is back-loaded, and expands. So the aggregate case rate for the unvaccinated group is weighted toward the first few weeks while the aggregate case rate for the vaccinated group is weighted toward the last few weeks. Remember that the overall weekly case rate was high in the first two weeks, and then started declining. This explains why the unvaccinated group with a much higher cumulative case rate than the vaccinated group despite our assumption that the vaccine is useless.

In the above diagram, you see that for the unvaccinated group (pink), the largest weights coincide with the several weeks with very high case rates while for the vaccinated group (blue), the largest weights fall on the most recent weeks with close to zero case rates. Simpson's paradox is a situation in which aggregating the data removes too much information, and distorts the interpretation.

The magnitude of this distortion is extreme! What should have been 0% shows up as 67%.

(In the preprint, I only have the proportion vaccinated at the start and end of the study. In the demo, I drew a straight line, which means I assumed an even pace through those weeks. The reality is that the earlier weeks were probably slower than the later weeks, which makes the distortion even worse.)

***

The situation of rapid vaccination and declining overall case rate are prevalent everywhere so every real-world study, particularly ones focused on high-risk groups, faces this problem.

This analysis does not show the vaccine is useless. It shows that if the researcher does not correct for this timing bias, the estimate of vaccine effectiveness is extremely optimistic.

***

My second book, Numbersense, opens with an example of Simpson's paradox, and then it keeps getting better. Get your copy here.

In the next post, I explore how the Danish researchers attempted to fix this bias.

 

 

 

 

 

 

 

 

 

 

Blood clots: is there a there

 

A number of mostly European countries have suspended administering the Astrazeneca-Oxford vaccine in order to investigate whether it has led to blood clots. These decisions have aroused angry reactions from the media. Here's an opinion column by Prof. Spiegelhalter that captures the mainstream view.

Many reporters have mis-spoken when they say the countries have banned the vaccine. They are pausing new inoculations so that the health authorities can assess the situation.

Spiegelhalter's argument - which was also that of Astrazeneca - relies on comparing the incidence rate of blood clots among those who have been vaccinated (millions) and the incidence rate of blood clots in the general worldwide population. He estimates that about 100 people in any given week experience blood clots, while so far only 30 cases have been reported post-vaccination (across many weeks).

As with any statistical argument, we should outline the unspoken assumptions under which this argument will prevail.

First, we must assume that the system used to collect information about blood clots is reliable and mostly complete. Spiegelhalter says the "yellow card" system used in the U.K. for reporting side effects has produced far fewer side effects than during the clinical trials. He interprets this as evidence that the vaccine is even safer than advertised. Another possibility is that the side effects are being under-counted. The hard part is to figure out by how much.

Second, we must assume that the people who have been vaccinated so far is a random sample of the entire worldwide population. On second thought, it's not obvious that the rate of blood clots should be the same in every country, or every age group. There may also be seasonality in the condition - the data on the vaccine has come only from the spring season while the general data are averaged over the year.

Third, we must assume that the characteristics of the people who have reported blood clots are similar to the average person who gets blood clots in an average year. It's worth noting there are reports of unusual blood clots in young vaccinated people. The problem may be blood clots in young people, not all people.

Fourth, we must assume that all blood clots are alike. Are there different types of blood clots? Are the reported blood clots similar to the average blood clot? Depending on the answer, we may need an incidence rate for specific types of blood clots (and also for specific age groups).

Fifth, we must assume the outcomes of those who reported blood clots post vaccination are similar to those in any given year. In particular, several deaths have been reported. Is the rate of death among those who experienced blood clots abnormal?

Sixth, I'd like to tally the incidence of blood clots in the unvaccinated populations as well. According to Spiegelhalter's back-of-the-envelope calculation, the incidence of blood clots is actually far fewer than expected (only 30 cases in several weeks vs 100 weekly), and so we should expect the data to reveal that the chance of experiencing blood clots in the last few weeks was much higher among the unvaccinated than among the vaccinated - assuming those weeks were normal weeks. (If this proves true, it raises the prospect that vaccination is protective against blood clots!)

Seventh, we must also assume that there is no one-time anomalous event that is causing a spike in blood clots. Some reports claim that there might be a bad batch of vaccine. It is possible for both of the following to be true: that the Astrazeneca-Oxford vaccine does not cause excessive blood clots, and that a particular batch of the vaccine caused an excess of blood clots among those received those doses.

Eighth, we must assume that the several weeks of data collected about the vaccine are fully representative of ongoing data. There is a precautionary tale here. At any moment in this pandemic so far, if we took the cumulative data and annualized them, we would have produced horrible forecasts of the future.

***

That's a lot of assumptions and the list is not necessarily complete. What we are dealing with is another observational (i.e. real world) study. Without the randomization of randomized clinical trials, it's just really hard to connect an effect to a cause.

Making assumptions is not fatal in statistical analyses - on the contrary, every statistical argument contains some set of assumptions. If I were conducting this investigation, I'd focus on collecting data to check the above assumptions. I believe the mainstream view may prove correct; this would imply most of the above assumptions hold.

It is also true that the current known incidence rate is small, benefits outweigh the side effects, etc. Notice that this is a different line of argument. The statistical argument claims the blood clots are normal and thus there is no problem at all. This second argument accepts the blood clots as abnormal but an acceptable price to pay; it is unnecessary if one agrees with the statistical argument.

***

I don't think it's wrong for European politicians to take a cautious approach. I wish the experts were not all over the media slamming the politicians and citizens about their alleged failure to understand science - because such ferocious rhetoric is drawing unnecessary attention to what should be a routine process. Just say you support a quick investigation, Astrazeneca should cooperate and provide any necessary data, and you expect the vaccinations will resume in short order because this is likely a false alarm.

 

P.S. [3/17/2021] Just listened to the hour-long press conference by the European Medicines Agency outlining the process of the investigation. They clearly state that they don't believe the extent of thie problem would ever outweight the benefits of the vaccine, but they also say the specific circumstances of those cases require examination. They expect to come to a quick conclusion, by Thursday afternoon.

Many of the issues mentioned in this post were brought up, such as specific types of blood clots, batches of vaccines (or manufacturers), and establishing base rates for specific time frames. The committee of experts will be looking into those and other factors. One reporter asked an important question - whether there will be a biological investigation to go along with the statistical analysis.

The scientist representing EMA mentioned the back-of-the-envelope calculation that would indicate that people who were vaccinated were far less likely than normal to have blood clots. I don't think this is their strongest argument; unless the vaccine somehow prevents blood clots, you'd have expected the frequency to be roughly normal - and if it's not, you now have to go down the rabbit hole of explaining the biases in the vaccinated population, which I can assure you opens up a can of worms.

Also worth noting (which I did in a comment below) is that both sides could be right. It may be that the vaccine causes excessive, potentially fatal blood clots of a specific type in a specific subpopulation. That means the vaccine is still safe for almost everyone. This bolsters the cost-benefit argument. The end result may be just a warning about a rare side effect for people with specific conditions.

 

Take 2 on the Johnson & Johnson vaccine trial results

In the previous post, I characterized the vaccine efficacy (VE) of Johnson & Johnson's vaccine (J&J) as 67%, which is the headline number in the FDA report.

The VE Constellation

If J&J had offered a single estimate of vaccine efficacy, we'd have been shocked.

They centered our attention on 67%, which is described as the VE for "centrally-confirmed moderate to severe/critical Covid-19 cases" with a case-counting window starting 14 days after the first shot (1D+14).

The following list is a partial run-down of other VE estimates that also appeared in the report, and how they relate to the headline number.

If the case-counting window is shifted forward to 1D+28, the VE becomes 66% (lower by mere decimals). This estimate throws out another two weeks of cases, and was added to the clinical trial protocol midway through the trial. I can't think of a reason to justify this metric, other than to compare it directly to Pfizer's headline number (2D+7 = 1D+28).

If that's the case, we also need J&J's VE starting the count at 1D + 35 to compare to Moderna's headline VE at 2D+14. This is not directly cited in the report but it's still 67%.

The VE is 55% if case counting starts the day after the first shot. This metric is not referenced in the text of the report but can be obtained from the cumulative incidence curve I featured in the last post.

The numbers above aren't quite the same as Pfizer's or Moderna's because J&J removes "mild cases." However, J&J reported only 4 "mild cases with onset >= 14 days post-vaccination". This makes the definition of "mild" pointless (unless there are many mild cases prior to Day 14, which we do not know). The VE including mild cases is still 67%.

The headline VE also does not contain all known cases by the time of data cutoff. That's because J&J requires "central confirmation" of positive cases. About 30% of cases have not been confirmed as of the FDA meeting. These cases were excluded from the analysis - which has the same effect as assuming they are false positives. The report said that if all the unconfirmed cases came back confirmed, the VE will be 66%, decimals fewer than the headline number but on the other side of 0.5.

J&J employs adjudicators to decide if a Covid-19 case is "severe." Therefore, they report another VE that ignores moderate cases. This secondary endpoint was added midway through the trial in an amendment to the trial protocol. When moderate cases are removed, the VE goes up to 77% at 1D+14, and 85% at 1D+28. The definition of "severity" includes ICU admission and deaths but neither is necessary, and the adjudicators have the final say. We do not know how many cases they adjudicated. It is not clear how clinicians can utilize such a finding unless "severity" is predictive of hospitalization, intubation, death, etc.

In describing the above result, the report says "Efficacy against severe disease appears to be greater when cases that occurred before 28 days are excluded." Mathematically, the severe cases on the vaccine arm are more front-loaded, and if the earlier cases were excluded, the VE rises. This exposes the very odd enterprise of reducing case counts through exclusions, which almost always leads to increased VE. This increased VE has a more restrictive interpretation (only severe cases, only cases occurring after 28 days, etc.) so nothing practical is gained. The vaccine is less effective, not ineffective, in those excluded areas.

Next, there is a VE for hospitalizations, which was 82% at 1D+14 and 100% at 1D+28. We are now getting into the territory of rare events, and the error bars on those numbers span 80%. Not surprisingly, this analysis is post-hoc, not originally specified in the trial protocol. There are several inscrutable sentences about what counts as a "hospitalization" on p. 33 of the FDA report, and if interested, I'd direct you there (link to PDF).

There is one other VE estimate that should interest my American readers. The J&J trial has participants from the U.S. (~50%), Brazil (~20%), South Africa (~10%) and other South America. For U.S. participants, the VE was 74% at 1D+14, and 72% at 1D+28. These estimates included the unconfirmed cases, which means it is only comparable to the number that includes those (66%). The size of the trial is not large enough to conclude that VE is higher for Americans (the confidence intervals completely overlap).

Double Zero

Recently, the media started describing the primary purpose of vaccines as preventing deaths, even though none of the clinical trials have been sized to measure the death rate with the required precision. In several trials, including J&J, there have been zero deaths attributed to Covid-19 in the vaccine arm, and people then erroneously concluded that the vaccine provides 100% protection against death.

Most lottery players have never won a lottery. However, all lottery players do not conclude that their chance of winning is 0%.

J&J reported 7 deaths on the placebo arm that have been adjudicated as Covid-19 related. All seven occurred in South Africa. So, in the U.S. section of the trial, the verdict thus far is 0 vs 0. If you're an optimist, that's 100% protection. If you're a fatalist, that's 0%. If you're a statistician, you'd wait for more data.

Double Testing

The central confirmation of positive cases is, as far as I know, unique to the J&J trial. This procedure is effectively a guard against false positives but since negative tests are not checked, the double testing always reduces the number of detected cases. The FDA report discloses that typically about 90% of the positive results are confirmed by the central lab.

The central lab uses the Abbott Alinity RT-PCR test, the result of which over-rides local test results. It appears that this type of tests has a false-negative rate of at least 20% (link). The other trials do not re-confirm positives.

Let's assume 10% of all positive test results are thrown out by the central lab. The report tells us nothing more. It is not at all clear that the confirmation rate is even across all cases. Is the probability of confirmation the same between placebo and vaccine cases? Is the rate of confirmation the same regardless of the date of onset of cases? Any such imbalance affects how we interpret the data.

Double Blind

The J&J trial is double-blind while Pfizer/Biontech, Moderna and Astrazeneca-Oxford were all single-blind. This rules out a potential source of bias, and raises the question of why the other pharmas did not conduct double-blind trials.

Blinding reduces the chance that participants know their vaccination status and thus behave differently. Despite being double-blind, there are hints that some people in the J&J trial may have inferred whether they got the vaccine or the placebo. On Table 4, we learned that there was double the number of people who withdrew from the study on the placebo arm (72) than the vaccine arm (32) - this difference is highly significant, even though the rate of dropout is small.

Double Jeopardy

As with all trials, J&J tests each participant on the day of the first shot before innoculating him or her, and throws out anyone who tests positive. Because of this reason, J&J removed about 4,000 people, which seems quite high in a trial with 40,000 enrollees.

While these participants are no longer part of the trial, they presumably took the shot before their test results were available, and were thus followed up. Surprisingly, the vaccine appeared to be ineffective in this subgroup at both 1D+14 (3 vs 4) and 1D+28 (1 vs 2), and also ineffective against "severe" disease (1 vs 0).

Double Alibi

In a side analysis about "asymptomatic cases," J&J disclosed 3 cases that were originally deemed asymptomatic but later re-classified as symptomatic. The investigators said they analyzed symptomatic cases first since it's the primary endpoint, at which time those cases were deemed asymptomatic and thus excluded. When it came time to look at asymptomatic cases, the cases have been re-classified, and thus were excluded from the secondary analysis. In other words, those cases fell through the cracks, neither here nor there.

Perhaps only statisticians find this little episode amusing. We're talking about a few cases. This shows the high stakes in the vaccine competition among the pharmas; every case matters.

 

The third worst-kept secret

It's the third time, and we've noticed a pattern. The FDA can't keep a secret. In late February, they approved the emergency use of the Johnson & Johnson (J&J) vaccine. On the day the Advisory Committee met to deliberate in public, there was zero suspense since the entire process was choreographed weeks in advance.

The headline result from the J&J trial are these:

J&J reported a vaccine efficacy of 67% during the trial. This result from a single multi-national clinical trial implies that the vaccine efficacy is above 60% in general. Emphatically, this statistic should not be interpreted as 6 out of 10 vaccinated people being protected (and 4 left unprotected).The baseline case rate after two months was about 2 percent during the J&J trial so 98 out of 100 participants did not get ill. Upon vaccination, everyone's chance of getting infected is cut by 60%. In addition, J&J also documented few side effects of concern. Given the FDA's stated intention of approving any vaccine that has over 30% efficacy (on par with the flu vaccine), the outcome of the meeting was never in doubt.

As you must know by now, the J&J VE number cannot be directly compared to VEs from other trials because the FDA allows each pharma to decide what it counts as a case, and which time window is used to count cases. J&J counts cases starting 14 days after the inoculation, and uses a more stringent set of rules to qualify a "case".

Since all vaccine trials are sized to deliver a preliminary verdict as fast as possible, none of these trials have sufficient data to draw strong statements about subgroups - especially not at the interim analysis. This is true also of J&J. In addition, it is premature to talk about the probability of rare events, such as "severe" cases, hospitalizations and deaths. This has not stopped the media from trumpeting "exploratory" and sometimes "post-hoc" findings in the same breath as the overall VE estimate, which is a more solid result.

(Notably, none of the post-hoc analyses comes with corrections for multiple comparisons, which is a basic tool to reduce the chance of spurious findings; the more things you look for, the more things you'll find, the greater the chance that your discovery is not replicable. This comment applies generally to all vaccine trials.)

The J&J vaccine is an adenovirus vector, so it's like the Astrazeneca-Oxford technology (link), and different from the mRNA technology used by Pfizer/Biontech (link) and Moderna (link). The adenovirus technology is also relatively new, having only had one prior drug approved for ebola by the EU while the mRNA has never had any (not counting the two EUAs granted for the novel coronavirus).

The J&J vaccine can be stored in standard refrigerators, which simplifies storage and transportation. However, the shots are still packaged in groups of 5 doses; once unpacked, the entire lot must be consumed within 2-6 hours.

The J&J vaccine treatment evaluated in this trial is a single shot. However, the pharma has an ongoing two-shot experiment that began in November (more later).

Exploring the case curve

Like the other pharmas, J&J provided a cumulative case curve starting from Day 0 (the day of the shot).

Let's review how this curve should be interpreted. In theory, the vaccine efficacy is eyeballed from the chart. If you pick a time, say Day 56, the cumulative incidence rate of the placebo group is roughly 2 percent (red line) while the cumulative infection rate of the vaccine group is just under 1% (blue line). So the vaccinated group experienced half of the incidence rate of the placebo group, which translates to 50% improvement, or VE of 50%. In short, the blue line is at half the level of the red line. (To be precise, the VE was 55% by Day 56.)

As with every trial, what's shown on the cumulative case curve does not lead to the headline VE number (which is 67% in the case of J&J). The headline number nullifies cases in the first 14 days. This is the same as resetting the count of cases on Day 14 to zero (while using the at-risk population on Day 0). I explained this procedure in greater detail in this blog post. Using the same technique as above, we see that the blue line is now at a third of the level of the red line. This two-thirds improvement corresponds to VE of 67%.

Since the primary success metric is VE, we should plot VE directly, which leads to the following chart:

I'm only drawing up to Day 56 because by that time, the analytically-ready population has already been cut by half - as most of the trial participants have not yet reached that amount of follow-up time. Remember that the FDA allows analysis when only half the participants have reached two months after treatment (currently interpreted as 56 days rather than 60 days). The data beyond that period is based on a small sample, which is also biased because of staged enrollment of lower-risk people first.

It is instructive to show J&J's curve on the same chart as Pfizer's.

Firstly, the Pfizer curve is much smoother throughout the first 56 days, with the difference particularly stark during the first 14 days. J&J's curve is a step function: VE is basically zero for two weeks, then VE magically jumps to 45 percent, and grows slowly thereafter.

On the one hand, someone can look at the J&J curve and say this justifies the presumption that VE = 0 in the first two weeks. The step function embodies the idea that the vaccine is useless for two weeks. On the other hand, someone can say a biological process affecting a large population over time should be smooth rather than a step function. My view is the latter so I wonder what happened with case counting in the first two weeks. Unfortunately, because J&J discounts cases in the first two weeks, the study report has very limited data for that period, e.g., it provides no information about how many "mild cases" were detected during the first 14 days.

Secondly, Pfizer's treatment has two shots, the second one administered around Day 21 while J&J's is one shot on Day 0. Interestingly, the two VE curves are closest together at Day 21 and then the Pfizer curve rises faster than J&J's. If I worked for Pfizer, I might see this as evidence that we timed the second dose just right. (Of course, there can be other reasons for the rate divergence.) When J&J has results for its two-shot trial, we'll have confirmation about this hypothesis.

Next, I look at weekly data, rather than cumulative.

The chart shows VSC (the vaccine's share of cases for a given week), and the timeline starts at 14 days after the first shot. The lower the VSC, the higher the VE. This view reveals the vaccine efficacy to be quite stable from week to week. The first data point for J&J (red) is the first week that J&J considers official, and has the lowest VSC (or peak VE), although we can't tell if these are just random fluctuations. The cumulative chart hides this information because the most recent week is weighed down by the data from previous weeks.

***

A good question to ask is whether every person requires the same level of protection from a vaccine. I believe the answer is no. People who have lower risk of exposure, people who are younger and fitter, people who take other mitigation measures more seriously, people who have fewer daily contacts with others, people who intend to wear masks after getting the vaccine, etc. have a lower baseline risk of infection, and so they may benefit from a one-shot vaccine.

I have quite a few pages of notes left, and cover some other interesting observations in the next post.