The statistics behind the J&J blood-clot decision

Last Friday, the U.S. ended the pause of the Johnson & Johnson vaccine, which may be linked to rare blood clots in women under 50. As with the EMA's stance on the Astrazeneca vaccine, they recommend continuing to administer these vaccines based on a cost-benefit analysis.

In today's post, I summarize the key information in the documents released during Friday's meeting of experts about those blood clots. The media have failed to cover the statistical reasoning behind the decision.

In its presentation, J&J highlighted the number that is most beneficial to its commercial interest - the risk of rare blood clots in the general population regardless of gender and age. The media is also fixated on this statistic, which is the least useful of all the numbers in these documents. Out of 8 million people who have been given the J&J shot since March (remember that this vaccine only got approved at the end of February), they have reported 15 cases of rare blood clots coupled with low platelet counts. This means the risk is 2 cases per million.

Measuring the risk of this side effect is a great example of why no statistic is purely objective. The subjectivity lurks behind how the risk is defined. Risk is a ratio of two numbers: the number of cases, and the relevant population at risk. The 2 cases per million number bakes in two choices: on the denominator, J&J selected the entire population regardless of age and gender; on the numerator, J&J selected the co-occurrence of rare blood clots and low platelet counts. Those are not the choices I expected. 

The denominator should be women only, or younger women only, given that all 15 cases occurred in women under 60 years old. The choice of the denominator is not immaterial! The CDC ultimately used women aged 18-49 as the relevant population, and this change of denominator bumped the risk from 2 up to 7 per million. (Including men roughly doubles the denominator while adding zero cases to the numerator.)

The cutoff age of 50 is also a choice. Two cases were found in women between 50 and 60 years old. By setting the at-risk population as women under 50, those two cases are removed from the numerator. Each additional case increases the risk by roughly 0.5 times if the denominator is not changed appreciably.

The CDC also looked at smaller subgroups. Women in their thirties accounted for the most cases, and the risk in that age group was 12 per million (7 cases among 600K vaccinations). Meanwhile, 1.4 million women aged 50-64 took the J&J vaccine, with two reported cases of blood clots and low platelets, leading to a risk of < 2 per million. Scientists can choose the most restrictive analysis by focusing only on women in their thirties, or the most expansive analysis, on all women. The CDC's choice of 50 is reasonable while not the only option.

The numerator is also unusual, and different from what the EMA used when examining the Astrazeneca vaccine. J&J and the CDC decided to define the side effect as rare blood clots coupled with low platelet count. This is a more restrictive case definition, which reduces the number of cases under study. Interestingly, this definition neatly draws a line between the adenovirus platform and the mRNA platform (Pfizer & Moderna), as so far, the mRNA vaccines have seen some cases of blood clots but zero cases of blood clots plus low platelets.

***

The next issue is how bad is that risk. For this, we have to compare to other risks. 

One obvious comparison is the baseline rate of blood clots and low platelet count. This is where the choice of numerator presents a difficulty. The baseline risk is essentially zero: that combination of conditions almost never happens. One of the CDC documents compared the observed risk to the risk of blood clots in women 20-50 years, which is invalid. The risk of blood clots plus low platelets should be quite a bit lower than the risk of blood clots (with or without low platelets). In a different CDC document, the risk of blood clots and low platelets is estimated to be under 1 case per million. In the J&J document, they claim the baseline risk is 0.1 per million. Neither of these numbers can be directly compared to the 7 per million observed risk because the baseline risk is per million Americans, not per million American females aged 18-50.

Nevertheless, it is abundantly clear that the risk of getting blood clots and low platelet count among women under 50 who took the J&J shot is much higher than the baseline risk. The risk of dying from this side effect is also much higher than the baseline mortality risk. (About 25-30% of the known cases have already died.) On an absolute scale, the condition is very rare. On a relative scale, it is concerning.

All 15 cases were reported between 6 and 15 days after the vaccination. Notably, this is the period during which scientists insist the vaccine has zero benefits. 

***

Another relevant comparison is the risk of death from Covid-19. This is a key to the cost-benefit analysis. How should we compute this risk? We should focus on the period from March and April, the two months in which the J&J vaccine was administered. 

This CDC slide indicates that among females aged 18-50, the Covid-19 death rate was around 0.3 per 100,000, or 3 per million. This means the chance of dying from Covid-19 is about the same as from dying from blood clots+low platelets after taking the J&J vaccine.

Yes, that's right. It's also confirmed in the CDC simulation model. 

So, what does the CDC mean when it says the benefits outweigh the costs? The simulation model finds that slowing down the vaccinations causes a lot of excess Covid-19 deaths among men, and among females over 50, and therefore, across all age groups and genders, stopping J&J causes more harm than good.

***
Two other small details caught my attention. There is another fatal case of blood clots and low platelets that was excluded - seemingly because the situation is too complex. There are also other cases being investigated including cases affecting men. 

When the investigation first started, there were 6 reported cases. About 10 days later, the number of cases more than doubled to 15. The vaccine didn't become more risky suddenly. This phenomenon is caused by self-reporting of cases. Such self-reporting has always been a weakness of how we monitor side effects. The risk may be over-estimated because of under-reporting of mild cases.

 

 

 

 

 

 

 

 

Comments

[A Palaz]

Hey Kaiser,

So some points from quick read.

The denominator can be changed, so of course can vaccine administration to match whatever group it contains.

So it is clear that a higher risk as you say for groups 30s.

Some notes on covid risk . The risks are much higher with comorbidities. So did vaccinated Clot ladies have comorbitities?

Notes on CDC analysis. Ladies of 18-29 have a far higher risk of covid hospital lization than men 8 vs 5 (just estimate form graph bars) but much lower mortality 1.5 vs 3 . Note in general women's of this age
have higher hospitalization rate than men. So why do women of this age have particular risk in covid hospitalization?

On blood clots and platelet
think it is chosen because there is a known antibody factor they have observed with linkage to this condition.

On blood clots. First why would a vaccine related factor (or covid) choose only one place in body for blood clot location?
Second in my opinion if we are to talk blood clots we would have to
include all - meaning many undetected and many that have progressed to brain heart lung and other should be included.

On vaccines. To my mind I think all these vaccines target the spike protein, correct me if I am wrong. So the stimulated immune reaction should be the same. Why then would side effects differ? Dosage, adjuvabts , vectors, and possibly something else?

Maybe also as you say reporting and self reporting differ andborvtiming of delivery to younger groups.

I find it quite extra or dinars the lack of seriousness,(?) in depth of data and analysis after over one year doing science

[Ed: The original comment was duplicated - the repeated section is removed.]

[ATM]

"This CDC slide indicates that among females aged 18-50, the Covid-19 death rate was around 0.3 per 100,000, or 3 per million. This means the chance of dying from Covid-19 is about the same as from dying from blood clots+low platelets after taking the J&J vaccine."

Is that supposed to be females aged 18-29? Otherwise, I can't quite get the numbers to make sense...

[Kaiser]

ATM: I'm eyeballing and averaging the first two bars. The second one is around 0.5, the first one is less than 0.2. I am also saying "about the same" because these are very small numbers and you need to imagine error bars around them.