Big Data, Plainly Spoken (aka Numbers Rule Your World)

What do we know about the Sinopharm vaccine? 2

The other day, I posted some notes about the JAMA paper featuring the interim analysis of the Sinopharm vaccine trial, a Covid-19 vaccine developed in China using the tried-and-true inactivated virus technology. The trials were conducted in UAE and Bahrain and the follow-up period for the median participant has reached 3 months since the second dose, which is a longer follow-up period than the earlier trials.

One takeaway from Part 1 and also my previous reviews of vaccine trial results is that vaccine efficacy (VE) numbers are not easily comparable. Trials were conducted in different countries at different times; definitions of cases were not standardized; case-counting windows varied. The duration of follow-up also matters because we expect the effect of any vaccine to wane over time.

***

In Part 2 of my review, I explore eFigure 1 in Supplement 3 of the JAMA report. This extremely important diagram traces how "cases" are qualified to be counted in the VE metric. All vaccine trials - not just Sinopharm - count only adjudicated, lab-confirmed, symptomatic Covid-19 cases.

The final VE metric in the Sinopharm trial included only 142 cases across 2 treatment arms and 1 placebo arm. e eFigure1 (partly shown on the right) revealed that there were 962 suspected Covid-19 cases (with symptoms) at the start of the process. That number was cut by 50-60% to 420 based on PCR testing negative.

The adjudicators then used their judgement to remove another 30-50% of the symptomatic, PCR-positive cases. We have 255 cases standing at this point.

Then the case counting window is applied, and another 113 cases vanished from the count. That leaves 142 cases that ultimately entered the VE calculation, resulting in the headline number of 70-80%.

For the sake of argument, if we got rid of the adjudication step, then the VE would have been reported as 37-56%. Because this calculation includes cases from first dose, compare it to 50-66% that was mentioned in the previous post. Thus, adjudication increased VE by roughly 10%.

Adjudication is not a unique feature of the Chinese trial. All vaccine trials employed adjudicators. The other pharmas have been far less forthcoming about what happened during this process. Pfizer, for example, disclosed to the FDA that they had 3,410 suspected cases (link) but I have not found any details on how that number got winnowed down to 170 cases actually counted in the headline 95% VE number.  

***

If you have the stomach for it, I am about to open another can of worms. This happens to be a very material issue common in many settings. When comparing two numbers, one can compute a ratio (sometimes called an index) or a difference. Ratios are very tricky to deal with, and easy to misinterpret. The VE is a ratio of case rates (think of it as a relative risk index).

In the Chinese trial, the headline VE was reported as 73-78%, based on 142 adjudicated cases of Covid-19 across the three arms. In one of the sidebar analyses, the scientists added "asymptomatic" cases, defined as cases found by PCR tests for which the participants did not report experiencing qualifying symptoms.

The following table shows the data disclosed in the report. You can see that the vaccines are clearly not as effective at reducing "asymptomatic" cases, as the differences between 16, 10, and 21 are not statistically significant.

In order to simplify the calculations, I will work with the following idealized example. I'm assuming 10,000 people in two arms. The vaccine arm counts 35 cases of which 20 are symptomatic. The placebo arm has 115 cases of which 100 are symptomatic. We can compute the vaccine's share of total cases to be 23% so that the corresponding VE is 70%. (In this calculation, I include asymptomatic cases.)

An alternative efficacy metric is to compute the difference in the case rates. This means 1.15% (115/10000) for placebo minus 0.35% (35/10000) for the vaccine, resulting in a difference of 0.80%. This number has a straightforward interpretation: for every 10,000 people vaccinated, 80 fewer get sick from Covid-19 relative to placebo.

What happens if we ignore asymptomatic cases from both arms as the pharmas did? The difference in case rates remain at 80 per 10,000 because we remove 15 from each side. This makes sense intuitively since the vaccine is assumed to be ineffective at reducing asymptomatic cases (15 vs 15). Therefore, whether we count symptomatic cases or all cases, the vaccine reduces the number of cases by 80 per 10,000 relative to placebo.

As a relative risk ratio, the VE actually increases from 70% to 80% after removing asymptomatic cases. Note though that the 70% reduction is off the total case rate while the 80% reduction is off the symptomatic case rate, which is a smaller number. This points to the challenge of interpreting VE as a ratio. All the different VE ratios based on different inclusion and exclusion criteria have different base rates and so they cannot be compared to each other! Eighty perent is not always better than 70 percent: 70 percent of a big number can be more impactful than 80 percent of a small number.

You can now see why the game of the vaccine trials is to restrict the VE calculation to regions where the vaccine performs well. (I'm using "regions" in an abstract sense which includes the geographical sense.) Such restrictions lead to a higher ratio reduction but the result is only relevant to those regions, usually where the base rate is lower. In other words, it's a conditional number, as I explained in the last post. It's a higher reduction off an often unintuitive, restricted base rate. The case-counting window is the same game as asymptomatics but with a different name.

***

Another frequently ignored issue in these vaccine trials is the weak blinding. The Chinese trial is actually double-blind whereas the Pfizer, Moderna, etc. are single blind (very commonly mis-reported as double-blind because medical reporters just assume all clinical trials are double-blind).

Blinding is a protective measure that prevents investigators or participants from behaving differently - conscious or not - because they know which arm individuals are in. When the placebo is saline solution, it's actually very hard to blind participants because those getting vaccines are much more likely to experience a variety of symptoms, such as numbness, headaches, mild fevers and so on.

As a result, a few designs use non-saline placebos. The Chinese trial used an adjuvant in the placebo. An adjuvant is an additive in the vaccine that helps it work better. It's unclear to me whether this placebo is preferred to saline. It depends on the chance that the adjuvant causes mild side effects similar to the vaccine (which is a good thing in terms of masking treatment status) and the chance that the adjuvant by itself could be beneficial (which makes the base rate higher and raises the bar for the vaccine). 

The only other exception I know of was one of the Astrazeneca trials, in which the placebo was a different unrelated vaccine. Participants might be less able to guess which arm they belong to in that trial. 

***

Finally, none of these trials is really serious about measuring asymptomatic cases. The only exception is again in one of the Astrazeneca trials: they actually asked participants to send in nasal swabs every week of the trial. In the typical vaccine trial, they only do PCR testing (a) if symptoms are self-reported and (b) at infrequent intervals, including before each shot, and then once a month or longer. With infrequent surveillance, only a fraction of the asymptomatic infections can be found.

 

Adolescent trial showcases pre-registration and the Law of Small Numbers

The scientific report on Pfizer's adolescent trial did not bring closure on the two key issues I flagged in my previous post (link). Here are my notes on this trial. 

The Key Efficacy Finding Does Not Require a Randomized Controlled Trial

The researchers explained that based on the low expected Covid-19 case rates among 2,000 teenagers (split into two groups), they were not going to see enough cases to offer the same kind of analysis as in the adult trial. They have convinced the FDA (or the other way round) to accept immunogenicity findings as a substitute. That is to say, Pfizer will show only that injecting teenagers with the vaccine generates antibodies intended to fight the SARS-CoV-2 virus. 

The sample size issue is real. The infection rate in teenagers is well known to be much lower than in adults. It's hard to find national numbers split by age group, though. I found out that in California, roughly 4% of children below 17 have tested positive for Covid-19 during the whole pandemic while the number was close to 10% for people above 18. These numbers are cumulative for the 12-15 months of the pandemic whereas a typical vaccine trial spans 3-4 months at the time of interim analysis. Detecting a difference in tiny signals (rare events) is challenging and requires a large sample size. Two thousand is like 20 times fewer participants than the adult trial. 

Typically, immunogenicity results advance a treatment to Phase 3. These lab tests prove that the vaccine produces a good amount of antibodies as intended. However, having antibodies does not guarantee the vaccinated people will be protected from infection. Nor do lab results suffice to decide what level of antibodies is sufficient to reduce infection (and transmission). Further, Phase 2 trials involve few people. Thus, a Phase 3 trial is required for FDA approval. Phase 3 trials involve thousands of people, and measure clinically relevant outcomes, which for a vaccine usually means reduction of infection.

With immunogencity as the key efficacy metric, the adolescent trial is in spirit an expanded Phase 2 trial. You don't even need to have a placebo group, nor randomize treatment. Injecting someone with saline will not produce antibodies against the coronavirus. They also measured antibodies in only 200 kids, a 20%-ish subset of those enrolled in the adolescent trial. They could have obtained the same outcome by recruiting 200 kids, giving them two shots, and measuring their antibody levels afterwards. No RCT is necessary.

The average antibody level of the 12-to-15-year-olds is compared to a number they previously obtained from the adult trial, involving those in the 16-to-25-year-old age group. If similar levels of antibodies are found, they presume that the vaccine will work in teenagers.

There are several assumptions required to justify such a claim. First, they assume the vaccine works with the same efficacy for 16-to-25-year-olds in the adult trial as the overall VE. This is not a given since the trial did not contain enough people in that (or any other) age group to produce a reliable estimate of subgroup efficacy. Second, they assume there are no material differences between the two age groups that affect immune response. Third, they assume that a direct link between antibody levels and reduction in case rate has been established in the 16-to-25-year-old age group. If that is true, I haven't seen the evidence.

In the protocol, the scientists were supposed to provide an "immunobridging" analysis, which sounds like a mathematical model that justifies the third assumption. They ultimately did not come through with that model. 

 

The Trial Provides Minimal Data on Safety

While the placebo arm is redundant for the pre-specified efficacy analysis, the placebo participants are useful to establish a background level of side effects, to help the vaccine developer make the case that the vaccine does not cause more adverse incidents than expected.

Nevertheless, the conclusion one can draw from a study of 1,000 vaccinated participants is highly limited. If a deadly side effect has a 1 in 5,000 chance of happening, there is over 80% chance that there will be zero cases of this side effect during this trial. With probably ~15 million Americans aged 12-15, such a side effect would affect 3,000 kids. Fewer than 200 American teens have died from Covid-19.

The trial did not identify any major concerns but what we don't know, we don't know.

 

What about that 100% VE talking point?

On Twitter and cable TV, the talking heads have mostly not presented the trial results as I have above. Below is a typical example:

Notice that they ignored the immunogenicity results and trumpeted a vaccine efficacy (VE) number. In the study, Pfizer reported a "surprise," saying that they found 16 cases of Covid-19 among the placebo group, and zero cases in the vaccine arm. 

In the NEJM paper, on the other hand, the immunogenicity outcome was unmistakably featured as the primary endpoint. This raises a critical scientific issue that is perhaps hard for an outsider to grasp.

All clinical trials are required to submit protocols and analysis plans well ahead of enrollment. This process is called pre-registration (investigators are not supposed to make edits, especially major edits, while the trials are running although this practice has become a casualty of the pandemic). Pre-specifying your analysis prevents data mining for results after the data have materialized. Such data mining frequently results in spurious findings - mistaking one-off phenomenon as generalizable outcome. Spurious findings will not replicate if the trials were to be repeated.

In the adolescent trial, pre-registration made Pfizer think about how to demonstrate efficacy. They feared that using the usual measurement based on symptomatic, PCR-positive Covid-19 cases is likely to saddle the study without a conclusion because they were only willing to run a trial with 2,000 participants. Therefore, they chose the immunogenicity outcome as the primary endpoint.

Now, after the trial finished, the Pfizer press release - and its readers - trumpeted the 100% VE metric as if that was the pre-specified analysis. This behavior usurps the principle of pre-registration. 

So let me now explain why there is a good chance the observed VE is an outlier. If the whole trial were repeated, I'd be surprised to see the same result repeated.

According to the FDA's press release on approving Pfizer's vaccine for teenagers, the case rate in the placebo arm was 16/978 = 1.6%. In Pfizer's adult trial, the corresponding rate was 162/18325 = 0.9%. So, the teenagers were getting sick at a rate 80 percent higher than that of adults. If I assume that the kids were equally likely to get infected as adults at 0.9%, the chance that we would find 16 or more cases in 978 kids is less than 2 percent. If we were to run this trial 100 times, 98 or more of them would observe fewer than 16 cases in the placebo group.

But teenagers have less than half the case rate of adults as mentioned above. If I assume that the case rate for kids is at 0.45%, the chance that we would find 16 or more cases in 978 kids is 0.001%. Don't forget, also, that the case rate only counts adjudicated, lab-confirmed, symptomatic Covid-19 cases occurring at least 7 days after the 2nd dose so the real-life infection rate is even higher than the 1.6%.

This is very strong evidence that what was observed during this trial was an outlier event. 

Nevertheless, this outlier event has been painted as a "pleasant surprise". We didn't think we'd have enough cases to show anything but how could we refuse the unexpected gift of 100% efficacy? The real surprise was not the VE but the inexplicable, highly unlikely incidence of Covid-19 among teenagers enrolled in this specific trial.  

In fact, this is exactly the reason why pre-registration is good practice. If we allowed this type of post-hoc data mining, spurious findings will bloom. 

***

See also my previous rant about the way the paper was written up as if the adolescent and 16-to-25-year-old age group were originally planned as two arms of a trial.

See also my post on the Law of Small Numbers, one of the seminal contributions by Tversky and Kahneman. The Law of Small Numbers is the false belief that the Law of Large Numbers applies to one's small sample.

 

 

 

 

What do we know about the Sinopharm vaccine? 1

WHO has approved the Chinese "Sinopharm" vaccine for emergency use (link), soon after an interim analysis of its Phase 3 trial was published in Journal of American Medical Association (report). We have heard nothing from the FDA about Sinopharm, nor has there been any news on the Astrazeneca/Oxford vaccine, so the only vaccines that have been approved in the U.S. remain three homegrown vaccines.

The Sinopharm vaccine is notable for using the long-established science of inactivated viruses. The vaccine does not require special cold storage, and its long-term safety record is supported by successes in fully approved flu and polio vaccines, an advantage that the newer technologies do not enjoy. The primary unknown to be resolved in the Phase 3 vaccine trial was the efficacy against SARS-CoV-2, as the flu shot based on inactivated viruses have "low" but still accepted efficacy in the 30-50% range (link).

The Phase 3 trial brings the welcome news that the Sinopharm vaccine also has efficacy in the 70-80% range against placebo, on par with Johnson & Johnson and Astrazeneca, both of which come from the adenovirus family. The new result clears well above the minimum acceptable standard of 50% required by regulators.

VE of 50% means vaccination cuts the chance of developing symptomatic Covid-19 by half. The background case rate - as measured by the placebo group - is in the low single digits. In the Chinese clinical trial, the reported placebo case rate was around 1.2% in the four-month study period. Cutting that number by 70% means the chance of getting sick with Covid-19 for those vaccinated is just under 0.4%, or 4 in 1000.

Recall that when generalizing the VE from a single clinical trial to the general population, we should look at the lower end of the 95% confidence interval for VE, in which case, we should say the VE of Sinopharm when applied to the general population is expected to be 60-65%. This proviso applies to all vaccine trials, not just the Chinese one.

The design of this Phase 3 trial is essentially identical to those of the U.S. trials that I have discussed before. If anything, the JAMA report and appendix provide more details than I have seen in the other scientific reports.

Similarities

Let's first describe the commonalities. The Sinopharm vaccine also requires two doses, taken 21 days apart. The control group is given two doses of a placebo. The trial ran from July to Nov 2020 in UAE and Bahrain.

The trial only counts adjudicated, lab-confirmed, symptomatic cases of Covid-19. The trial investigator - that is, the vaccine developer - only investigates a case if the participant experiences qualifying symptoms from a pre-published list. The infection must be confirmed by a PCR test. Finally, a committee of "experts" determined if the PCR-positive, symptomatic case should be counted or not. In broad terms, this procedure is how all the other pharmas have run their vaccine trials.

The headline VE number of 70-80% counts only cases that are detected from 14 days after the second dose. This also is in line with current practice. I am not a fan of this maneuvre. You are forced to interpret the efficacy as a "conditional" metric. Infection rate is cut by 70% - but not for everyone, only if one does not fall ill between the first dose and 14 days after the 2nd dose.

This maneuvre is a spiritual cousin of how the Bureau of Labor Statistics computes unemployment rate. When the unemployment rate is 4 percent, one cannot infer that 96 percent of working-age people have jobs. It actually means 96 percent of the "labor force" have jobs. The labor force excludes people for many different reasons, such as anyone who has given up on the job market. So a proper way to interpret a 4 percent unemployment rate is as a conditional - 96% of people are employed only if they are classified as part of the work force. Conditional metrics can be highly misleading as it shoves a chunk of the population under the rug. (See this post on unemployment rate - or read Chapter 6 of Numbersense (link) for a deeper study).  

Similarly, a college might report that the average salary of its graduates is $80,000. That usually means it's an average of $80,000 only for those who have found jobs (and self-reported their salaries to the careers office). If you're a parent, and you want to know if your kid might earn $80,000 after graduating from this college, it depends on whether the kid finds a job. The $80,000 number provides zero information about the chance of graduates landing jobs. If the kid fails to find a job, the relevant salary number is 0.

Sinopharm also computed a variety of VE metrics, just like everyone else. If they had counted cases from first dose, the VE drops to 50-66%. If they had included asymptomatic cases (but counting only from 2D+14), the VE drops to 64-74%. None of this is surprising as we have seen this movie before in all the other trials.

The Chinese vaccine also claims 100% efficacy against severe Covid-19. As with the other trials, I find this claim absurd. None of these trials have enough people in them to measure the rare event of severe Covid-19. This Sinopharm claim is based on 2 cases in placebo versus 0 cases in the vaccine arms. Also available to the Chinese team is the other dubious statement that "no one who's been vaccinated died in the trial". I'm sure you've heard that one too about the U.S. trials. (No one died in the trial.)

Also as with other trials, subgroup analyses were computed robotically but nothing of value could have come out of them as sample sizes were too small.

In a nutshell, using the rules of the game that have been set by the regulatory agencies, the Sinopharm vaccine demonstrated similar "spectacular" levels of efficacy - even though it is based on old technology. If, as some people have argued, J&J's 67% efficacy should be considered equal to Pfizer and Moderna's, then this vaccine should be considered in the same class. (Same with Astrazeneca).

Differences

There is no obvious deviation from the protocols of the U.S. trials.

One minor difference is the Chinese trial tested two vaccines (developed from different strains of the virus) against placebo. When I stated the results in ranges, that's because the two vaccines showed slightly different numbers. Thus, there were three arms to the trial, each with equal number of participants (roughly 14,000 per arm).  

In one respect, the trial result is stronger in my view - the follow-up period is longer. The follow-up time for the median participant in the Chinese trial was described as 77 days - wait a minute, they defined that as 77 days from the start of the case-counting window, thus this is 91 days (3 months) from the second dose. Recall that the FDA set a rule that they will not approve any vaccines for which the data do not have a median follow-up period of two months - and that clock starts ticking from second dose. Eventually, the FDA approved two vaccines (Pfizer, Moderna) when the published data did not even meet the two-month rule; and for Johnson & Johnson, they condoned a flexible interpretation of two months as 8 weeks (56 days).  

I was very excited to find eFigure 2 in Supplement 3 of the JAMA report (which erroneously pointed readers to Supplement 2). This set of charts lays out exactly why the VE number computed as an interim analysis must be interpreted carefully.

Look at the enrollment timeline in Bahrain, from Aug to Oct 2020. In particular, consider someone who enrolled after late September, which was easily over a quarter of the participants. They took the second shot in the second half of October but the study does not count cases until 14 days later (Nov). The cutoff date of the analysis is end of December so this set of participants have at most 1.5 months of follow-up beyond the 2D+14 time point.

Now look at the cumulative case curve. This subset (marked by the brown box) can only influence the curve between 0 and 45 days. Meanwhile, those who enrolled at the start of the trial (blue box) took their second shot by mid September so their follow-up period lasted from late September to late December (up to 3 months).

It would have been fine if the timing of enrollment was randomized - that every weekly cohort of participants have the same characteristics. But this was not true of any of the trials. Pfizer added more minorities (higher risk) later in the trial. Astrazeneca started with younger and lower risk participants. The Chinese trial enrolled older people (60 years plus) later. None of these reports discussed possible biases related to staged enrollment, and none of them corrected for such biases.

You might be wondering how the scientists found data to draw out the case curve beyond 60 days, given that only a fraction of the Bahrain participants have reached that amount of follow-up. The answer is data from the UAE sites. In other words, the right side of the case curve contains predominantly data from UAE while the left side contains data from both UAE and Bahrain.

That is no small matter either. In the same eFigure 2, the researchers reveal that the case rates in UAE and Bahrain looked very different.

UAE has twice the number of cases (but 5 times the population) of Bahrain. Also cases were increasing during the study period in UAE while they were decreasing in Bahrain.

When I reviewed the previous vaccine trials, I was only able to speculate about these potential biases because those reports omitted this type of data. With this JAMA report, it's clear the biases are present, and I urge the other pharmas to publish similar data. Even Sinopharm can publish more. It's not enough to know the timing of enrollments; it's important to know the potential covariate imbalance between weekly cohorts.

***

I have split my notes into two parts. Part 2 of my review of the Sinopharm report will be posted this week.

Instant Classic for Science Communications Students

When I blogged about Pfizer's adolescent trial (here), I was wondering when and if they will release the data or the scientific report to support those press releases (one by Pfizer, and one by the FDA). A report has just been published in the prestigious New England Journal of Medicine.

I predict this will become a staple of future science communications textbooks.

The 12-15 year old age group presents extreme difficulties for conducting clinical trial research - the case rate is expected to be extremely low, and there is a strong desire to fast track approvals. Those two issues are incompatible. You need a huge sample size to detect low rates of events, let alone differences between low rates of events. From the start, the FDA has allowed a head fake: the primary endpoint of the adolescent trials will be immunogenecity results, i.e. test of antibody levels in the vaccinated. This is no ordinary trial, as we will not be comparing adolescents to adolescents, vaccinated against placebo. We will be comparing adolescents to 16-25 year olds, vaccinated against vaccinated.

So what does the highly respected NEJM say about the 16-25 year olds, the designated "control" group? I searched for every mention of 16-25 year olds in the report. Below are some representative examples of the matching sentences (Scroll to the very end of the post to see the entire list of mentions).

• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.

You can read every word of the above sentences, or every word of the entire report, including every word in the footnotes, and you walk away thinking this adolescent trial included two age groups (12-15 and 16-25). What if this weren't true? What if the number of 16-25 year olds in this adolescent trial is exactly zero?

As I was reading the prestigious NEJM report, I was feeling a sense of dread, which worsened as page after page repudiated what I thought was true: that the 16-25 year olds data came from the previous adult trial. Indeed, one of the key issues I flagged in my previous blog post is that the analysis plan involved comparing one age group in one trial to another age group in a different trial. Did I mislead my readers? Was I so blinded by my own biases that I made up this detail?

It surely was not possible that scientists and peer reviewers and the editors of the prestigious NEJM would allow such a basic but important detail to be omitted. It has got to be my brain fog... am I losing grip of my sanity?

So I checked my notes... The very first line of the Pfizer press release from March 31 stated unmistakably: "In participants aged 12-15 years old, BNT162b2 demonstrated 100% efficacy and robust antibody responses, exceeding those reported in trial of vaccinated 16-25 year old participants in an earlier analysis, and was well tolerated." (their italics)

Next, I checked the FDA press release on May 10. The source of data for the 16-25 year olds has been wiped out. In the safety section, the FDA stated "The side effects in adolescents were consistent with those reported in clinical trial participants 16 years of age and older. " As if the adolescent trial contained participants 16 years of age and older. In the efficacy section, the FDA stated "The immune response to the vaccine in 190 participants, 12 through 15 years of age, was compared to the immune response of 170 participants, 16 through 25 years of age. " Unless you know otherwise, you're going to assume the trial contained participants of both age groups.

I warned you up top that you're witnessing an instant classic in science communications.

It takes discipline and skill to write an entire report that dances around a key detail. A true masterclass in deception. This is an instance of "story time," lulling readers in with clinical trial data but presenting conclusions from analyses that do not enjoy the benefits of an RCT. But "story time" trivializes what has happened here. I hate to use the F word. I'm not thinking the four-letter F word. The five-letter one. F R A U D.

***

Also of interest is the other side of the equation. How does one develop a nose for detecting fraud in data? I have to admit that it would have been very hard for anyone to notice the deception in this NEJM paper - that's why it belongs on the syllabuses of science communications courses.

There were just two tiny cracks. The first was when they disclosed that all 12-to-15-year-old participants came from the U.S., while 16-to-25-year-old participants were globally recruited. This would have been unusual to do in a single trial enrolling 12-25 years old, stratified into two age groups.

The second crack was in the way the waterfall chart was presented.

Notice the two age groups were presented in two separate panels. What we expect to see if the trial actually enrolled both age groups is something like this:

The waterfall starts with the entire set of enrollees and then there is a split by age groups. This chart came from the Phase 2 Moderna study of two half-doses that I discussed in this post.

As I said, these are subtle signs, easily missed.

***

Is it never allowed to pull a control group from a different place? Doing so does not automatically disqualify the analysis but such an analysis does not enjoy the exalted gold-standard status of a randomized controlled trial. Such workarounds must be scientifically and rigorously justified. Work must be presented to convince readers that any biases coming from different times, different places, different ages, different demographics, different anything have been measured and corrected for.

Here, the authors did not present any evidence about possible biases. They didn't even acknowledge the source of the control group data.

***

Listed below is every single mention of the 16-to-25-year-old control group in the Pfizer adolescent study in the NEJM scientific report. Each mention represents an opportunity for the researchers to inform readers that this control group came from a different trial.

• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was an immunogencity objective.
• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The immunogenicity objective was to show noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants.
• Noninferioirty was assessed among participants who had no evidence of previous SARS-CoV-2 infection with the use of the two-sided 95% confidence interval for the geometric mean ratio of SARS-CoV-2 50% neutralizing titers in 12-to-15-year-old participants as compared with 16-to-25-year-old participants 1 month after dose 2.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was assessed on the basis of the geometric mean ratio of SARS-CoV-2 50% neutralizing titers.
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Severe injection-site pain after any BNT162b2 dose was reported in 1.5% of 12-to-15-year-old participants and in 3.4% of 16-to-25-year-old participants.
• After BNT162b2 injection, severe headache and severe fatigue were reported in a lower percentage of 12-to-15-year-old participants than of 16-to-25-year-old participants.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• The use of antipyretic agents was slightly more frequent among BNT162b2 recipients who were 12 to 15 years of age than among those who were 16 to 25 years of age.
• [Footnote of Table 1]: Results are for the reactogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and a subset of participants in the 16-to-25-year-old cohort.
• Among the 16-to-25-year-old BNT162b2 recipients, 11% reported any adverse event and 6% had vaccine-related adverse events. Among BNT162b2 recipients, severe adverse events were reported in 0.6% of those who were 12 to 15 years of age and in 1.7% of those who were 16 to 25 years of age.
• One BNT162b2 recipient in the 16-to-25-year-old cohort discontinued vaccination because of severe vaccine-related injection site pain and headache, both of which were reported 1 day after dose 1 and resolved within 1day.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.
• Appendicitis was reported in 2 participants: 1 BNT162b2 recipient in the 16-to-25-year-old cohort and 1 placebo recipient in the 12-to-15-year-old cohort.
• The immune response to BNT162b2 in 12-to-15-year-old adolescents was noninferior to that observed in 16-to-25-year-old young adults.
• The geometric mean ratio of the BNT162b2 neutralizing geometric mean titer in 12-to-15-year-old participants to that in 16-to-25-year-old participants 1 month after dose 2 was 1.76.
• Among all participants regardless of serologic evidence of previous SARS-CoV-2 infection, the serum-neutralizing geometric mean titer 1 month after BNT162b2 dose 2 was 1283.0 in the 12-to-15-year-old cohort and 730.8 in the 16-to-25-year-old cohort.
• Substantial increases in the 50% neutralizing titer from baseline were observed, with GMFRs from baseline to 1 month after dose 2 of 118.3 among 12-to-15-year-old participants and 71.2 among 16-to-25-year-old participants.
• [Caption of Figure 2]: The results shown are for the reacogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and the subset of participants in the 16-to-25-year-old cohort who had electronic diary data available.
• [Footnote of Table 2]: The geometric mean ratio and two-sidded 95% confidence intervals were calculated by exponentiating the mean difference of the logarithms of the titers (the 12-to-15-year-old cohort minus the 16-to-25-year-old cohort) and the corresponding confidence intervals (based on the Student's t distribution).
• Antipyretic use after both dose was slightly higher in the 12-to-15-year-old cohort than in the 16-to-25-year-old cohort.
• All 12-to-15-year-old participants in this trial were enrolled at U.S. sites, whereas the 16-to-25-year-old participants were recruited globally.

If X causes Y, does it follow that X and Y are correlated?

Andrew points out an error in a new book authored by Dan Kahneman, Cass Sunstein, et. al. The authors said that while correlation does not imply causation, "wherever there is causality, there is correlation." In other words, if X causes Y, then X and Y must be correlated.

I like to say statistics is common sense systematized, but here is an example that shows statistical analysis requires deep thinking.

That statement is true if there is nothing else in the system but X and Y. In any real-world system, there are more than two relevant factors.

Since the summer is heating up much of the U.S., let's talk about room temperature. One should expect room temperature (Y) to be positively correlated with outside temperature (X): as outside heats up, inside temperature rises.

Not so if we introduce an air conditioner (C) to this system. The air conditioner regulates the room temperature (Y) to some pre-set number, say, 70 F (21 C). In other words, the effect of C is to keep Y around 70F no matter what the outside temperature (X) is.

The air conditioner destroys the correlation between X and Y. If we only collect data on outside and inside temperatures, we shall see no correlation! Despite outside temperature clearly affecting room temperature in a linear way, the data will be uncorrelated.

To learn the true relationship between X and Y, we must measure the lurking variable (C). If we have all three data series, then a regression will assign the proper effects of X and C on Y.

***

If you are comfortable with algebra, read Cosma's post here for a formal discussion.

If you're looking for intuition, keeping reading.

If we receive a data set with X and Y (outside and inside temperature), we can plot a scatter plot or do a regression, and we'll discover no correlation between X and Y. Using the erroneous thinking in the new book, we'd conclude that X and Y could not have a cause--effect relationship since they are not correlated.

And you've just learned that is a wrong conclusion. If we also have data on air conditioning (C), and we throw that into the regression, we will discover the correct relationships, that inside temperature is positively correlated with outside temperature, and negatively correlated with air conditioning.

In practice, whether the analyst draws the right conclusion comes down to whether the analyst knows about C, and whether data on C can be obtained. What separates a good analyst from an average one is the "numbersense" of anticipating what missing data might be useful to shed light on the data you already have.

Also, in any practical system, X and Y are not zero-correlated! Rather, the correlation of X and Y is artificially attenuated because of the lurking variable C. The alarm bell is very likely to ring loudly if the analyst sees zero correlation. If the expected correlation exists, though, that's when the analyst may neglect to consider lurking variables.

 

P.S. [5-27-2021. A number of correspondents have brought up the issue of linearity so I'm adding some comments here. The strict statistical definition of "correlation" is linear. This may be different from our folk understanding of the word. The left chart below shows a positive linear correlation between X and Y:

When the correlation is linear, we can fit a straight line through the scatter plot of X and Y. The chart on the right shows a specific example of non-linear correlation. For example X may be the amount of sun and Y is the plant growth.

If the standard correlation formula is applied (such as the one included in Excel), you get zero correlation between X and Y on the right chart. Roughly speaking, the two sides of the chart cancel each other out.

The lesson is that no correlation means no linear relationship but not no relationship. The same advice is frequently dished out for regression analysis. If fit is low for linear regression, it doesn't follow that there is no relationship between X and Y.

So one can say that if X is a cause of Y, and the effect of X on Y is shaped like the inverted U, then X and Y will show zero correlation (as defined by the linear correlation formula). Nonetheless, this particular issue is present whether or not we impose a causal interpretation on the X-Y relationship.

What I said in the last paragraph of the main post still applies. In real life, we rarely see a perfect inverted U, or perfect other shapes that would result in zero correlation.

Menu engineering hijinks

Andrew has a great post about menu engineering. 

Designing restaurant menus has become a science, and the referenced BBC article offers up a parade of the peer-reviewed findings from that field. 

Some headline grabbers include weight of menu is associated with upscale restaurants (kind of like weight of business cards), scripted writing is also deemed higher class, longer words are better, first items are ordered more, and words that "mimic mouth movements" related to the food are more popular.

This type of research has two ingredients that raise their viral potential: the researchers use "conventional wisdom" as foil, either legitimizing them through "science" ("I was right!") or casting doubt ("Everyone is wrong!"); the findings make for light, fun headlines.

Nevertheless, Andrew's post concerns the quality of the underlying research. 

***

Since I've been waxing about research methods of Covid-19 studies, let's also talk about the type of methodology typically deployed for food science studies. 

The researchers recruit a bunch of people, usually a few hundred at the most, sometimes college students, and they put them in a "lab experiment" in which they are exposed to (say) menus that are heavier or lighter in weight. Because it's an lab experiment, the participants are randomly assigned to these treatment groups, and then they fill out a survey which measures outcomes. 

This research method is very common in social science academia, but in recent years, people are discovering that they often fail to replicate. (Lots of posts about this scientific crisis on Andrew's blog.)

The lab experiment method has a number of problems. First, the lab scenario is completely contrived. The participants are not actually hungry, ordering food, or consuming them. They are not spending real money.

Second, the participants are not a representative sample of the population to which the research will be generalized. The recruits are not typical diners (and certainly, not customers of upscale restaurants).

Third, saying in a survey that they will pay more for something doesn't mean they will actually pay more real money for it.  

Fourth, such research typically assumes no meaningful variation in human behavior. The sample size is too small to account for such variability. The investigators are looking for a universal behavior - and they include just enough samples to achieve a magical p-value threshold required for scientific publication.

Fifth, almost all such research isolate a single factor. It is a well-known fallacy in design of experiments to add the effects of separate one-factor studies - remember, each experiment assumes all else equal.

***

I left the following comment:

Now, wake me up when they do an actual statistical experiment. They could partner with a restaurant chain, design an experiment involving all of the above factors, print different versions of the menu, and randomly assign them to real diners who make real orders. As a foodie, I’d love to be part of this research team!

 

 

Science by press release should be discouraged

I've been holding off on making comments on a few Covid-19 topics, hoping that some data or scientific reports would be released. Nothing seems forthcoming, though. The FDA/CDC has been busy making many pronouncements but unfortunately, solid scientific evidence is hard to find. Please add links to the comments if you are aware of any data or research reports that address the topics below.

 

Pfizer and Moderna both claimed their vaccines are definitely effective for at least six months.  

The entire basis for this claim appeared to be press releases from the two pharmas (Pfizer, Moderna). The FDA/CDC then parroted what was written. These pronouncements shocked me because I was eager to learn what analytical technique they might use to cure the destruction of the placebo group.

Yes, you heard that right. At this point, the vaccine trials are handicapped. During the  EUA hearings in December/January, both pharmas have argued that it is unethical to withhold a "miraculous" vaccine from people, and therefore, everyone in the placebo group will be offered the vaccine. This deliberate dismantling of the experimental design should have robbed us of any opportunity to assess longer-term efficacy or safety from the clinical trials. (See my prior post that raised this issue on the day Pfizer got approved.)

Allowing the placebo group to get vaccinated presents two problems: the remaining placebo group shrinks in size, making error bars wider; and the self-selected exit from the placebo group destroys the covariate balance that is the hallmark of a randomized controlled trial. Any attempt to salvage such an experiment will be messy.

The Pfizer press release ignores this issue completely. The Moderna press release disclosed this:

"As of April 13, all placebo participants have been offered the Moderna COVID-19 Vaccine and 98% of those have received the vaccine."

This tells us a lot and very little, depending on what question you're asking. The way this sentence is tagged on after presenting the six-month efficacy finding suggests that the analysis was completed before the placebo group was dismantled. However, looking at the timeline, I find that hard to believe. More likely, at least a generous section of the placebo group has been vaccinated by the time they did this analysis.

The 98% number also surprised me, and reveals an interesting tidbit for the very first time. The participants of these clinical trials are vaccine enthusiasts who immediately jumped at the chance of getting vaccinated at their first chance. The clinical trials were ostensibly single-blind; nevertheless, a participant could roughly guess which arm they belonged to based on the degree of side effects. This potential bias is a wonky issue for serious researchers to chew on.

Moderna claimed to have followed the median participant for 6 months while in the Pfizer analysis, only about one quarter of the participants have reached 6 months. These details are important because the subset of participants driving the results in the preliminary analysis is the same group that drives the results of this follow-up analysis. In other words, the subgroup that did not reach 2 months at the two-month analysis did not reach 6 months at the six-month analysis.

Pfizer is approved for adolescents.

I also failed to find any data or scientific reports to support the press releases. There is a Pfizer press release and an FDA press release. If you've been reading this blog during the pandemic, you know that the merits of statistical findings cannot be evaluated based on a press release. It matters a lot how they count cases, what case-counting window they use, what is excluded, etc. etc.

The first issue with the adolescent trial is its small sample size (only about 1200 per arm, 1/16 of the size of the adult trial). In the adult trial, when half the trial population reached two months after the second dose (so roughly 3 months in), the total number of cases across both arms was 325, or 0.7%. If the case rates were similar, we should expect only 18 cases across both arms in the teenagers 3 months in. The real number should even be lower because the case rate for this age group should be smaller than for adults.

It appears that the FDA is primarily relying on an "immunogenicity" outcome, which is a measure of the amount of antibodies found after vaccination. This evaluation breaks the experimental design; it's really no longer a clinical trial. That's because the level of antibodies in the placebo group is expected to be zero. What's more, no one knows what is the minimum level of antibodies sufficient for protection against infection, developing symptoms, severe disease or death!

That's why the level of antibodies of these teenagers is compared - not to the placebo group in the teenager trial - but to 16-25 year olds in the adult trial! Alarm bells are ringing: different ages, different sizes of age range, different time period, different trial, not randomized.

And there's more. They did not measure antibodies in all 1200 vaccinated teenagers. They only had data on 190 people. (This bit of information is omitted in the Pfizer press release but found in the FDA press release. That's exactly why we shouldn't do science by press releases.)

[5/20/21: Added the next paragraph]

In short, the immunogenicity result does not compare teenagers who were randomly assigned to the vaccine group with those in the placebo group. It says that the vaccinated teenagers have at least similar levels of antibodies as the 16-25 age group in the adult trial from last year. Also, only a small subset of trial participants have measured immunogencity outcomes.

Both press releases also boast a 100% efficacy number, without providing the error bar. The two press releases give different numerators and denominators. Let me take the FDA number, which claims a case rate of 16/978 = 1.6% on the placebo arm of the teenager trial. This rate appears way too high. It is double the number I gave before which came from Pfizer's adult trial. Further, the 16 cases count only those happening beyond 7 days after the second dose, while the 325 cases I used above count all cases. Unless there were no cases between first dose and 2D+7 (itself a shocking development), the case rate comparable to the 0.7% would be well above 1.6%. Are they defining cases in the same way as in the adult trial? I don't know since I can't find a trial protocol, a scientific report or any data for the adolescent trial.

***

The theme of my recent talk is that not all research studies are created equal. Even though the adolescent finding and the six-month finding are both analyses of clinical trial data, neither analysis preserves the structure of the experiment, and thus should be classified as observational studies.

The FDA/CDC should commit to releasing the data and scientific reports supporting these decisions. Science by press release should be discouraged.

 

Curve watching during the pandemic

The pandemic has revealed the importance of a most exciting area of statistics - causal inference from observational data. We've been inundated with all kinds of reports about the effect of [vaccines, masking, social distancing, school closures, outdoor dinings...] on the pandemic. Most of these "studies" focus on either one factor or one region or one demographic segment. Then, the headlines and PR coming out of these studies apply the specific results to the general public. Vaccines work! Masks don't work! Closing schools doesn't make sense!

The quality of the causal inference is tested when we apply the findings of these studies to other settings. So in today's post, I explore the claim that vaccination campaigns are responsible for the decline in cases seen in many countries.

In order to shut down our biases, I am presenting the graphs first without country labels. In each chart, the time period is from November 2020, before any vaccinations, to April 2021. I used data from OurWorldinData.org. The red line shows the new cases per million while the blue line shows the proportion of the population who have been vaccinated (defined as those who received at least one shot if it's a two-shot vaccine).

If we only look at individual countries, here are the stories one can tell:

Panel A presents a clean picture. Cases have tumbled from a peak in mid January to a level below that of November by end of April. The decline begun soon after initiating the vaccination campaign, which has reached about half the population. So vaccines worked!

In Panel B, this country's highly successful vaccination campaign has covered over 60 percent of the population by end of April. That appeared to be a critical mass, at which point cases dropped below the level of November. One can build a story about herd immunity.

The herd immunity story is shaky though. The 60-percent threshold does not apply to Panel A. Using the same logic, one would argue that the country reached critical mass even at 30% vaccinated. But at 30% vaccinated, the country in Panel B was approaching peak cases!

In Panel C, the sharp drop in cases happened as the proportion vaccinated rose from 20 to 40 percent while cases grew during the first two months of the campaign.

In Panel D, cases plunged within weeks of the start of the vaccination campaign, and reached a low plateau with fewer than 10% of the population vaccinated.

Then, you have Panels E, F and G. If one only analyzes those countries, then one comes to the opposite conclusion. In these places, the cases apparently have not been slowed by vaccinations. Over 60% of the people in Panel E have taken a shot but no sharp decline is evident yet. For Panel F, there have been a spurt in cases in April with the vaccination proportion nears 60%.

The picture is very confusing. That's because the vaccine is not the only factor affecting the rise and fall of cases. Any model will fit some of the data some of the time; a good causal model must work across multiple countries.

Now, I reveal the countries. These are the top 11 countries ranked by the proportion vaccinated by end of April. I excluded countries with low overall Covid-19 incidence. Seven of these countries have experienced a nice drop in cases per million.

***

It's also a good idea to take a peek at the other end of the spectrum. What's going on in countries that have barely begun their vaccination campaigns.

Here are 12 countries with very low vaccination rates but with case incidence in a similar range to the 11 countries shown above.

None of these countries have seen spikes in cases, almost all of them are seeing downward trends in April, many of them exhibit up-and-down cycles over these months, and a few have marked plunges such as Georgia in December and Montenegro since March.

***

Simple one-factor models aren't going to work to explain trends across countries and time. A good causal model should include a baseline trendline, a vaccine factor, plus lockdowns and other mitigation measures.

Another challenge for data analysts: the data post-vaccination will be hard to compare to past data. There will be fewer Covid tests among the fully vaccinated. Should cause of death be assigned differently if the patient is known to have been vaccinated? Will we see more deaths from co-morbidities? These are open questions - when I learn more, I'll report back.

 

 

The miracle of randomization in AB testing and RCTs

I'm in the midst of preparing my talk this Friday at a University of Denver conference (more info here). The talk will focus on the range of research methods that have been utilized in pandemic-related scientific research, why research methods matter, and how they are related to research quality.

I have some materials that probably will get edited out of the final presentation but I don't want to just throw them away. So I am posting some of them here. Today, I like to use a practical example to illustrate the miracle of randomization, which is an important feature of controlled scientific experiments.

It's hard to imagine running an A/B test or a clinical trial without randomization but this idea is not that old - the British statistician RA Fisher is often credited as having invented it in the 1920s. Randomization means the participants are randomly assigned (via a coin flip) to A or B groups (or test or control groups).

The consequence of randomizing treatment is profound. It leads to "covariate balance". If 50% of the research population is female, we expect 50% of the test group as well as 50% of the control group to be female. Depending on the total sample size, the margins of error around those percentages vary.

The one act of randomization confers balance between the test and control groups on all covariates of interest - not just gender but also race, income groups, comorbidities, employment status, anything that is relevant to your study.

What's more, you get balance on other covariates for free! Any covariate, including things you've never dreamed about.

It is this last point that has enormous practical implications.

Let me explain using a real-life A/B test. Let say Amazon wants to test sending customers an email on Christmas eve, and measure whether this email increases spending in the next week. The data scientist draws up the list of targeted customers, then randomly select half of this list to receive the email, and the other half to be excluded from the mailing.

If everything goes as planned, at the end of the week, the data scientist can pull the data, tabulate the spending on both groups of customers, and any difference in spending can be attributed to the email. Of course, the analyst knows that household income, gender, religion, ethnicity, education, prior spending, etc are all factors that affect holiday spending. Since treatment was randomized, all of those factors are well balanced between test and control groups, and therefore cannot explain any statistically significant change in spending.

So far so good.

In a meeting with the IT team, it was disclosed that a systems error caused all Yahoo.com emails to be dropped from mailing. Let's assume that people still using Yahoo.com emails are bigger spenders on Amazon (reasonable since these people are probably older). Has this error ruined our test?

The short answer is no. That's because randomization not only balances variables we know about, but also balances unknown variables. In this case, if the analyst pulls out the proportion of Yahoo.com email addresses from the test and control groups, it's highly likely that the difference is immaterial. Therefore, we can still interpret any difference in spending as caused by email.

It gets even better!

You might think the above scenario is concocted and unrealistic (though I can assure you it's not.) What makes it a tad unrealistic is the eventual discovery of the error. Most errors are committed and never revealed.

For example, people with Yahoo.com emails may not be dropped by our IT team; they may simply be victims of spam filters. Most email providers out there use one of several spam filters, which are black-box algorithms that change constantly. For any number of reasons, it's possible that Yahoo's spam filter algorithm dumped Amazon emails into the spam folder during the week of our email test, and so people with Yahoo.com emails mostly didn't see the emails.

This situation cannot be predicted by the data scientist, nor will it likely to discovered after the fact. However, our test analysis can proceed as usual because again, the test group will have about the same proportion of Yahoo.com emails as the control group. To the extent that a difference in spending is observed, it isn't due to the spam filter.

We can call this the unintended consequences of randomization. This is really really cool stuff.

***

Not long ago, I wrote about real-world studies of vaccine effectiveness that mimic randomized controlled trials by synthesizing test and control groups. The matching process involves explicitly balancing the two groups on a list of matching variables (age, gender, etc.).

This process does not have the magic of a randomized experiment. In practice, balance is enforced on those matching variables but there is no reason to believe the two groups are also balanced on other known covariates not used for matching. Further, it is even less likely that balance is achieved on unknown covariates.

This is one reason why experiments are considered the gold standard for causal studies.

 

 

The statistics behind the J&J blood-clot decision

Last Friday, the U.S. ended the pause of the Johnson & Johnson vaccine, which may be linked to rare blood clots in women under 50. As with the EMA's stance on the Astrazeneca vaccine, they recommend continuing to administer these vaccines based on a cost-benefit analysis.

In today's post, I summarize the key information in the documents released during Friday's meeting of experts about those blood clots. The media have failed to cover the statistical reasoning behind the decision.

In its presentation, J&J highlighted the number that is most beneficial to its commercial interest - the risk of rare blood clots in the general population regardless of gender and age. The media is also fixated on this statistic, which is the least useful of all the numbers in these documents. Out of 8 million people who have been given the J&J shot since March (remember that this vaccine only got approved at the end of February), they have reported 15 cases of rare blood clots coupled with low platelet counts. This means the risk is 2 cases per million.

Measuring the risk of this side effect is a great example of why no statistic is purely objective. The subjectivity lurks behind how the risk is defined. Risk is a ratio of two numbers: the number of cases, and the relevant population at risk. The 2 cases per million number bakes in two choices: on the denominator, J&J selected the entire population regardless of age and gender; on the numerator, J&J selected the co-occurrence of rare blood clots and low platelet counts. Those are not the choices I expected. 

The denominator should be women only, or younger women only, given that all 15 cases occurred in women under 60 years old. The choice of the denominator is not immaterial! The CDC ultimately used women aged 18-49 as the relevant population, and this change of denominator bumped the risk from 2 up to 7 per million. (Including men roughly doubles the denominator while adding zero cases to the numerator.)

The cutoff age of 50 is also a choice. Two cases were found in women between 50 and 60 years old. By setting the at-risk population as women under 50, those two cases are removed from the numerator. Each additional case increases the risk by roughly 0.5 times if the denominator is not changed appreciably.

The CDC also looked at smaller subgroups. Women in their thirties accounted for the most cases, and the risk in that age group was 12 per million (7 cases among 600K vaccinations). Meanwhile, 1.4 million women aged 50-64 took the J&J vaccine, with two reported cases of blood clots and low platelets, leading to a risk of < 2 per million. Scientists can choose the most restrictive analysis by focusing only on women in their thirties, or the most expansive analysis, on all women. The CDC's choice of 50 is reasonable while not the only option.

The numerator is also unusual, and different from what the EMA used when examining the Astrazeneca vaccine. J&J and the CDC decided to define the side effect as rare blood clots coupled with low platelet count. This is a more restrictive case definition, which reduces the number of cases under study. Interestingly, this definition neatly draws a line between the adenovirus platform and the mRNA platform (Pfizer & Moderna), as so far, the mRNA vaccines have seen some cases of blood clots but zero cases of blood clots plus low platelets.

***

The next issue is how bad is that risk. For this, we have to compare to other risks. 

One obvious comparison is the baseline rate of blood clots and low platelet count. This is where the choice of numerator presents a difficulty. The baseline risk is essentially zero: that combination of conditions almost never happens. One of the CDC documents compared the observed risk to the risk of blood clots in women 20-50 years, which is invalid. The risk of blood clots plus low platelets should be quite a bit lower than the risk of blood clots (with or without low platelets). In a different CDC document, the risk of blood clots and low platelets is estimated to be under 1 case per million. In the J&J document, they claim the baseline risk is 0.1 per million. Neither of these numbers can be directly compared to the 7 per million observed risk because the baseline risk is per million Americans, not per million American females aged 18-50.

Nevertheless, it is abundantly clear that the risk of getting blood clots and low platelet count among women under 50 who took the J&J shot is much higher than the baseline risk. The risk of dying from this side effect is also much higher than the baseline mortality risk. (About 25-30% of the known cases have already died.) On an absolute scale, the condition is very rare. On a relative scale, it is concerning.

All 15 cases were reported between 6 and 15 days after the vaccination. Notably, this is the period during which scientists insist the vaccine has zero benefits. 

***

Another relevant comparison is the risk of death from Covid-19. This is a key to the cost-benefit analysis. How should we compute this risk? We should focus on the period from March and April, the two months in which the J&J vaccine was administered. 

This CDC slide indicates that among females aged 18-50, the Covid-19 death rate was around 0.3 per 100,000, or 3 per million. This means the chance of dying from Covid-19 is about the same as from dying from blood clots+low platelets after taking the J&J vaccine.

Yes, that's right. It's also confirmed in the CDC simulation model. 

So, what does the CDC mean when it says the benefits outweigh the costs? The simulation model finds that slowing down the vaccinations causes a lot of excess Covid-19 deaths among men, and among females over 50, and therefore, across all age groups and genders, stopping J&J causes more harm than good.

***
Two other small details caught my attention. There is another fatal case of blood clots and low platelets that was excluded - seemingly because the situation is too complex. There are also other cases being investigated including cases affecting men. 

When the investigation first started, there were 6 reported cases. About 10 days later, the number of cases more than doubled to 15. The vaccine didn't become more risky suddenly. This phenomenon is caused by self-reporting of cases. Such self-reporting has always been a weakness of how we monitor side effects. The risk may be over-estimated because of under-reporting of mild cases.