Instant Classic for Science Communications Students

When I blogged about Pfizer's adolescent trial (here), I was wondering when and if they will release the data or the scientific report to support those press releases (one by Pfizer, and one by the FDA). A report has just been published in the prestigious New England Journal of Medicine.

I predict this will become a staple of future science communications textbooks.

The 12-15 year old age group presents extreme difficulties for conducting clinical trial research - the case rate is expected to be extremely low, and there is a strong desire to fast track approvals. Those two issues are incompatible. You need a huge sample size to detect low rates of events, let alone differences between low rates of events. From the start, the FDA has allowed a head fake: the primary endpoint of the adolescent trials will be immunogenecity results, i.e. test of antibody levels in the vaccinated. This is no ordinary trial, as we will not be comparing adolescents to adolescents, vaccinated against placebo. We will be comparing adolescents to 16-25 year olds, vaccinated against vaccinated.

So what does the highly respected NEJM say about the 16-25 year olds, the designated "control" group? I searched for every mention of 16-25 year olds in the report. Below are some representative examples of the matching sentences (Scroll to the very end of the post to see the entire list of mentions).

• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.

You can read every word of the above sentences, or every word of the entire report, including every word in the footnotes, and you walk away thinking this adolescent trial included two age groups (12-15 and 16-25). What if this weren't true? What if the number of 16-25 year olds in this adolescent trial is exactly zero?

As I was reading the prestigious NEJM report, I was feeling a sense of dread, which worsened as page after page repudiated what I thought was true: that the 16-25 year olds data came from the previous adult trial. Indeed, one of the key issues I flagged in my previous blog post is that the analysis plan involved comparing one age group in one trial to another age group in a different trial. Did I mislead my readers? Was I so blinded by my own biases that I made up this detail?

It surely was not possible that scientists and peer reviewers and the editors of the prestigious NEJM would allow such a basic but important detail to be omitted. It has got to be my brain fog... am I losing grip of my sanity?

So I checked my notes... The very first line of the Pfizer press release from March 31 stated unmistakably: "In participants aged 12-15 years old, BNT162b2 demonstrated 100% efficacy and robust antibody responses, exceeding those reported in trial of vaccinated 16-25 year old participants in an earlier analysis, and was well tolerated." (their italics)

Next, I checked the FDA press release on May 10. The source of data for the 16-25 year olds has been wiped out. In the safety section, the FDA stated "The side effects in adolescents were consistent with those reported in clinical trial participants 16 years of age and older. " As if the adolescent trial contained participants 16 years of age and older. In the efficacy section, the FDA stated "The immune response to the vaccine in 190 participants, 12 through 15 years of age, was compared to the immune response of 170 participants, 16 through 25 years of age. " Unless you know otherwise, you're going to assume the trial contained participants of both age groups.

I warned you up top that you're witnessing an instant classic in science communications.

It takes discipline and skill to write an entire report that dances around a key detail. A true masterclass in deception. This is an instance of "story time," lulling readers in with clinical trial data but presenting conclusions from analyses that do not enjoy the benefits of an RCT. But "story time" trivializes what has happened here. I hate to use the F word. I'm not thinking the four-letter F word. The five-letter one. F R A U D.

***

Also of interest is the other side of the equation. How does one develop a nose for detecting fraud in data? I have to admit that it would have been very hard for anyone to notice the deception in this NEJM paper - that's why it belongs on the syllabuses of science communications courses.

There were just two tiny cracks. The first was when they disclosed that all 12-to-15-year-old participants came from the U.S., while 16-to-25-year-old participants were globally recruited. This would have been unusual to do in a single trial enrolling 12-25 years old, stratified into two age groups.

The second crack was in the way the waterfall chart was presented.

Notice the two age groups were presented in two separate panels. What we expect to see if the trial actually enrolled both age groups is something like this:

The waterfall starts with the entire set of enrollees and then there is a split by age groups. This chart came from the Phase 2 Moderna study of two half-doses that I discussed in this post.

As I said, these are subtle signs, easily missed.

***

Is it never allowed to pull a control group from a different place? Doing so does not automatically disqualify the analysis but such an analysis does not enjoy the exalted gold-standard status of a randomized controlled trial. Such workarounds must be scientifically and rigorously justified. Work must be presented to convince readers that any biases coming from different times, different places, different ages, different demographics, different anything have been measured and corrected for.

Here, the authors did not present any evidence about possible biases. They didn't even acknowledge the source of the control group data.

***

Listed below is every single mention of the 16-to-25-year-old control group in the Pfizer adolescent study in the NEJM scientific report. Each mention represents an opportunity for the researchers to inform readers that this control group came from a different trial.

• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was an immunogencity objective.
• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The immunogenicity objective was to show noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants.
• Noninferioirty was assessed among participants who had no evidence of previous SARS-CoV-2 infection with the use of the two-sided 95% confidence interval for the geometric mean ratio of SARS-CoV-2 50% neutralizing titers in 12-to-15-year-old participants as compared with 16-to-25-year-old participants 1 month after dose 2.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was assessed on the basis of the geometric mean ratio of SARS-CoV-2 50% neutralizing titers.
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Severe injection-site pain after any BNT162b2 dose was reported in 1.5% of 12-to-15-year-old participants and in 3.4% of 16-to-25-year-old participants.
• After BNT162b2 injection, severe headache and severe fatigue were reported in a lower percentage of 12-to-15-year-old participants than of 16-to-25-year-old participants.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• The use of antipyretic agents was slightly more frequent among BNT162b2 recipients who were 12 to 15 years of age than among those who were 16 to 25 years of age.
• [Footnote of Table 1]: Results are for the reactogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and a subset of participants in the 16-to-25-year-old cohort.
• Among the 16-to-25-year-old BNT162b2 recipients, 11% reported any adverse event and 6% had vaccine-related adverse events. Among BNT162b2 recipients, severe adverse events were reported in 0.6% of those who were 12 to 15 years of age and in 1.7% of those who were 16 to 25 years of age.
• One BNT162b2 recipient in the 16-to-25-year-old cohort discontinued vaccination because of severe vaccine-related injection site pain and headache, both of which were reported 1 day after dose 1 and resolved within 1day.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.
• Appendicitis was reported in 2 participants: 1 BNT162b2 recipient in the 16-to-25-year-old cohort and 1 placebo recipient in the 12-to-15-year-old cohort.
• The immune response to BNT162b2 in 12-to-15-year-old adolescents was noninferior to that observed in 16-to-25-year-old young adults.
• The geometric mean ratio of the BNT162b2 neutralizing geometric mean titer in 12-to-15-year-old participants to that in 16-to-25-year-old participants 1 month after dose 2 was 1.76.
• Among all participants regardless of serologic evidence of previous SARS-CoV-2 infection, the serum-neutralizing geometric mean titer 1 month after BNT162b2 dose 2 was 1283.0 in the 12-to-15-year-old cohort and 730.8 in the 16-to-25-year-old cohort.
• Substantial increases in the 50% neutralizing titer from baseline were observed, with GMFRs from baseline to 1 month after dose 2 of 118.3 among 12-to-15-year-old participants and 71.2 among 16-to-25-year-old participants.
• [Caption of Figure 2]: The results shown are for the reacogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and the subset of participants in the 16-to-25-year-old cohort who had electronic diary data available.
• [Footnote of Table 2]: The geometric mean ratio and two-sidded 95% confidence intervals were calculated by exponentiating the mean difference of the logarithms of the titers (the 12-to-15-year-old cohort minus the 16-to-25-year-old cohort) and the corresponding confidence intervals (based on the Student's t distribution).
• Antipyretic use after both dose was slightly higher in the 12-to-15-year-old cohort than in the 16-to-25-year-old cohort.
• All 12-to-15-year-old participants in this trial were enrolled at U.S. sites, whereas the 16-to-25-year-old participants were recruited globally.

Comments

[Cris]

“Is it never allowed to pull a control group from a different place? Doing so does not automatically disqualify the analysis but such an analysis does not enjoy the exalted gold-standard status of a randomized controlled trial.”

They are comparing two age groups. There is no way that this study would ever “enjoy the exalted gold-standard status of a randomized controlled trial.” No matter where the control data came from.

If they compared antibody levels before and after the vaccination, then the timing of the vaccination is irrelevant (we’re not looking at how many people got infected, which is dependent on outside factors). So the 16-25 group having been vaccinated earlier is a minor detail. I agree it should have been mentioned, and I wonder why they didn’t. But it’s not this big thing you make it out to be IMO.

[Kaiser]

Cris: Thanks for the comment. We agree that it is odd that they didn't simply acknowledge it. I flag this as a science communications issue.

[A Palaz]

Hi Kaiser,

So just some comments also on communication in this study.

The criteria for covid in study and the testing method and method diagnosis, I think may have some variation from protocol. Maybe also not, but it is not clearor separated.

The 12-15 yr old is likely supervised by guardian. So is some in 16-25 but its not clear.

The testing environment with screen tests at maybe some schools and maybe some college and university is I think not mentioned.

A possible in overlapping between reactoginicity set and efficacy set is not clear.

PS in note on your post on immunity a good interesting general site is

https://www.frontiersin.org/journals/immunology#

If I find the specific papers will link