Big Data, Plainly Spoken (aka Numbers Rule Your World)

NYT in story-first mode

NYT published a brain-dead article filled with statistical fallacies (link).

Since the Times has people on staff who know statistics, the real explanation for such an article is "story-first thinking". Data mined to support a story line.

***

The headline of the article provokes its least convincing statistic - that only seven black students were offered spots at Stuyvesant, one of the nation's top public high schools famous for STEM, out of 762 offers.

The position of the author is that the above statistic reveals systematic "racial and ethnic disparities". In statistics of education, "disparity" is a descriptive statistic describing a gap, which is different from "discrimination," which suggests foul play. The former is correlation while the latter is causation. This author mischievously used the term disparity while avoiding discrimination, even though the entire article presumes the disparity is unfair.

***

In fact, Stuyvesant (plus seven other similar schools in New York) only considers one factor in admissions, the student's score in the Specialized High Schools Admissions Test (SHSAT), and so the admission decision is not tainted by any judgmental criteria. All test-takers are ranked in order of test score, and then, each student is allocated a spot in their most preferred school - if all slots in the school have been filled, then the next preferred school is considered, and so on.

This allocation method guarantees "fairness" in the sense that if a student isn't allocated to a school higher on his/her preference list, it's because every offer for that school went to students who have scored higher on the SHSAT.

***

If we are interested in the question of whether the allocation of offers is fair across ethnic groups, the right question is: among students of similar ability (for example, score bands), do black or Latino students have a lower offer rate than whites and Asians?

This question of fairness pops in everywhere in education. In Chapter 3 of Numbers Rule Your World (link), I examined it in the context of designing fair SAT questions.

Surprise, surprise. Such data have not been released. Since there are no data, I'll explain conceptually how the numbers work.

***

Let's assume 30% of test-takers are Asians, and 20% each are blacks, latinos, and whites, and 10% others. Assume 50% of offers are given to Asians, 30% to whites, 5% each to blacks and latinos, and 10% to others. Overall, 15% of the test-takers receive an offer.

Given those assumptions, the offer rate for Asians is 25%, for whites it's 23%, for blacks and Latinos, it's 4%. Asians and whites are 1.5-1.7x more likely than the average to get an offer, while the offer rates of blacks and Latinos are a quarter of the average.

The NYT claims this proves that the process unfairly favors Asian and white students at the detriment of blacks and Latinos.

Not so fast. Remember all offers are completely determined by test scores, without any judgmental factors. So the proportion of students who receive offers is the same as the proportion who score above the cutoff score. Think of the cutoff score as the score of the last student taken up by a school. This cutoff score is an outcome of the offer process, not an input.

***

We can now split each subgroup into those who score above the cutoff and those below. Among Asians, the split is 25%-75%; whites, 23%-77%; blacks, 4%-96%; Latinos, 4%-96%; others, 15%-85%.

The reason why Asians and whites have more offers is because they have higher SHSAT scores.

Let's now insist that the offer rate for every ethnic group must equal the overall rate of 15%. For Asians and whites with a higher score distribution, this necessarily means that we cannot accept everyone above the cutoff score. It turns out that we can only accept about 60-67% of those who score above the cutoff score, and 0% of those who score below, in order to get to a weighted average of 15%.

For blacks and Latinos, we'd take 100% of those who score above the cutoff, but that isn't enough to get to 15%. We also must take 12% of those who score below the cutoff.

This is the unspoken math behind the NYT article. They are proposing: rejecting 40% of high-scoring Asians and whites while accepting 100% of high-scoring blacks and Latinos; and accepting 12% of low-scoring blacks and Latinos while rejecting 100% of low-scoring Asians and whites.

Shame.

 

 

Game of telephone for data messaging

We all know about the game of telephone. If one person says something to another, and the second person to a third, and so on, the message received at the end of the chain will be substantively different. What gets me thinking about this today is an article by Gelman and friends, which essentially describes the game of telephone as it applies to data messaging: "Through endless repetition, numbers of dubious origin take on the veneer of scientific fact, in many cases in the context of vital public-policy debates."

I love their leading example. Apparently, (at least one) university in the U.S. circulated a public-service notice telling people to wear a hat on cold days because "most body heat is lost through the top of the head." The authors attempted to trace the origin of this message, and discovered that the advice purportedly came from a statistic, which claims that 50 or even 80 percent of body heat is lost through the (apparently cap-less) head.

[Most of the article is about tracing statistics on "corruption", and about the concern that politicians mislead the public using junk statistics.]

***

This little scenario reveals a number of things about our relationship with data.

First, repeating data seems to follow what Kahneman calls "fast thinking", not "slow thinking". People tend not to slow down, and apply logic, and ask: could we really be losing 80 percent of body heat through our head? Could a human being survive after losing 80 percent of its body heat? (Apparently, the more correct answer is 10 percent.)

Second, because of the fad built upon "unconventional wisdom," spread by the likes of Malcolm Gladwell and Steven Levitt, people may believe illogical data -- the more bizarre, the better. Shock value is the currency of social media, so this fad lives.

Third, I'm interested not just in the likelihood that the message has changed through repetition, but in what direction the message shifts. My working assumption is that the default setting of most people is story-first (as opposed to data-first). The way data are consumed is typically to support some hypothesis, i.e. preconceived notion. In the game of telephone, the message is continuously shaped by the story-first mentality of the messenger.

This idea is an expanded version of what the authors called "decorative statistics", i.e. making numbers "sound big and impressive".

Each person receives a message, but only partially due to memory failure. Each person has its own belief, such as about the effect of not wearing hats on cold days. The outgoing message will tend to be pulled toward those prior beliefs.

When I find some time, I should run a simulation using Bayes' Rule to propagate messages at each stage and explore the above idea more rigorously.

***

Another factor enables this game of telephone in data stories. Every statistic ever reported has nuances - it is only true for specific populations, after certain exclusions, accounting for particular biases, etc. However, in messaging data, we don't have the luxury of attaching two paragraphs of explanations for each number. Therefore, the nuances are lost.

The orphaned data are out in the open. This invites anyone to adopt these statistics and shape them to their own liking. Fill in the blanks. If one is completely rigorous, one plugs the gaps by researching the origins of the data, but who has the time, right? So, instead, one just adds whatever one thinks "makes sense"... this is where we ends up in a story-first mode, because making sense is in the context of that preconceived story. If we want people to wear hats, we like the story that hats keep heat contained in the body, and if we can't quite remember the statistic, we are susceptible to guesstimating a number that supports our story.

 

How do storytellers use data?

A friend forwarded an article about minimum wages in Hong Kong published by CNN (link). The article is filled with statistics and data. It's interesting to observe how the reporter chose to present them.

(excerpt from this amazing infographic by SCMP)

The following list is in order of appearance in the article:

• minimum wage went up by 32 cents (in the title of the article)
• one of the world's priciest cities
• the least affordable housing market on the planet
• parking spaces that go for nearly a milion dollars each
• new minimum wage is 40 HK dollars (US$5.1) per hour
• previous minimum wage was 37.5 HK dollars (US$4.78) per hour
• minimum wage was first established in 2011
• minimum wage was frozen to 2021 levels since the pandemic
• tied with Los Angeles as the fourth place on the Worldwide Cost of Living Index in 2022, with only New York, Singapore and Tel Aviv in front of it
• New York's minimum wage is US$15 per hour, Tel Aviv's about US$8.27-8.45, depending on total hours worked, Singapore "offers different minimum wages depending on sector"
• the median hourly wage last year was 77.4 HK dollars ($9.86)
• severe wealth gap, with the richest households making 47 times more than the poorest
• the income of the poorest decile (bottom 10%) fell by more than 20% compared to their incomes before the Covid outbreak
• the proportion of people living on minimum wage is actually lower than a decade ago: only 2.6% of employees earned less than $40 an hour in 2021, compared to 6.4% in 2011 [for some reason, this number is stated in HK dollars and not translated into US dollars, as are all the other values in the article. the $40 an hour is the new minimum wage equivalent to $5.1 US]
• the gap between median and minimum wage has increased
• food costs have risen more than 7% from December 2019 to December 2022
• energy costs have also increased by more than 11% during the same period

***

What should we make of the 32-cent increase in the headline? It is designed to shock American audiences and the context for interpreting that number is not provided until later in the piece, much later.

The first piece of hidden data is the absolute value of the minimum wage, which was US$4.78 prior to the raise. This means that the raise was about 7%. Scroll to the bottom of the list, we find that food inflation was 7% aggregated over a 3-year period, so roughly 2.5% per year. Also for context, for most American corporate workers, it's been a long time since we got salary increases in the 7% range.

Another important piece of data is that the median wage was less than US$10 per hour. Knowing this statistic did not stop the reporter from comparing Hong Kong's minimum wage of $5 to New York's of $15. Half the population in Hong Kong earn less than two-thirds of New York's minimum wage. According to this site, the median hourly wage in New York City is $28. So those numbers aren't directly compared.

Further, below the fold, the reporter mentioned that only 2.6% of people earn less than the minimum wage in Hong Kong. The proportion earning below minimum wage has been cut by 60% in the last decade. In New York City, according to that same site, about 20% earn less than the minimum wage, which is almost 10 times higher than in Hong Kong.

Singapore is perhaps the most comparable place to Hong Kong. Singapore is described by most writers as not have a (universal) minimum wage, which is euphenism for saying the minimum wage is $0. I wonder what is the story behind that.

Of course, it's natural for labor activists to push for bigger increases in minimum wages. I just find it fascinating how the statistics were processed, selected and sequenced for publication. A different selection and sequencing would have yielded a different story line.

Were the U.S. polls accurate?

Andrew chimes in to defend the polling profession (link). He's responding to a piece in the London Review of Books that claims that the polls for the U.S. 2022 mid-term elections predicted a red wave that did not materialize.

He pointed to the Economist model that is an aggregation of polls that did not insert punditry type assumptions. This model is different from, say, Nate Silver's models for FiveThirtyEight, which incorporates some subjective judgement.

The poll aggregation model did remarkably well in predicting the overall outcomes in both the House and the Senate. In the House, the model predicted 225 Republican seats (208-244) compared to the eventual outcome of 222. In the Senate, the model predicted 51 Republican seats (46-55) compared to the eventual outcome of 49.

The poll aggregation model did not predict a red wave, as the interval estimates of the number of seats for both parties substantially overlap. In other words, it predicted a close election, which it turned out to be.

***
Andrew speculated that the LRB writer thought the polls predicted a red wave because he was "sticking to an existing narrative." The journalist regurgitates the "conventional wisdom" which happens to be wrong.

Andrew distinguishes between poll aggregation and poll forecasting, the latter combining poll data with punditry. He also focuses on the aggregate outcome (number of races predicted correctly), not the outcomes of individual races (e.g., did the polls get in right in state X?).

***

Andrew's post triggered a few thoughts of mine.

I dislike the conventional metrics of accuracy that treat every race as equally hard to predict. Given that most races in all U.S. elections are uncompetitive - as in, even a chimpanzee can predict the winner - a serious metric of accuracy should only consider competitive races. This issue is similar to that of grade inflation in education: the nominal scale (A-F) is misleading when all grades are compressed into the A-B range.

Worse than sticking to an existing narrative, the journalist is reinforcing a false narrative. I see the world as comprising two types of people in their relationship to data: data-first versus story-first (link). People who are story-first starts with the story and finds data that support the story. In the case of the 2022 elections, the Republican mouthpieces were everywhere pushing the narrative of the red wave. The polling data were an inconvenience that didn't matter.

Democrats can also be story-first. In fact, some Democrats might have allowed the false narrative to persist in the hope that fearing the red wave would drive their supporters to the polls.

When I came up with the data-first/story-first framework, I wanted something that is non-judgmental. Story-first is not always wrong; indeed, the data can be consistent with the story. In my experience, story-first is the default human instinct, even though being a data person, I wish the opposite. I'm definitely data-first but data-first isn't right for every decision either, especially when the data are inadequate, contradictory, or inappropriate.

The recent U.S. elections have been so close that it's virtually impossible to predict the result accurately. Statistical models are calibrated to say exactly that. The interval estimates shown above overlap substantially. This is a situation in which the statistician would be unwilling to put a strong bet on either side. For example, in a Bayesian sense, the Republicans were found to win control of the Senate in about 60% of simulations. This means the Democrats won control in the other 40% of simulations. Real life is just one simulation!

The model hitting the right ranges of results was a matter of skill in collecting and analyzing data. The model coming quite close to real life was a matter of luck. People find it frustrating when statisticians don't give a clear-cut answer. The real issue is not the statistics, it's the difference between story-first and data-first thinking. Story-first thinking is typically black and white; data-first thinking often involves shades of gray - as Andrew likes to say, we "embrace uncertainty".

 

P.S. [4/28/2023] Reader Mark P. has some further thoughts on this topic. His point about the inutility of election forecasting reminds me of this past post.

Every headline produced by real world studies has been retracted and rewritten

Back in March, I reviewed several early "real-world studies" of vaccine effectiveness (here, here, here, here), including the Dagan et. al. study from Israel that eventually became one of the most frequently cited studies on the unreasonable effectiveness of mRNA vaccines.

The researchers of these early studies acknowledged the difference between observational datasets, and data coming from randomized clinical trials; specifically, the problem of selection bias inherent in the former but obviated by the design of the latter. Matching methods were deployed to reduce the effect of selection before generating a vaccine effectiveness number that mirrors vaccine efficacy in an RCT.

I also later reviewed several studies (Denmark, CDC, CDC 2) that utilized regression adjustments. Regression adjustment is an exercise in wishful thinking: for example, an age-adjusted estimate of vaccine effectiveness is the value of VE after removing the effect of age. Mathematically, the regression adjustment imposes a common age distribution on the vaccinated and unvaccinated subgroups, so age becomes a non-factor.

However, the age-adjusted regression estimate is relevant only if
(a) the estimate of VE for any given age range is reliable, and
(b) the specific age distribution imposed on both subgroups is realistic.

In the present situation, it is hard to believe either of those assumptions. For vaccination campaigns everywhere prioritize high-risk subgroups, particularly older people with or without comorbidities.

The headlines coming out of those early real-world studies were loud and clear: real-world evidence confirmed that the vaccines performed as well as and maybe even better than the clinical trials indicated. The media highlighted VE estimates of over 90% on all reported cases of Covid-19.

As a reminder, this was a CNN headline from February 2021.

The first sentence of the article cites the 90% number: "Pfizer-BioNTech's Covid-19 vaccine appears to reduce symptomatic coronavirus infections by more than 90% in the real world, Israeli researchers said Sunday."

If you want the other view, Fox News said the same thing.

***

By end of May, amidst reports of breakthrough infections, the story started to evolve. The new headlines now focused on VE estimates of close to 100% on reported severe cases, hospitalizations, or deaths. Readers were admonished for ever believing that vaccines should prevent disease, even though the primary endpoints for all clinical trials and early real-world studies were prevention of symptomatic, PCR-positive Covid-19 cases in the selected case-counting window.

Since June, the U.K., despite having one of the world's fastest vaccination programmes, began to suffer another surge in cases. The new headlines now drew attention to hospitalizations rather than cases.

The report cites a Public Health England study, saying "although the Delta variant reduces the effectiveness of vaccines against symptomatic infection, two doses of COVID-19 vaccine still protect against severe disease."

Israel, the early star in vaccinations, faced a similar scenario, and their reporters followed the same path, delivering headlines such as this:

It's not about infections anymore; only hospitalizations matter.

It's not just the headlines that changed. This new batch of studies employs different methods of analysis. Matching and regression adjustments were both sent to the scrap pile. The new analyses now wantonly ignore selection and other kinds of biases, and treat the observational data as if they were collected from randomized clinical trials. They assume the vaccinated and unvaccinated groups are balanced on all variables, an assumption not remotely plausible.

Here is an example of such a calculation, lifted from Matan Holzer's twitter account (see title of bottom chart):

In these type of studies, vaccine "efficiency" for hospitalizations is based on the ratio of the odds of hospitalizations for vaccinated vs unvaccinated to the odds of vaccination. The same ratio is obtained by dividing the hospitalization rate of the vaccinated by the hospitalization rate of the unvaccinated. This is the analysis that would have been performed to evaluate VE in a clinical trial - although those trials did not contain enough participants to attain statistical significance on the hospitalization metric.

Such analyses gave rise to headlines that proclaim that only unvaccinated people are sick enough to go to hospitals.

What are the data supporting the above headline (link)? It's the share of hospitalizations that are attributed to the unvaccinated people.

In the meantime, the following chart from BBC shows how the new wave of infections is  growing even faster than in the last surge, which happened before vaccinations.

***

By mid July, hospitals in the U.K. and Israel are admitting a rising number of fully vaccinated patients. So, the story is shifting again.

The previous metric comparing vaccinated and unvaccinated groups as if they were randomly assigned is abandoned. Experts now pretend that they did not introduce this analysis to the public, and criticize those looking at this number as a probability illiterate - failing to grasp Bayes' Rule. (here) Nate Silver chimed in:

I agree that the efficiency metric is poorly conceived, a fact that was as obvious in May as it is in July. I don't like the "story-first" mentality adopted by many scientists, which led to all those headlines about almost no vaccinated people getting hospitalized.

As the hospitalization story falls apart, the media is lining up behind the VE metric on deaths. We are being told that the vaccines have broken the link between infections and deaths, which reminds me of when they said the vaccines severed the link between infections and hospitalizations.

The math has been clear on this from the start: hospitalizations lag cases, and deaths lag both. We are going to find out in the coming weeks whether this latest narrative sticks

We need a post-mortem on the real-world studies, a honest evaluation on why these findings have low predictive value.

 

 

P.S. [7/23/2021] The same story is unfolding in Israel. Weeks ago, they happily cited naive estimates of vaccine effectiveness - first on infections, later on severe disease. Now that those metrics don't show what they used to show, suddenly those metrics are worthless. This Times of Israel article is highly informative.

Instant Classic for Science Communications Students

When I blogged about Pfizer's adolescent trial (here), I was wondering when and if they will release the data or the scientific report to support those press releases (one by Pfizer, and one by the FDA). A report has just been published in the prestigious New England Journal of Medicine.

I predict this will become a staple of future science communications textbooks.

The 12-15 year old age group presents extreme difficulties for conducting clinical trial research - the case rate is expected to be extremely low, and there is a strong desire to fast track approvals. Those two issues are incompatible. You need a huge sample size to detect low rates of events, let alone differences between low rates of events. From the start, the FDA has allowed a head fake: the primary endpoint of the adolescent trials will be immunogenecity results, i.e. test of antibody levels in the vaccinated. This is no ordinary trial, as we will not be comparing adolescents to adolescents, vaccinated against placebo. We will be comparing adolescents to 16-25 year olds, vaccinated against vaccinated.

So what does the highly respected NEJM say about the 16-25 year olds, the designated "control" group? I searched for every mention of 16-25 year olds in the report. Below are some representative examples of the matching sentences (Scroll to the very end of the post to see the entire list of mentions).

• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.

You can read every word of the above sentences, or every word of the entire report, including every word in the footnotes, and you walk away thinking this adolescent trial included two age groups (12-15 and 16-25). What if this weren't true? What if the number of 16-25 year olds in this adolescent trial is exactly zero?

As I was reading the prestigious NEJM report, I was feeling a sense of dread, which worsened as page after page repudiated what I thought was true: that the 16-25 year olds data came from the previous adult trial. Indeed, one of the key issues I flagged in my previous blog post is that the analysis plan involved comparing one age group in one trial to another age group in a different trial. Did I mislead my readers? Was I so blinded by my own biases that I made up this detail?

It surely was not possible that scientists and peer reviewers and the editors of the prestigious NEJM would allow such a basic but important detail to be omitted. It has got to be my brain fog... am I losing grip of my sanity?

So I checked my notes... The very first line of the Pfizer press release from March 31 stated unmistakably: "In participants aged 12-15 years old, BNT162b2 demonstrated 100% efficacy and robust antibody responses, exceeding those reported in trial of vaccinated 16-25 year old participants in an earlier analysis, and was well tolerated." (their italics)

Next, I checked the FDA press release on May 10. The source of data for the 16-25 year olds has been wiped out. In the safety section, the FDA stated "The side effects in adolescents were consistent with those reported in clinical trial participants 16 years of age and older. " As if the adolescent trial contained participants 16 years of age and older. In the efficacy section, the FDA stated "The immune response to the vaccine in 190 participants, 12 through 15 years of age, was compared to the immune response of 170 participants, 16 through 25 years of age. " Unless you know otherwise, you're going to assume the trial contained participants of both age groups.

I warned you up top that you're witnessing an instant classic in science communications.

It takes discipline and skill to write an entire report that dances around a key detail. A true masterclass in deception. This is an instance of "story time," lulling readers in with clinical trial data but presenting conclusions from analyses that do not enjoy the benefits of an RCT. But "story time" trivializes what has happened here. I hate to use the F word. I'm not thinking the four-letter F word. The five-letter one. F R A U D.

***

Also of interest is the other side of the equation. How does one develop a nose for detecting fraud in data? I have to admit that it would have been very hard for anyone to notice the deception in this NEJM paper - that's why it belongs on the syllabuses of science communications courses.

There were just two tiny cracks. The first was when they disclosed that all 12-to-15-year-old participants came from the U.S., while 16-to-25-year-old participants were globally recruited. This would have been unusual to do in a single trial enrolling 12-25 years old, stratified into two age groups.

The second crack was in the way the waterfall chart was presented.

Notice the two age groups were presented in two separate panels. What we expect to see if the trial actually enrolled both age groups is something like this:

The waterfall starts with the entire set of enrollees and then there is a split by age groups. This chart came from the Phase 2 Moderna study of two half-doses that I discussed in this post.

As I said, these are subtle signs, easily missed.

***

Is it never allowed to pull a control group from a different place? Doing so does not automatically disqualify the analysis but such an analysis does not enjoy the exalted gold-standard status of a randomized controlled trial. Such workarounds must be scientifically and rigorously justified. Work must be presented to convince readers that any biases coming from different times, different places, different ages, different demographics, different anything have been measured and corrected for.

Here, the authors did not present any evidence about possible biases. They didn't even acknowledge the source of the control group data.

***

Listed below is every single mention of the 16-to-25-year-old control group in the Pfizer adolescent study in the NEJM scientific report. Each mention represents an opportunity for the researchers to inform readers that this control group came from a different trial.

• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was an immunogencity objective.
• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The immunogenicity objective was to show noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants.
• Noninferioirty was assessed among participants who had no evidence of previous SARS-CoV-2 infection with the use of the two-sided 95% confidence interval for the geometric mean ratio of SARS-CoV-2 50% neutralizing titers in 12-to-15-year-old participants as compared with 16-to-25-year-old participants 1 month after dose 2.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was assessed on the basis of the geometric mean ratio of SARS-CoV-2 50% neutralizing titers.
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Severe injection-site pain after any BNT162b2 dose was reported in 1.5% of 12-to-15-year-old participants and in 3.4% of 16-to-25-year-old participants.
• After BNT162b2 injection, severe headache and severe fatigue were reported in a lower percentage of 12-to-15-year-old participants than of 16-to-25-year-old participants.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• The use of antipyretic agents was slightly more frequent among BNT162b2 recipients who were 12 to 15 years of age than among those who were 16 to 25 years of age.
• [Footnote of Table 1]: Results are for the reactogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and a subset of participants in the 16-to-25-year-old cohort.
• Among the 16-to-25-year-old BNT162b2 recipients, 11% reported any adverse event and 6% had vaccine-related adverse events. Among BNT162b2 recipients, severe adverse events were reported in 0.6% of those who were 12 to 15 years of age and in 1.7% of those who were 16 to 25 years of age.
• One BNT162b2 recipient in the 16-to-25-year-old cohort discontinued vaccination because of severe vaccine-related injection site pain and headache, both of which were reported 1 day after dose 1 and resolved within 1day.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.
• Appendicitis was reported in 2 participants: 1 BNT162b2 recipient in the 16-to-25-year-old cohort and 1 placebo recipient in the 12-to-15-year-old cohort.
• The immune response to BNT162b2 in 12-to-15-year-old adolescents was noninferior to that observed in 16-to-25-year-old young adults.
• The geometric mean ratio of the BNT162b2 neutralizing geometric mean titer in 12-to-15-year-old participants to that in 16-to-25-year-old participants 1 month after dose 2 was 1.76.
• Among all participants regardless of serologic evidence of previous SARS-CoV-2 infection, the serum-neutralizing geometric mean titer 1 month after BNT162b2 dose 2 was 1283.0 in the 12-to-15-year-old cohort and 730.8 in the 16-to-25-year-old cohort.
• Substantial increases in the 50% neutralizing titer from baseline were observed, with GMFRs from baseline to 1 month after dose 2 of 118.3 among 12-to-15-year-old participants and 71.2 among 16-to-25-year-old participants.
• [Caption of Figure 2]: The results shown are for the reacogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and the subset of participants in the 16-to-25-year-old cohort who had electronic diary data available.
• [Footnote of Table 2]: The geometric mean ratio and two-sidded 95% confidence intervals were calculated by exponentiating the mean difference of the logarithms of the titers (the 12-to-15-year-old cohort minus the 16-to-25-year-old cohort) and the corresponding confidence intervals (based on the Student's t distribution).
• Antipyretic use after both dose was slightly higher in the 12-to-15-year-old cohort than in the 16-to-25-year-old cohort.
• All 12-to-15-year-old participants in this trial were enrolled at U.S. sites, whereas the 16-to-25-year-old participants were recruited globally.

One-dose vaccine strategy elevates PR over science

I fear that the U.K. policy of one-dose vaccines will backfire, and cause the pandemic to continue longer than necessary. Here are several reasons why.

Partial protection provides a convenient excuse for vaccinated people to do away with inconvenient mitigation measures

Assuming that the trial results hold, people receiving two vaccine shots have a very low chance of getting infected, and thus, it is acceptable that they return to normal living after vaccination. (I myself would not, since I think two doses are closer to 70 percent effective in reality, and we don't know that the vaccine stops asymptomatic spread, and so continuing to reduce contacts is advisable. Yet, under this scenario, one should not begrudge people who decide to return to normalcy.)

If people are given partial protection through just the first shot, it is irresponsible to return to normal living. Even the vaccinated must continue to wear masks, maintain social distancing, and stay at home as much as possible for the foreseeable future. Even now, it has proven impossible to enforce mitigation measures - at least not in the U.S. where I live. Vaccination presents a convenient justification for non-compliance.

This behavioral problem alone should have doomed the one-dose lobby.

One-dose treatment solves a problem that does not exist

The first weeks of vaccination has shown that the problem is demand not catching up to supply, not the other way round. We have more doses shipped than doses that are going into arms. If people don't like your cakes, you can't solve this problem by selling half-cakes!

The high efficacy of single doses is wishful thinking

Anyone who believes that the efficacy of the first dose is 90% lives in a fantasy world. Such a conclusion requires cherry-picking the best-case scenario, ignoring results that don't fit the theory, ignoring the margin of error, assuming that the second shot is essentially worthless, and assuming that the first shot is effective for the pertinent period of time without any boost from the second shot. The last two assumptions expose the one-dose strategy as a tautology - a conclusion that stems directly from one's assumption.

In a prior post, I explained how any guess of single-dose efficacy requires an assumption of (a) the efficacy of the second dose and (b) the duration of protection of the first dose, for neither of which do we have direct evidence coming from the vaccine trials.

It's a jigsaw puzzle, not a Barbie doll

People who propose the one-dose strategy are treating the clinical trial results like a Barbie doll. They are switching the green dress with the red skirt, or the high heels with flats. In fact, the toy is a jigsaw puzzle, in which each piece has its exact place, and cannot be haphazardly substituted.

For example, the headline vaccine efficacy numbers of 95% - recall the decimal obsession by Moderna - is based on a case definition in which they only count cases that occur at least 7 to 14 days after the second shot. You simply can't talk about that number in the same breath as efficacy for the first shot which requires counting cases after the first shot and before the second shot. They are not comparable numbers.

Use the scientific method properly

What supporters of the one-dose strategy should do is to lobby the FDA and/or pharmas to immediately start new clinical trials to test the one-dose strategy against the two-dose treatment.

From a scientific perspective, the biggest problem is post-hoc theorizing. Let's use stock market performance as an example. Do you feel like the business press are having it both ways? If stock prices went down yesterday, it's because the unemployment numbers were terrible. But if stock prices went up yesterday, it's because the terrible unemployment numbers were not as bad as imagined, or because businesses expected the job market to start improving, or because executives shrugged off the terrible unemployment numbers. They end up fitting a story to the observed data, as opposed to using the observed data to confirm a hypothesis.

Given the setup of the vaccine trials, before seeing these results, no one would have argued for one dose. The argument emerges after one sees the particular outcome from the trial. Trouble is, if the trial were run many times, would we have seen the same efficacy curve? (The variance across different trials already tells us the answer is NO.)

If the front part of the curve looks less favorable (say 30%), these same people would say we can't draw inferences about one dose (as opposed to one dose isn't going to work). In other words, this analysis contains an optimism bias, as does any theorizing after peering at the data.

***

Despite what I said above, I rate it as likely that certain countries will eventually follow the U.K. example and allow one-dose treatment. (As of now, the FDA has sensibly decided not to go down this path.)

Politicians score easy PR points by claiming they are helping "more people" when in reality, such a policy will lengthen the public-health crisis. Previously, I already noted that the PR agency of the UK government should be given an award.

 

P.S. Sadly, it appears that the incoming Biden administration is intent on elevating PR over science, as news has emerged that they will "release all available vaccines" and "not withhold second doses". This is euphemism for giving out one dose to everyone (as opposed to reserving the second doses for those who have gotten the first dose). For those "scientists" who are jumping behind this policy, I'd request that they answer the questions listed on my previous post. I can be persuaded that the one-dose strategy will lead to a better outcome but I'm having trouble understanding what set of assumptions lead to such a policy.

Added to the problems listed above, I think this latest news may persuade some of us to become vaccine skeptics. Those of us who believe the data show the vaccine needs two doses to work, and who do not get assurance that the second dose will be available in the right time frame, may decide to opt out of the vaccine.

 

P.P.S. In justifying this misguided policy, people are going to say "don't let the perfect be the enemy of the good". This is the cue that they are about to ignore the science. No one ever said the vaccine trial results are "perfect"; in fact, many of us believe they are less than perfect. I have written frequently about the shortcuts that were taken. We are not letting the perfect be the enemy of the good; we are opposing letting the unknown replace the good enough.

Other than some handwaving, I have not heard any description of what "the good" entails. If you can't quantify it, it's not science. If we are not going to argue about the science, then we must debate the assumptions, and evidence supporting such assumptions. Let's go!

Story-first thinking from the business press to Election Night

I'm been toying with this idea. I want to move away from the tired data vs intuition dichotomy. I don't think it accurately describes what is going on. The people who are labeled as intuitionists often include data in their arguments. The people who are experts at harvesting insights from data readily admit using intuition. So that distinction is hard to nail down.

Instead, I think a more useful dichotomy is data-first versus story-first. Some people have a story, and they are good at building support for the story by assembling pieces of data. Other people like to shake the story out of the data. You know who you are.

***

Last week, I came across a good example of story-first thinking in the business press. An article in Business Insider has the headline "Bed Bath and Beyond is slashing the coupons that have long been synonymous with the brand..." The article talks about how the struggling retailer may have found a lifeline in the pandemic; good for them!

Then came this juicy bit:

The writer tells us that BB&B customers don't care about those coupons anyway. To support that, she said "50% of [...] customers said that they shopped at the store because of its coupons". I mean, if half your customers came to the store because of the coupons, I'd say coupons are really important to this business!

Don't worry though because she rescued the story by telling us that the 50% is "a lower number than in previous years."

***

What's going on here? I don't think she was blinded by intuition. She literally used data in this argument and throughout the article so she qualifies as data-driven.

You can't say she messed up the data either. Her reasoning is standard - it just doesn't work with the numbers she's got. If only 5 percent of Bed Bath and Beyond customers said they shopped there because of coupons, I wouldn't have written this blog post.

I also do not think the writer is being manipulative. I think what she did came naturally to her - the story-first mindset, by which you assemble the pieces of data that support the story you have in mind. This style of thinking is commonplace, in my experience.

***

It's the day before Election Day in the United States. We are about to drown in a flash flood of story-first thinking. Journalists have been writing articles to warn against premature conclusions on Election Night. They even paint the worst-case scenario in which Trump leads in in-person voting but ultimately loses the election when mail-in ballots are tallied. (If I am not mistaken, the New York Times was the first to sound this alarm, way back in early September.)

For my international readers, here's how messy elections are here. There is no federally managed and standardized electoral process. Every jurisdiction, sometimes at the state level, sometimes at lower levels, sets their own rules. As a result, there is a built-in unfairness because your ease of voting depends on where you live. In particular, people who live in densely populated cities frequently have to wait in line, sometimes hours, to cast a vote. Because of the raging pandemic, some people just don't find it safe to stand in lines for hours, and many jurisdictions have created alternatives such as mail-in voting and drive-through voting.

When it comes to mail-in voting, the two parties have been fighting over it. An incomplete list of the major issues are: whether one needs to apply for a mail-in ballot or if the state should just distribute everyone a mail-in ballot; how many collection boxes should there be for mail-in ballots; whether churches or other private groups can set up their own collection boxes for mail-in ballots; the deadlines for sending in mail-in ballots; whether the Post Office should prioritize or retard the delivery of ballots; what to do with "late"-arriving mail-in ballots; what constitutes a "valid" mail-in ballot; when states can start counting mail-in ballots; how much time states are given to count mail-in ballots, etc. etc.

What all sides agree on is that incomplete data nuggets will arrive throughout Election Night, and the final story is unlikely to emerge until days or weeks later - unless the voting margin is decisively large.

So I put this question up on Twitter last week: "Given the media keep telling us to be patient next Tuesday, will the media refrain from horse-race reporting until the final tally is available?"

What's going to happen is that the political pundits will assemble these data nuggets to support whatever story they have in mind. In fact, this has already started. On cable news, they have been reporting the number of mail-in ballots already received in various states, and the party registration of these voters (including, mind you, a good chunk of unaffiliated voters who don't want to register with either party).

And then, they are telling us which party is ahead - clearly based on assumptions of how much "disloyalty" there exists within each party, and the vote split among the unaffiliated.

How do these journalists or data journalists form their assumptions? Remember that historical data are not useful since there is an explosion of mail-in voting due to the pandemic. The assumptions are colored, partially or wholly, by how they think the election will turn out. In other words, they will be engaging story-first thinking.