Tokenism is an act of virtue signaling: taking an action in order to create an appearance of having solved a problem but the appearance runs ahead of reality. The canonical example is the hiring of "token minorities" to demonstrate that businesses have inclusive hiring practices. The symptom is the appearance of a single female or minority face in a group picture. Remember this Buzzfeed reveal?
From a statistics perspective, tokenism works by elevating the anecdote - the outlier example being peddled as representative.
***
This post is my third set of notes inspired by Pfizer's 6-month study on the safety and efficacy of its Covid-19 vaccine. The other two posts are here and here.
Today, I draw attention to this paragraph (bolding is mine):
The current report focuses on phase 2/3 safety assessments in ≥16-year-old and prespecified VE assessments in ≥12-year-old participants through up to 6 months post-immunization. Because enrolment of 12-15-year-olds began on October 15, 2020, 6-month post-immunization data are currently unavailable for this age cohort. Shorter duration safety, immunogenicity, and efficacy data for 12-15-year-olds are reported separately; however, data for this cohort are included in overall VE analyses reported here.
The original trial involved people 16 years old or above, and 44,000 participants enrolled between end of July and end of October 2020. Later, a teenager trial was added, with enrollment of ~2,000 between October and January, 2021. The overlap between the two trials was extremely short, as at interim analysis of the adult trial (conducted in mid November), Pfizer said there were only 100 12-15-year-olds with data.
The interim analysis generated a result of the form: "Vaccine efficacy for ≥16-year-old is 96% from 7 days after 2nd dose to median 2 months." This new study led to a result of the form: "Vaccine effiacy for ≥12-year-old is 91% from 7 days after 2nd dose to median 4 months." (I explained in a previous post why I don't think it is appropriate to say 6 months.)
***
Did we just prove that the vaccine works just as well for 12-15 year olds? It may seem so but this is an example of what I'm calling "tokenism" in statistics.
In order to have a statistically significant result for teenagers, we must have enough samples of participants between 12 and 15 years old. The teenager trial did not include enough people, which was why the researchers pre-specified an immunogenicity outcome (antibody levels) as the primary endpoint.
Adding the 2,000 teenagers to the study with 44,000 adults does not produce more teenager outcomes. In fact, it's such a small portion of the aggregate sample it's not likely to have moved the aggregate result. What's more, because of the later enrollment dates, almost no one in the teenager group has reached 4 months post 2nd dose so their effect on the cumulative case curves is strictly limited to short-term efficacy, the part of the curve for which most of the adult participants contributed data.
Nevertheless, after merging the two age groups, the researchers are able to phrase their finding as "Vaccine effiacy for ≥12-year-old is 91% from 7 days after 2nd dose to median 4 months." This is not a false statement. It is tokenism.
To see tokenism clearly, I find it useful to speak facetiously. If I find 2,000 cadavers and inject half of them with the vaccine, and then add this group to my dataset of 44,000 living people, now I can say my vaccine works for both living and dead!
***
In a previous post, I explain why the headline VE result "from 7 days after 2nd dose to up to six months after 2nd dose" is better interpreted as "up to about 4 months after 2nd dose". This is another example of tokenism.
The published statement paints the picture that the researchers have data for participants up to six months after 2nd dose - one presumes that even if not all participants have six months of follow-up, most would have. Yet, inside the paper, they disclosed that only 7 percent of the evaluable participants have been followed up for at least six months, a sharp drop due to allowing placebo participants to get the vaccine earlier than planned.
Tokenism does not require lying. The businesses have hired the minority workers but the hiring has made barely a dent on the aggregate statistics.
***
Tokenism is hard to diagnose unless the consumer of the analytical finding has the time and desire to dig into the details of the study. An outcome for >= 12-year-olds on top of an impressive outcome for >=16-year-olds suggests the new study contained a sufficiently large sample of 12-15 year olds. A follow-up period described as "up to six months" suggests that most of the participants have been observed for about six months.
In the same way, a picture of an outing with a couple of token women signals that the company promotes workforce diversity.
Comments