Big Data, Plainly Spoken (aka Numbers Rule Your World)

Round 2: 10 Ways to Rank the Rio Summer Olympics

In a recent post, I took data from the Tokyo Olympics to demonstrate how one can find any number of data-driven stories in a sufficiently complex dataset. I outlined 10 different ranking schemes that place a different country first.

A few readers noticed and complained that the metric used for Germany was "not positive". They didn't like the insight that Germany saw the greatest decline in medals from Rio 2016 to Tokyo 2020. While "not positive" metrics have their practical uses, I indeed was hoping to make a purely "positive" post, and ran out of time. (I said the challenge can be completed; I never said it was easy!)

Since today is a new day, and this is a new post, I will now close the gap. Germany's older athletes, those over 45 years and over, were by far the strongest. Incredibly, two out of third earned medals.

Now, surely no one can question the completeness of my answer.

***

Since a few of you were cheering for me to fail, I decided to double down, and present to you another 10 metrics, proving that if you spend enough time with the data, you can pretty much say anything you want.

Scratch that. In the course of doing this second round, I bumped into one of those awkward data science moments: you see, the exploratory analysis caused me to look up an athlete's bio, and doing so, I discovered that the spreadsheet I snatched online did not show Tokyo athletes as expected but athletes from Rio 2016. So in this second round, what I will deliver is the same challenge but using strictly the data from the Rio 2016 Summer Olympics.

As before, I start with the official top 10 medals table from Rio 2016.

Nine of the 10 countries are the same as before. The Netherlands was swapped for South Korea. (Russia competed under its proper name rather than ROC.)

The United States won on four metrics, golds, silvers, bronzes and total medals - and plenty of others, if you start digging around.

China won the second most medals at Rio 2016. But the Chinese team had the highest proportion of female athletes - by a wide margin.

Team GB came in #3. No other team, though, produced a gold winner older than Nick Skelton, who at 59 proved that age is no barrier.

Russia was 4th in the overall medals table. Its women athletes were the most efficient at earning medals.

Germany was #5. No other team achieved more success amongst older athletes - those 45 years old and over.

France came in #6. Among the top 10 countries, the French medal-winners had the highest age.

Japan was #7 overall. They were the mirror image of the French: the average Japanese medal-winner was the youngest.

Australia showed up in the 8th spot. The Aussie gold medalists were even younger than the Japanese.

The Azzurri placed 9th overall. Previously, we learned before that the Italians had the most close calls. It turned out their gold medal winners tended to be younger.

In Rio, South Korea squeaked in at the 10th position. The nation was unique in sending a team in which the women were older than the men.

I end on a strong note. This last metric was very satisfying as it single-handedly placed South Korea on on side of the ledger with the other nine countries on the other side.

 

***

There you have it. Take the Rio Olympics, and one can achieve the same result - one can find 10 metrics that rank each of the top 10 countries as #1.

This is the power of statistics. How one uses this power is a different matter. Use it to inform, not to mislead.

(For those who may be snorting at this exercise, notice that every statement I made about the Rio Olympics presented historical facts. I did not generalize these observations to the future. Some of these insights are likely to be artifacts of this historical dataset. It's when someone uses such insights to speak about the future when you should be concerned.)

Primer on Regression Adjustments 2

In the previous post of this series, I describe the standard process of using regression to "adjust" or "correct" for biases in a dataset using regression. If we use the unadjusted sample average height from Sample A2 to estimate the average height in the population, we will be off by almost 1 inch, which is a large error of two standard errors. The under-estimation is caused by over-sampling of women, who are shorter on average than men. The gender-adjusted sample average height almost exactly replicates the population average height, as if the analyst had waved a magic wand!

Using regressions to correct for biases is an extremely popular method for dealing with selection biases endemic in observational datasets. Most real-world studies of vaccine effectivness deploy regression adjustments. Nevertheless, any serious scholar of observational data analyses would not think regression is sufficient to solve the key problems in the field.

In this post, I reveal why these regression adjustments are not the magic wand they are made out to be.

The previous post ended with this question: does the software executing regression know that Sample A2 contains a gender bias, and how?

I hope you realize that the software does not know, and cannot know. Knowing there is gender bias requires knowing what the gender ratio is in the entire population of 25,000 people. There is no way anyone can infer that from inputting 900 records from Sample A2 into the regression software.

Since we ran the regression, and it gave us an equation, and the adjusted sample average is tantalizingly close to the population average, something good must have happened, right?

Assuming you have no idea what the gender ratio ought to be, and you are forced to make an assumption, what might you do?

You might assume that the population is evenly split by gender. This is exactly what the software did. Let's verify.

Take the average of the male average (69.5) and the female average (63.7) from the above table, and you get 66.6, the exact value of the gender-adjusted sample average height.

This is a hidden assumption of all such regression models. Recall the regression equation:

66.6 + 2.9 if Male, -2.9 if Female

For men, the equation adds 2.9 inches and for women, it subtracts 2.9 inches. In order for the sample average to be equal to 66.6, you need males to balance out females, which implies a 50-50 split. (Your specific software may choose a different way of coding the gender variable but since these are equivalent formulations, the math works out the same.)

When adjusting for gender, the software is computing what the sample average height would have been if the sample were evenly split between the two genders. The software does not have any foresight into whether the sample has a gender bias, or how much that bias is -- it just assumes that the population gender ratio is 50-50, and then waves the magic wand.

**

How then can we explain the "magic" regression did to Sample A2? It's simple. I have designed Population A to be evenly split by gender, and so the naive assumption proves correct in this case.

Let's see what happens in Population B, which is 60% males, and 40% females.

Since this population of 25,000 skews male, the population average height is 67.1, slightly higher than for Population A.

As before, we give the analyst a sample of 900 records, Sample B. As she quickly discovers, Sample B also has a majority of females. The sample average height is 66.1.

Once again, if the unadjusted sample average was used to estimate the population average, we would be off by 1 inch, which is a large error. The analyst waves her magic wand, and dumps the data into regression software. Out pops the following equation:

66.6 + 2.6 if Male - 2.6 if Female

The gender-adjusted sample average height is 66.6, which is closer to the population average but still not close enough.

We can verify that this adjustment embeds the assumption of a 50-50 gender split. The average of 69.3 and 64.0 is 66.6.

If we knew Population B should be 60% males, we could enforce this. Take the sample averages for males and females from the regression, and then apply this gender ratio.

60% x 69.3 + 40% x 64.0 = 67.2

Magic! Our new sample average is now an accurate estimate for the population average of 67.1. In other words, regression works really well if you know the true gender split, which is to say, you know the exact amount of gender bias. Otherwise, it makes the naive assumption that the population is balanced by gender.

Almost nothing in the real world is balanced equally. Imagine doing age-group adjustments. Is there any population that is evenly split by age groups?

***

The analyst receives yet another sample, Sample B2, of 900 people from Population B. She finds that males account for 60% of the new sample, and the unadjusted sample average height is 66.6.

Since she doesn't know what the underlying gender ratio should be, she dumps the data into a regression software - to pre-emptively cure any gender bias. Out pops the equation:

66.1 + 2.4 if Male - 2.4 if Female

Using the standard process that her teachers have taught her, she now reports that the gender-adjusted sample average height is 66.1. This is an estimate cured of any potential selection bias by gender. She calls this a debiased estimate of the population average height, after removing any effect of gender.

Since we in fact created Population B, we know the following statistics:

We know that the population average height is 67.1, and therefore, the gender-adjusted sample average is off by 1 inch, another large error. What's worse, the gender adjustment shifted the sample average in the wrong direction from 66.6 to 66.1!

Waving the magic wand appears to have produced undesirable effects. What happened?

It turns out that Sample B2 is a random sample from Population B. This explains why the proportion of men in Sample B2 is 62%, similar to that in the population. There is in fact no gender bias at all in Sample B2.

Nevertheless, the regression software defines unbiased as a 50-50 gender split, and the adjusted sample average uses those weights. Since in Population B, the correct gender split is 60-40, the adjustment moves the number in the wrong direction!

***

Now that you understand how regression adjustments work, do you think this type of regression adjustments have successfully corrected for "a huge list of confounders" in the real-world studies of vaccine effectiveness?

There is a reason why when I covered real-world studies last year, I focused on several studies that deployed more sophisticated methods, such as matching and propensity scores. Studies that just dump data into a regression software will not generate good estimates unless the analysts have a good sense of what the population distributions are of all the relevant confounding variables. Or, unless you believe in the fairy tale of equal distributions everywhere.

***

In the next post, I will wade deeper into this morass, and study other scenarios.

I did the simulation and analyses using JMP software and all the figures come from it. I may put up some R and Python code if this series generates enough interest.

I leave you with another question, perhaps rhetorical. If it is true that regression is the magic wand that cures all biases, why is there a need to run a randomized clinical trial? We should just collect observational data, dump them into a regression that corrects for all confounders, and the resulting adjusted vaccine effectiveness will accurately replicate the true value.

***

Click here to go back to the first post of the series that sets up this post.

[9-1-21] Post #3 is now available.

 

 

 

 

Primer on Regression Adjustments 1

I was once asked to write five learning objectives for some 30 lectures of a semester-long course -- amounting to over 100 bullet points possibly read fewer than 100 times in total by the students. My syllabus was summarily rejected; it probably caused a five-alarm fire in some dean's office. For example, the learning objective submitted as "Understand when to use regression analysis, and how to interpret the results" displeased because this outcome is not "measurable". I didn't argue with that and dutifully changed it to "Execute the five steps of running regression analysis", which read like gold to the currriculum board.

This is a nugget of what's wrong with statistics education. Our students can mechanically write code to run all kinds of models but they don't learn how to think critically about what they have just produced. In a series of posts, starting today, I present some materials on regression analysis that you won't find in a standard statistics textbook. At the end of this series, I hope you come away realizing that regression is a useful tool but it is not a magic wand that fixes all problems.

A lot of recent scholarly papers on the pandemic uses regression to "correct for" biases in the real-world datasets. Observers are hoodwinked into thinking that regression cures all biases. For example, even John Burn-Murdoch at the Financial Times tweeted the following a few days ago, citing a number of experts he talked to:

John believed that regression effectively cured "a huge list of confounders". You won't believe that after you finish this series of posts about regression adjustments.

***

We start with a Population A of 25,000, split evenly male and female. We have data on their heights in inches. (1 inch = 2.5 cm, 1 feet = 12 inches = 0.3 m, 1 metre = 3.3 feet = 39 inches).

The following tabulation confirms the gender ratio. The population average height is 66.5 inches (5.5 feet).

An analyst is given Sample A, which contains 900 records taken from Population A. These records are randomly selected from the population. The average height of the 900 people ("sample average") is 66.9, which is tantalizing close to 66.5. As you learned in Stat 101, this is the beautiful foundation of statistics. You can take the sample average, and use it to estimate the population average height, and you'd be very close.

How close? This is measured by the "standard error", which is 7.8/sqrt(900) = 7.8/30 = 0.26 inches. The standard error is the building block of the familiar concept of "margin of error", which is +/- 2 standard errors. Thus, the margin of error in this case is ~0.5 inch on either side of 66.9. The range is 66.4 to 67.4. Statistical theory predicts that 95% of the time, the true average height in the population is found in this range, and indeed, our sample is part of this 95%.

For later reference, just remember that 0.5 inch (12 mm) is a big error on this scale. Half an inch is the difference between the median person sample and the 97.5th percentile person sample. So our tolerance for inaccuracy is described in small fractions of an inch.

We now exit this perfect world, in which the sample of data received is a random sample of the population, that is to say, contains no biases. Almost all real-world datasets are going to suffer from selection biases.

The analyst receives a new dataset, Sample A2, also of 900 people, and determines the sample average height to be 65.5 inches. Inspecting the gender of Sample A2, the analyst discovers that it contains 70% females and 30% males. 

Since females on average are shorter than males, the sample average height of 65.5 is below the population average height of 66.5. The difference of 1 inch is a major error, as we learned from the calculation above.

Sample A2 has a selection bias favoring women. The analyst now waves her magic wand. It's called "regression." She is now going to "adjust" or "correct" the sample estimate for gender bias. The gender-adjusted sample average height is 66.6 inches. This number comes from the constant term of the following regression model:

66.6 + 2.9 if Male - 2.9 if Female

The adjusted number of 66.6 is almost exactly the average height of the population of 25,000. Thus, regression has magically removed the gender bias and delivered an accurate estimate of the population average height. What's not to like?

This is where most textbooks end the discussion. Quod erat demonstrandum.

For me, this is where the class starts.

***

Before the next post, I leave you with the following questions to ponder:

In the real world, we don't have access to the population dataset. We don't know what the population average height is, nor do we know what the gender ratio is in the population of 25,000. All we have is the sample of 900 people. Does the software executing that regression know that the sample is biased?

If you think the software knows, then how does it know?

If you think the software doesn't know, then what is it correcting?

 

P.S. [8-27-2021] Post 2 is now up. It contains the answer to the above question.

P.S. [9-1-2021] Post 3 explains why "removing" the effect of gender ironically acknowledges its importance.

 

 

 

 

 

Receipts from a year of blogging the pandemic

Today is a good day to review some of the things you've read on this blog all through the pandemic.

***

As governments pushed very hard for third doses of the Covid-19 vaccine, it may feel like ancient history when - back in January, only about six months ago - many experts were pushing for one-dose mRNA vaccines, claiming that a single dose gives you close to 90% effectiveness.

I sounded the alarm in a post called One Dose Vaccine Elevates PR Over Science (January 2021), which began with:

"I fear that the U.K. policy of one-dose vaccines will backfire, and cause the pandemic to continue longer than necessary."

We are now certain that those who argued for single doses advocated a policy that likely resulted in unnecessary suffering.

In another post, appearing right after the Pfizer EUA in December 2020, I laid out the reasons why the data cannot support a one-dose regimen. (One Dose Pfizer is Not Happening and Here's Why) Nothing I said in December has aged. They are all still true.

***

In that same post, I predicted that "Partial protection provides a convenient excuse for vaccinated people to do away with inconvenient mitigation measures." Little did I know that CDC would subsequently endorse this folly by telling vaccinated people to drop their masks.

The CDC guidance was based on hope-fueled, over-interpretation of the data. Until recently, most experts have claimed that the vaccine stops infection, even asymptomatic infection, and spread. Now, they retracted those claims. But none of those outcomes were ever formally measured in the clinical trials.

Eight months ago, when vaccinations were just starting, you read here: "I think two doses are closer to 70 percent effective in reality, and we don't know that the vaccine stops asymptomatic spread, and so continuing to reduce contacts is advisable." Many places that removed those restrictions have been forced to reimpose them.

***

"One of the key lessons of managing this pandemic so far is that good data drive good decisions, and bad data drive bad decisions. Unfortunately, policymakers have signed up for bad data, so no one should be surprised if future policies turn out badly."

That was the conclusion of another post from January, Actions have Consequences: the Messy Aftermath of One-Dose Pfizer. The situation, as of August, has not improved. At every turn, governments have failed to collect relevant data, sufficient data, and good-quality data. In fact, they have actively interfered with the ability to learn from data.

The central example from that January post explains one (of many) reasons why real-world vaccine effectiveness studies have wildly exaggerated the effectiveness of the mRNA vaccines.

"A fundamental best practice of running statistical experiments on random samples of a population is that once the winning formula is rolled out to the entire population, the scientists should look at the real-world data and confirm that the experimental results hold."

"The action of the U.K. government (and others who may follow suit) has severely hampered any post-market validation. It is almost impossible to compare real-world evidence with the experimental result, because most people are not even getting the scientifically-proven treatment per protocol!"

That's right. The original vaccine efficacy measure came from clinical trials in which the participants followed a precisely-timed two-dose regimen. In the real world, many countries deviated from the prescribed protocol, adopting policies such as one doses, or extending the dose interval from three weeks to three months, or mixing and matching vaccines. It is an insult to science to pretend we are comparing apples to apples.

These issues did not catch anyone by surprise. When I raised those concerns in January, Pfizer was just being rolled out and not a single real-world study has commenced.

A particularly galling detail is very telling. In the U.K., the average duration between two doses is 80 days (almost 3 months!) When the U.K. studies apply the (in)famous 2D+14 case-counting window - that is to say, the researchers nullify all cases that occur after the first shot and before 14 days after the second shot, they are discounting cases for more than three full months! The rationale for the case-counting window is that the vaccine requires time to attain optimal effect - surely, a vaccine that needs 94 days to become effective isn't one that has practical value!

***

It is sobering to look at the data today (well, yesterday when I pulled this chart), courtesy of the fine folks at OurWorldinData:

Israel and the U.K. won countless headlines in the first half of the year when their case rates dropped to historic lows - they were the loudest in attributing the entire drop in reported cases to one and only one cause: widespread vaccinations. Today, they have some of the highest infection rates, much higher than in South Africa and India, which had experienced brutal surges but have relatively low vaccination rates.

In the first week of May, I wrote a post called Curve Watching During the Pandemic. Even when the cases were low in Israel and the U.K., if one was willing to look at non-conforming data, one would have noticed gaping holes in the idea that vaccinations were the sole explanations for the trends at the time.

In that post, I concluded: "Simple one-factor models aren't going to work to explain trends across countries and time. A good causal model should include a baseline trendline, a vaccine factor, plus lockdowns and other mitigation measures." To this day, none of the studies covered by the media follow this strategy.

***

It is well past time to admit that the real-world vaccine studies have dramatically over-stated the effectiveness of the mRNA vaccines. This is a failure of science. Not only, it is a systematic failure because I do not know of a single study that has committed the opposite error of under-estimating the VE! It's clear science has not fielded an A team in this crisis.

On this blog, I reviewed many of the influential real-world studies that created the narrative that vaccines are as effective in the real world as they were in clinical trials. I have indicated a wide variety of problems with such studies, and all the reasons why their conclusions are over optimistic.

If you are interested in research methodologies, read this series of posts from March 2021:

Real-world Studies: How to Interpret Them (Mayo clinic/nference)

Real-world studies: limits of knowledge (Mayo clinic/nference)

The Confusing Picture in Israel and in the Israel Study (Israel Clalit)

Note on a Simpson's Paradox in Real World Vaccine Effectiveness Studies (Denmark)

What the Danish Study tells us about the CDC Study on Real-World Effectiveness (Denmark, CDC)

Eventually in July, Public Health England published a study that provided some raw data which I harnessed to explain these abstract concepts. This exercise led to a post called Real World Vaccine Studies Consistently Overstate Vaccine Effectivness. In this post, I laid out several key adjustments that should be made to real-world VE calculations, and how these adjustments would have resulted in far less rosy pronouncements. My conclusion about mRNA vaccines: "A realistic estimate of VE is probably closer to 60%, which is an excellent number."

This is no mere squabbling amongst scientists. I repeat what I said above: "Good data drive good decisions, and bad data drive bad decisions." Throughout this crisis, the policymakers did not work with good data, by this, I include good analytical findings.

***

In my very first review of a real-world study (Mayo Clinic/nference), I stated the fundamental challenge facing real-world studies: "The simplest first analysis is to compare the case rate of the vaccinated people to the case rate of the unvaccinated people. This is hopelessly flawed because in a real-world study, we must not assume 'all else equal.' People who have received the vaccines at this point are apples and oranges to those who haven't."

That was March 2021. Fast forward to August, and I'm saddened to report that the situation has deteriorated. Unfortunately, the more recent studies have almost all adopted the "simplest first analysis" that is "hopelessly flawed".

Take the recent full court press relating to the first few days of data after Israel started giving third doses to older people. On the next day, the case rate amongst those who got the third shot is immediately below those who only had two shots. To a statistician, this gap presents the best estimate of selection bias we know of - no vaccine can be expected to work at full strength within one day so the more likely explanation for it is that people who rushed to the front of the line have lower baseline propensity to get infected by the coronavirus. In other words, the third dose unintentionally reveals who the lower-risk people are.

I discuss this in a recent blog post, which showed up on the dataviz section of my blog. This selection bias is yet another reason why the early real-world studies - even those using more advanced methods - over-estimated the VE. Thus, I concluded: "Statistics is about grays. It's not either-or. It's usually some of each. ... When they rolled out two doses, we lived through an optimistic period in which most experts rejoiced about 90-100% real-world effectiveness, and then as more people get vaccinated, the effect washed away. The selection effect gradually disappears when vaccination becomes widespread. Are we starting a new cycle of hope and despair? We'll find out soon enough."

***

This is a very long post to say thank you for your past year of support, and tell your friends about this blog. I can't promise I will get everything right but so far, the record looks good :)

 

P.S. The big news of this week is the FDA's "full approval" of the Pfizer vaccine. I am not devoting more space on the blog for it than this paragraph because there is nothing to discuss: the only additional information that was publicized since the interim analysis in December was the "6-month" update, which is nothing but (link). This was not a decision based on science. In fact, on cable news yesterday, the experts were not talking about the science. The entire rationale for this decision sounded as if someone at the FDA spent time reading consumer surveys. I refer you to Peter Doshi who nicely summarized the issues at BMJ (link).

 

Demo of how data can tell any story you want, ode to Tokyo

A reader sent me to this BBC article, which is typical of a genre that popped up once the Tokyo Olympics closed.

As sports fans know, the United States ended up topping the medals table, whichever way you look at it. But to journalists, that makes for a boring headline.

So, they invented new ways of ranking the countries. For example, using medals per capita, they elevated tiny countries like San Marino to the top. Or, using GDP data, they found China to be the winner.

***

In this light-hearted post, I gave myself a challenge: elevate each of those top 10 countries from the official medals table to the top by inventing new metrics; also stick with Olympics data and don't bring in other datasets. What I am demonstrating is that when you have enough dimensions in your dataset, you can pretty much give top rank to anything you want.

My starting point is the official medals table. The United States come up top:

China is #2. But China is top if I look at the average number of golds won per athlete.

ROC came in #3. But ROC is top if we count the average number of medals (any color) won per athlete... and the ROC women did particularly well on this count.

Team GB came in 4th. Nevertheless, I can make them #1 by looking at the breadth of sports in which the British athletes won golds.

Japan, the host nation, ended up #5 on the medals table. But Japan is #1 in terms of the proportion of medals won that were golds.

Things get tougher as we go down the list. But no worries. Data to the rescue.

Australia placed 6th overall in the medals table. When it comes to their women athletes winning golds relative to the men, they outshone the other teams by far.

Italy showed up at #7. But the Azzurri suffered the most close calls of them all, as it had the most silvers and bronzes relative to golds.

Germany was 8th. It has possibly the most disappointing campaign as its gold haul dropped by the most from Rio 2016.

The Netherlands grabbed the 9th spot. In contrast to their neighbor, the Dutch earned the greatest growth in number of medals.

France rounded off the top 10 in the official ranking. But the next host nation is #1 when I look at how much the French men outshone their women in winning medals in Tokyo.

***

So I completed my challenge. You can indeed find a way to rank any of these nations #1.

That's a fun way of saying if you have enough dimensions in your dataset, you can pretty much prove anything.

 

 

Tokenism in statistics

Tokenism is an act of virtue signaling: taking an action in order to create an appearance of having solved a problem but the appearance runs ahead of reality. The canonical example is the hiring of "token minorities" to demonstrate that businesses have inclusive hiring practices. The symptom is the appearance of a single female or minority face in a group picture. Remember this Buzzfeed reveal?

From a statistics perspective, tokenism works by elevating the anecdote - the outlier example being peddled as representative.

***

This post is my third set of notes inspired by Pfizer's 6-month study on the safety and efficacy of its Covid-19 vaccine. The other two posts are here and here.

Today, I draw attention to this paragraph (bolding is mine):

The current report focuses on phase 2/3 safety assessments in ≥16-year-old and prespecified VE assessments in ≥12-year-old participants through up to 6 months post-immunization. Because enrolment of 12-15-year-olds began on October 15, 2020, 6-month post-immunization data are currently unavailable for this age cohort. Shorter duration safety, immunogenicity, and efficacy data for 12-15-year-olds are reported separately; however, data for this cohort are included in overall VE analyses reported here.

The original trial involved people 16 years old or above, and 44,000 participants enrolled between end of July and end of October 2020. Later, a teenager trial was added, with enrollment of ~2,000 between October and January, 2021. The overlap between the two trials was extremely short, as at interim analysis of the adult trial (conducted in mid November), Pfizer said there were only 100 12-15-year-olds with data.

The interim analysis generated a result of the form: "Vaccine efficacy for ≥16-year-old is 96% from 7 days after 2nd dose to median 2 months." This new study led to a result of the form: "Vaccine effiacy for ≥12-year-old is 91% from 7 days after 2nd dose to median 4 months." (I explained in a previous post why I don't think it is appropriate to say 6 months.)

***

Did we just prove that the vaccine works just as well for 12-15 year olds? It may seem so but this is an example of what I'm calling "tokenism" in statistics.

In order to have a statistically significant result for teenagers, we must have enough samples of participants between 12 and 15 years old. The teenager trial did not include enough people, which was why the researchers pre-specified an immunogenicity outcome (antibody levels) as the primary endpoint.

Adding the 2,000 teenagers to the study with 44,000 adults does not produce more teenager outcomes. In fact, it's such a small portion of the aggregate sample it's not likely to have moved the aggregate result. What's more, because of the later enrollment dates, almost no one in the teenager group has reached 4 months post 2nd dose so their effect on the cumulative case curves is strictly limited to short-term efficacy, the part of the curve for which most of the adult participants contributed data.

Nevertheless, after merging the two age groups, the researchers are able to phrase their finding as "Vaccine effiacy for ≥12-year-old is 91% from 7 days after 2nd dose to median 4 months." This is not a false statement. It is tokenism.

To see tokenism clearly, I find it useful to speak facetiously. If I find 2,000 cadavers and inject half of them with the vaccine, and then add this group to my dataset of 44,000 living people, now I can say my vaccine works for both living and dead!

***

In a previous post, I explain why the headline VE result "from 7 days after 2nd dose to up to six months after 2nd dose" is better interpreted as "up to about 4 months after 2nd dose". This is another example of tokenism.

The published statement paints the picture that the researchers have data for participants up to six months after 2nd dose - one presumes that even if not all participants have six months of follow-up, most would have. Yet, inside the paper, they disclosed that only 7 percent of the evaluable participants have been followed up for at least six months, a sharp drop due to allowing placebo participants to get the vaccine earlier than planned.

Tokenism does not require lying. The businesses have hired the minority workers but the hiring has made barely a dent on the aggregate statistics.

***

Tokenism is hard to diagnose unless the consumer of the analytical finding has the time and desire to dig into the details of the study. An outcome for >= 12-year-olds on top of an impressive outcome for >=16-year-olds suggests the new study contained a sufficiently large sample of 12-15 year olds. A follow-up period described as "up to six months" suggests that most of the participants have been observed for about six months.

In the same way, a picture of an outing with a couple of token women signals that the company promotes workforce diversity.

 

 

 

 

 

Statistical statements turn on a few words

In the previous post, I started to review the newly released study about the "6 month" efficacy and safety of the Pfizer-Biontech vaccine.

This post examines the central efficacy claim of the paper. Its interpretation turns on a few words.

The paper starts with the finding that "VE against COVID-19 was 91% ... through up to 6 months of follow-up among evaluable participants..." and ends with the claim that "the data in this report demonstrate that BNT162b2 [Ed: the name of the Pfizer vaccine] prevents COVID-19 effectively for up to 6 months post dose 2..."

On the surface, this sounds like a straightforward extension of the interim analysis published in December 2020 that led to the emergency use approval of the Pfizer-Biontech vaccine. That study generated the now-famous VE of 96%. At the time of the study, roughly (but not quite) 50% of the participants have been tracked for two months after their second doses. Fast forward four months. Using exactly the same methodology, pre-specified in the trial protocol, the scientists updated the VE number "up to 6 months of follow-up", resulting in a revised VE of 91%, so there have been negligible attenuation of the vaccine's power.

That's not quite 2what happened, though. Strictly speaking, the quoted statements describe the collected data. However, the meaning of these statements turn on a few words. I'll first focus on why they said "up to 6 months" rather than "6 months".

(In a related post, I explain why when I read journal papers, I don't make assumptions of what the researchers may or may not have done. I focus on what they commit to paper.)

***

This headline result is on line 4 of the following table:

The famous VE of 96% was based on a case counting window that runs from 7 days after second dose (2D+7) to mid-November, when just under half of the study population have reached two months post second dose. Recall the train metaphor from the last post: the participants are seated in different carriages based on the dates of enrollment (embarkment).

So what's new in this study? The head of the train has moved from 2 months to 6 months of follow-up while the rear of the train also moved 4 months forward so almost all participants should have reached the 4-month mark, meaning their case history contributes to the case rate from 2D+7 up to 2D+4 months.

Given the focus on the "6 month" point by the investigators, I was expecting to find the words six months in this table. They are not there. The closest thing we can see is >= 4 months, which is the last line of the above table.

The reason why six months can't be found is that a proper six-month efficacy analysis has become impossible when the FDA allowed the vaccine developers to offer the vaccine to everyone in the placebo group. In the paper, this is known as the "vaccine transition option". Between December 2020 and March 2021, all but ~2,000 participants jumped off the train... before they arrived at the six-month marker. Once they are off the train, there is no re-boarding so no further updates on "long term efficacy" is possible.

***

The paper glosses over how the vaccine transition option totally changed the game. I'll now fill in the details. The following chart shows how the transition option caused follow-up times to be cut short across the board.

Without the option, as the Pfizer trial was originally designed, we should have 55 percent of the participants reaching 6 months or longer after second dose by March 2021, with all but two thousand odd people exceeding 4 months of follow-up. We should have had solid evidence on efficacy "up to 6 months".

With the transition option, the train was stopped, and almost all of the 44,000 passengers exited. What's more, the conductor instructed passengers to use exits at the back of the train as the doors near the front would not open. As a result, the proportion with at least 6 months of follow-up crashed from 55% to 7% while the number with fewer than 4 months of follow-up doubled. The vast majority of participants (about 65%) left the train with >=4 and <5 months of follow-up.

What the investigators meant by "up to 6 months" is really "four to seven months," with 80% of "evaluable participants" with fewer than 5 months of follow-up time. The last row in that VE table labeled ">= 4 months" is really better described as "4 to <5 months".

***

Those who decided to stay on the train for 6 months or longer are not your regular passengers - they consist solely of people who do not want to know if they were given the vaccine or the placebo, and so for some reason, they don't care if they are inoculated with a 96%-efficacious vaccine during a deepening pandemic. We don't know the psychology of those individuals but we are pretty sure they are not the average person.

This is another reason why the numbers on the right side of the case curve (>= 5 months) are neither reliable nor valid. If they show the error bars around those rates, they will be huge. If they conduct a survey, they will possibly learn the reasons why most exited the train but a few remained on it.

***

Elsewhere in the paper, the case-counting window was described as "from 4 months to the data cut-off".

If this were an ordinary clinical trial, these words suggest that the data were censored at the time of the interim analysis. Censoring is a technical term that signals the curtailment of the follow-up period. Common reasons for censoring include the participant reaching an endpoint (once someone gets infected, the outcome is known, removing the need to keep testing the person for infection), or the participant has dropped out or gone missing.

A different type of censoring is triggered by performing an interim analysis. At the full analysis, each participant has amassed the required follow-up time and every participant can be analyzed for the identical amount of follow-up. At interim analysis, the train is stopped at the half way point, so that the early enrollees have been observed for more time than the later enrollees. Every participant who has not previously been censored is now censored through an act of the investigators'.

Interim analyses have become common in clinical trials and so, censoring by "data cut-off" follows. But what happened in the Pfizer trial is not censoring by data cut-off. It is censoring by the vaccine transition option. Study participants censored themselves by exiting the train. This is a type of "informative censoring," which requires bias correction (not discussed in the paper).

March 13, 2021 was the data cut-off date for this study. In a standard interim analysis, censoring by data cut-off means anyone who hasn't been censored for some other reason will have their follow-up period stopped on March 13. When the censoring is due to the vaccine transition option, each individual has a different end date - the date on which they accepted the transition option. Almost everyone who accepted did so before March 13. Thousands exited in December 2020 alone.

The use of "4 months to data cut-off" creates the impression that the censoring occurs just like any interim analysis that stops follow-up on data cut-off. In reality, almost nobody was censored on data cut-off date; they were censored before data cut-off date, when they were told whether they received the vaccine or placebo.

***

In this post, I covered two examples of statistical statements that turn on a few words. "Up to 6 months" and "to data cut-off" don't mean what we might think they mean. That's because the vaccine transition option was not anticipated or pre-specified.

In summary, the six-month study is primarily about the 4th month after 2nd dose. There is some decline in efficacy but not large. The current estimate is less reliable than the previous analysis because the vaccine option curtailed follow-up time across all participants. Besides, we are unable to learn about efficacy beyond four months - that is the cost of giving the placebo participants the vaccine. I also have doubt that the people who stayed on board are representative of the average participant.

 

P.S. [8-9-2021] Here are my other posts about this study: post 1, post 3

How to count the fare after letting most passengers leave the train

We don't live in normal times and so the vaccine trials have been accelerated in order to deliver the promise of stemming the pandemic.

What does a normal clinical trial look like? In the most fundamental setting, you recruit participants, screen them for eligibility, randomly split them into two equal groups, give one group the medicine being tested, and the other group a placebo, wait a pre-determined follow-up time, and when the trial ends, you tally the outcomes in both groups and compare them.

In designing and running randomized controlled trials (RCTs), every effort is taken to make sure that the two groups of participants are identical and identically treated so that any difference in outcomes can be attributed solely to the medicine (vs placebo) and nothing else.

***

One way to accelerate trial results is to conduct an interim analysis. In the simplest setting, one conducts an analysis half way through the trial. This procedure assumes that the first half of the trial is predictive of the second half. It also assumes that the sequence of enrollment is random. 

Think of participants as boarding a train from the front to the rear. During a normal follow-up period, the train moves a fixed distance forward.  Passengers can only alight at a fixed disembark point on the platform, so at the end of the journey, the rear of the train has stopped there. In other words, every participant is tracked for the same amount of time, say one year.

For an interim analysis, the train is stopped midway to its destination. An inspector boards the train to check everyone's documents. The people at the front of the train has traversed a longer distance than the people at the back of the train. In statistical jargon, we say that the outcomes are "censored". 

At interim analysis of the vaccine trials, someone may have just received their first shot a week before the date of analysis, and so the investigators have followed them for one week only - and they can only contribute to the case counts up to 1D+7 days (7 days after first dose).

If you look at a cumulative case curve like this at interim analysis, imagine the front of the train running from left to right from Day 0 to Day 238. The interim analysis stops the train. The passengers at the front of the train have moved a further distance to the right than those who boarded last. The number of participants who can contribute to the case counts decreases as you go along the time axis from left to right.

As I have pointed out before (link 1, link 2), most vaccine trials have some form of staged enrollment, usually recruiting lower-risk participants first, and so the assumption of random sequencing of enrollment fails. These are the types of biases that appear when one deviates from the standard design. Investigators should correct for this and other biases.

At final analysis, all passengers have exited the train at the same disembark point. That means all participants have experienced the entire timeline. So the bias correction is only necessary at interim analysis but not at final analysis.  

A different adjustment - unrelated to bias - is that the bar for statistical significance is raised at interim analysis to account for additional uncertainty due to shorter follow-up. (Nevertheless, statistical significance has become collateral damage of the pandemic.)

***

That is a long setup for the topic of today's post, which is the "6-month" analysis of the Pfizer vaccine trial. A preprint of the scientific results has finally been published last week (here). The report is notable not just for what it says but what it doesn't say. My goal is to help fill out the details so you can digest the analysis with full context.

The first thing I noticed on the summary page of the report is that the investigators carefully inserted the words "up to", which includes the possibility that only a small proportion of people have reached 6 months of follow-up after the second dose. 

Enrollment in the Pfizer trial took place between end of July and end of October of 2020, and this interim "6 month" analysis froze the data on March 13, 2021. At this point, the head of the train is at about 7.5 months out while the rear of the train is at 4 months so we know for sure not everyone has reached 6 months of follow-up.

A key sentence in the paper discloses that about 55% of the participants have reached 6 months following their second doses: "During blinded and open-label periods combined, 55% of BNT162b2 [Ed: name of the Pfizer vaccine] recipients had ≥6 months follow-up post-dose 2."

It's important to realize that the above sentence did not state simply that "55% of trial participants had ≥6 months follow-up post-dose 2." The wording makes all the difference!

They didn't say the simple sentence because it's not true. The statement would have applied under normal circumstances but we live in extraordinary times. Since December, when Pfizer obtained EUA for the vaccine, a steady stream of placebo participants have been given the vaccine - they have effectively disembarked from the train before reaching the destination (link). By Pfizer's own publicity (link), they aimed to offer all placebo participants the vaccine by March 1, two weeks before the analysis date. 

As a result of the early-than-planned exits, only a fraction of the original placebo arm had ≥6 months follow-up post-dose 2. How many remained? How about 6%? This fact was disclosed in the adjoining sentence: "6% of placebo recipients had ≥6 months follow-up post-dose 2".

This is the data challenge that the analysts faced due to the decision to dismantle the placebo group. By design, 55% of the placebo group should have had 6 months of follow-up by March 13 but because of the "vaccine transition option", the number reaching that goalpost shrank from 55% to 6%... from about ~10,000 to ~1,000. To use the terminology from above, 9,000 or so persons have been censored at earlier dates so they have zero impact on the later part of the case curve. The later part of the curve is where we find "long-term efficacy".

For the interim analysis published in December, 2020, the FDA required a median follow-up period of two months following the second dose. Fast forward four months, and the middle of this train should have reached six months, triggering the six-month update. And yet, this is just theory that ignores the vaccine transition plan for the placebo group. In reality, the six-month mark is at the 6th percentile of follow-up times, not the 50th percentile. Nevertheless, passengers who left (and received the vaccine) are not allowed to re-board the train, and thus conducting the analysis later will make no difference.

Tellingly, in the VE table (Figure 2), the last line computes VE >= 4 months after dose 2. That's strange when the study is used to confirm "six month safety and efficacy." 

***

As an analyst, the crisis you're facing is that the vaccine transition option effectively destroys the careful RCT design, turning the data into observational ("real-world") data. This means the key to success is bias identification and correction. 

Unfortunately, that's not the path taken. The paper makes no mention of how many placebo participants chose to receive the vaccine, and when they did so. Timing matters a lot. It does not provide background statistics comparing those who transitioned, and those who didn't, nor any differences between those who transitioned earlier relative to those who transitioned later.

What the investigators did instead is to devise a mechanism by which they subtracted people from the vaccine arm to match the censoring of the placebo arm. The paper spent almost no time on analytical methods. We do have this sentence as a clue: "During the blinded period, 51% of participants in each group had 4 to <6 months of follow-up post dose 2; 8% of BNT162b2 recipients and 6% of placebo recipients had ≥6 months follow-up post-dose 2."

For placebo participants, the blinded period is before they were told they got saline shots during the trial and given the vaccine. The people in the vaccine arm were apparently also sent the notices (this wasn't described in the paper), and most of them responded to learn they were vaccinated.

The investigators then censored everyone in the vaccine arm who responded, meaning they de-boarded the train. That is the reason why the proportion of people in the vaccine arm with over 6 months follow up also plunged - from 55% to 8%.

The massive exodus from both arms of the trial appeared numerically similar but the reasons for exiting were night and day. Pfizer said it would be unethical to hold out their 95% efficacious vaccine from the placebo participants for the sake of science. In order to give them the vaccine, they must first be unblinded, i.e. told that they were assigned the placebo shots before. This reason does not apply to the vaccine-arm people as they have already taken both shots.

With the unblinding of the vaccine arm, the researchers threw out a lot of fine data. Instead of estimating the 6-month case rate of the vaccine arm using ~10,000 people, they did so with 10 times fewer participants. Students learn in Stat 101 that sampling error is a function of the square root of the sample size: to halve the error, you need 4 times the sample size. Reverse this. If you reduce the sample to a quarter of its size, you double the error. If you reduce it 9 times, you triple the error. That's one way to quantify the price of the vaccine transition plan. (The dismantling of the placebo arm is inevitable but it is the investigators' choice to inflict it also on the vaccine arm.)

I'd have taken a different path. The VE is based on a ratio of the case rates of the vaccine arm and the placebo arm. By preserving the original sample on the vaccine arm, I would have one good rate estimate (vaccine arm) and one bad estimate (placebo arm). In their analysis, they used two bad estimates. 

***

The challenges of observational data do not end there. It's not just about the size of the sample. It's also about the quality.

The participants who stayed on the train are unlikely to be a random sample of the original population. What would you choose to do if you were given a chance to confirm that you have been given the vaccine, and if not, to receive a vaccine that attained a 95% efficacy in the middle of a worsening pandemic? I can only guess at why a small number of participants decided not to unblind themselves. Maybe they already figured out their status. Maybe they think they don't need a vaccination, perhaps because they have lower risk, or they don't care if they are vaccinated or not. No matter what, these are the people who populate the right side of those cumulative case curves which speak to long-term efficacy. It's hard to believe that the ones who stayed on the train are identical to those who left.

I have more to say about this study. The above provides a basic understanding of why this is no ordinary study.

P.S. [8/9/2021] Here are the other posts: post 2, post 3

Speaking in French

With a bit of journalistic magic, I spoke French last week and answered a few questions about vaccine studies. The original piece is here.

***

Here are my answers in English:

What happened to the "real-world studies" arguing that vaccines are almost 100% effective?

To speak plainly, the real world studies have overstated vaccine effectiveness by a lot. After the pandemic, a post-mortem will reveal many methodological problems that are easy to spot, but in the current climate, there is no room - neither in the scientific community nor the public - for such a discussion to take place. 

Real-world data are messy, and their analyses require prudent splicing of scientific principles and subjective judgment. To give you an idea, a recently popular technique is called a "test-negative case-control design". Under ideal situations, it should give a reasonable estimate of vaccine effectiveness. The entire edifice relies on tracking people who have come forward to get tested for Covid-19. In the real world, though, some people had a series of test results, which may include positives as well as negatives, and each investigator decides on the rules for which test to count. Moreover, test results are not 100% accurate. We also don't have evidence that the vaccine is identically effective for any arbitrary subset of people.

In the past few months, the methods used in real-world studies have evolved. The first studies deployed well-established, complex statistical methods - such as matching and regression adjustment - that correct for some of the obvious biases because people who are vaccinated differ from people who are unvaccinated in many ways, such as age and ethnicity. More recent studies have largely abandoned those careful approaches, and pushed naive, entirely inappropriate analyses that compare the unadjusted case rates of vaccinated and unvaccinated populations.

As I explained in a recent blog post, this naive methodology commits many errors, such as counting infected people who have taken two mRNA doses as "unvaccinated" cases because they were infected before 14 days after the second dose. This case-counting window, typically set to start 14 days after the second dose, creates a lot of analytical problems. Using this window, analysts remove cases from the vaccinated group but they cannot symmetrically apply the same rule to unvaccinated people, because they did not get any injections, and thus the day of the second shot loses meaning. This is a key difference between the clinical trials and real-world studies. In the trials, unvaccinated people receive placebo shots, and so we know when they reach 14 days after the second shot.

For this and various other reasons, the real-world data must be corrected to get a good estimate of effectiveness. The more recent studies have failed because the analysts have abandoned any pretense to deal with real-world complications. The methods they use are designed for analyzing clinical trial data, which do not have the messiness of real-world data.

 

These studies have been widely reported in the media and by policy makers. Were we too hasty? Was there a temptation to exaggerate the effectiveness of vaccines "for a good cause"?

The analyses and reports were too hasty but it is understandable due to the evolving emergency. In my book Numbersense, I worried about a Dark Age of science arriving on the back of "big data". With the widespread availability of datasets, and the access to tools, it has become too easy to develop analyses and cram them into science journals. Hundreds of new research papers appear each month on Medxriv. Neither journalists nor citizens like me have enough time to read every study carefully and judge their merits. These papers contain incomplete data and simplified descriptions of the methodologies, making it very hard to follow. Many of my blogs about specific research papers require weeks of preparation.

The reporting in the media has gotten sloppy. How many times have you seen the vaccine trials referred to as "double-blind"? If you pull out any of the study protocols (Pfizer, Moderna, AstraZeneca, etc.), and just read the first page, you'll learn that the vast majority of them were "single-blind". Much of the coverage also ignores the type of research methodology. I've seen reports in which a result coming from a clinical trial with 40,000 participants is mentioned in the same paragraph as another result from a lab experiment with fewer than 100 samples. We can't even get the basic facts right, let alone the results, which demand interpretation.

This state of affairs is unlikely to change because a virtuous cycle has emerged. Media reports love stories about how "perfect" the mRNA vaccines are. Researchers who deliver confirmatory evidence get the headlines and the glory.

Public relations professionals seem to believe that the only way to convince 80 percent of the world to get vaccinated is to paint a black-and-white picture, a strategy that has the ends justifying the means. Among the "truths" they claimed in recent months included 90% protection against symptomatic cases, even higher protection against hospitalizations, and 100% protection against transmission and spread. As I documented in another recent blog, each of these claims has been dented by reality. Unfortunately, this has not resulted in a more nuanced perspective. They simply retreat and replace the debunked claims with new black-and-white truths. I don't want to second-guess the PR industry, as they are the experts in their field.

 

By trying to do too much (even if it means putting science on the back burner), don't we risk fuelling mistrust of vaccines?

 

I hope the lesson people take from the erroneous analyses isn't mistrust of vaccines or scientists but a greater appreciation that science is an enormously useful yet imperfect tool. The key message in my books about statistical thinking is always this: as data quietly exert greater influence over more aspects of our lives, so we must become smarter consumers of data analyses.

 

As with everything about the pandemic, the debate over vaccines has been politicized and turned into a black-or-white affair. There are three aspects to this debate I'd like to see covered more: the differential risks faced by different segments of the population, the differential levels of risk tolerance of individuals, and the idea of the common good. The first two points suggest that there should be varying receptivity to vaccines, regardless of one's trust of science.

 

Every mitigation measure, whether vaccination, mask-wearing or social distancing, works only if citizens care for each other's wellbeing. As a statistician, I know that one will win the lottery, no matter how low the odds are, if one can amass enough tickets. Every infected person is a purchased lottery ticket, and the prize is the rare mutation that extends the pandemic. The fight against the novel coronavirus is a test of whether humans are capable of acting together for the common good.

 

 

***

 

I should have said "Every infected person purchases millions of lottery tickets."