Big Data, Plainly Spoken (aka Numbers Rule Your World)

Book review: Don't Trust Your Gut

Seth Stephens-Davidowitz has a new book out early this year, "Don't Trust Your Gut", which he kindly sent me for review. The book is Malcolm Gladwell meets Tim Ferriss - part counter intuition, part self help. Seth tackles big questions: how to find love? how to raise kids? how to get rich? how to be happier? He invariantly believes that big data reveal universal truths on such matters.

In 2013, I speculated about the challenges faced by big data analysts in this piece for Significance. Big Data has become a real thing in the decade since its publication. Seth's book interests me as a progress report on the state of "big data analytics".

If there is a common core of this style of analytics, as demonstrated by Seth's examples, I'd include the following elements:

a) Big Data is less defined by the amount of data, but by several characteristics I previously labeled OCCAM (see this post).

The data are typically collected by passive observation (e.g. tax records, dating app usage, artist exhibit schedules). Meaningful controls are absent (e.g. no non-app users, no failed artists). The dataset is believed to be complete. The data aren't specifically collected for the analysis (an important exception is the happiness data collected from apps for that specific purpose). Several datasets are merged to investigate correlations.

b) Much - though not all - of the analyses use the most rudimentary statistics, such as statistical averages.

This can be appropriate, if one insists one has all the data, or "essentially" all. An unstated axiom is that the sheer quantity of data crowds out any bias. This is not a new belief: as long as Google has existed, marketing analysts have always claimed that Google search data are fully representative of all searches since Google dominates the market. (Seth's previous book covers his analyses of Google search results.)

c) If the analyst incorporates model adjustments, these adjusted models are treated as full cures of all statistical concerns.

The last few chapters on activities that cause happiness or unhappiness report numerous results from adjusted models of underlying data collected from 60,000 users of specially designed mobile apps. The researchers broke down 3 million logged events by 40 activity types, hour of day, day of week, season of year, location, among other factors. For argument's sake, let's say the users came from 100 places, ignore demographic segmentation, and apply zero exclusions. Then, the 3 million points fell into 40*24*7*4*100 = 2.7 million cells... unevenly but if evenly, each cell has an average of 1.1 events. That means many cells contain zero events. By the magic of model adjustments, all cells have non-zero estimated activities and associated happiness quotients. The estimates in many cells reflect an underlying model that hasn't been confirmed with data - and the credibility of these estimates rests with the reader's trust in the model structure.

I observed a similar phenonmenon when reading the well-known observational studies of Covid-19 vaccine effectiveness. Many of these studies adjust for age, an obvious confounder. Having included the age term, which quite a few studies proclaimed to be non-significant, the researchers spoke as if their models are free of any age bias. Taking this line of logic further, such a belief implies that no one should bother with running expensive randomized clinical trials because a self-selected trial coupled with post-hoc regression adjustments is a perfect substitute!

d) A blurred line barely delineates using data as explanation and as prescription.

Take, for example, the revelation that people who own real estate businesses have the highest chance of being a top 0.1% earner in the U.S., relative to other industries. This descriptive statistic is turned into a life hack, that people who want to get rich should start real-estate businesses. Nevertheless, being able to explain past data is different from being able to predict the future.

A kind of feedback loop is also at play. The average odds of getting rich surely depends on how many people are entering the industry. So if people read Seth's book, and rush into real estate, the data may no longer rank this industry as best.

e) Most of the featured big-data research aim to discover universal truths that apply to everyone.

For example, an eye-opening chart in the book shows that women who were rated bottom of the barrel in looks have half the chance of getting a response in a dating app when they messaged men in the most attractive bucket... but the absolute response was still about 30%. This produces the advice to send more messages to presumably "unattainable" prospects.

Such a conclusion assumes that the least attractive women are identical to the average women on factors other than attractiveness. It's possible that such women who approach the most attractive-looking men have other desirable assets that the average woman does not possess.

It's an irony because with "big data", it should be possible to slice and dice the data into many more segments, moving away from the world of "universal truths," which are statistical averages, said differently. In the last chapter of the book, Seth describes one study that tackles this problem.

***

I wish Seth had devoted more space to explaining model adjustments and statistical issues, and discussing whether the remedies are sufficient. His target audience is clearly people with minimal mathematical training, and I understand his decision not to open such cans of worms. The references at the end should satisfy those interested in exploring further.

As with Gladwell, I recommend reading this genre with a critical eye. Think of these books as offering fodder to exercise your critical thinking. Don't Trust Your Gut is a light read, with some intriguing results of which I was not previously aware. I enjoyed the book, and have kept pages of notes about the materials. The above comments should give you a guide should you want to go deeper into the analytical issues.

I think there is a lot more that can be done with big data, we are just seeing the tip of the iceberg. So I agree with Seth that the potential is there. Seth is more optimistic about the current state than I am.

The Musk sampling plan, thought through

Elon Musk likes to stir things up, and last Friday he claimed Twitter exaggerated its user statistics by counting "spam" accounts. Of course, Musk has made a bid to buy the social-media company, and he may just be looking for a discount.

He posted a tweet asking for ideas of how to estimate the proportion of spam/fake accounts on Twitter.

Later he suggested "ignore the first 1,000 followers, then pick every 10th".

Let's play this out a bit. When I'm done, you'll realize why sampling is a profession.

***

Each twitter account has a user name (mine is @junkcharts), and no two users can have the same user name. Each user has two basic counts: followers and following. The typical user has more following than followers - that would be true of most content consumers. If one is a content producer, the opposite is hopefully true: more followers than following. Elon Musk, for example, has over 90 million followers but he only follows 100 or so accounts.

So what happens when the first 1,000 followers are ignored, as Musk stipulated? Presumably he believes the earlier followers of any account are unrepresentative. It's not clear this exclusion does what he intends. The exclusion rule won't affect his own account as 1,000 is a rounding error. The set of Musk followers - with or without the first 1,000 - is essentially the same.

Applying this exclusion to other people's accounts may be problematic because 1,000 followers is a pretty high bar. 2838, 70, 133, 564, 3432, 608, 16K, 10K, 982, 4648. Those are the follower counts for the first 10 accounts shown on Musk's feed. Half of those accounts have fewer than 1,000 followers. After removing the first 1,000 from these smaller accounts, there is no follower left to pick from. The next three accounts have between 3000-5000 followers; excluding a third to a fifth of their followers from sampling seems severe. Thus, the Musk sampling plan introduces a large-account bias.

***

The second step of the Musk sampling plan is described as pick every tenth follower. So, take Musk's 90 million followers minus the first 1,000, and we'll get a gigantic sample of 9 million.

Here comes the hard part. What he wants to know is the proportion of those 9 million accounts that are spam accounts. Who is going to decide whether each of the 9 million accounts are "spam", and how?

Nine million is too big a sample to handle. He may be applying the "rule of thumb" that a random sample should be 10% of the population. That's a myth busted in any intro Stats class. We typically only need to interview 1,000 Americans to generalize the sample responses to the entire population of 300 million, which is far, far, far, far smaller than 10%.

The required sample size is not a fixed number. If one can tolerate a larger margin of error, the sample can be reduced. What's our tolerable margin of error? If the true proportion of spam accounts is 5%, as Twitter management asserted, we may want a rather precise estimate, within plus or minus 2%. Reviewing Stats 101, we learn that translates to a standard error of 1%, and a sample size of 475.

What that means is if we take a random sample of 475 of Musk's followers, and learn that 5% (24) of those are spam accounts, then we can conclude that 5% of his followers are spam accounts, plus or minus 2%. (Others may disagree.)

It would be much easier to verify 500 accounts than 9 million.

***

Perhaps Musk didn't intend to pick every 10th follower until the list of 90 million is exhausted. What he might have in mind is to pick every 10th until the required sample size is reached. Let's say we want a sample of 500, so we stop the random sampling procedure after 500 picks, or having gone through 5,000 followers.

Can you see what's wrong with this? In effect, rather than being randomly picked from the entire list of followers, the 500 all came from a narrow slither of 5,000 followers, precisely followers #1001 to #6001 after expelling the first 1,000. But... the list of followers is ordered not at random, but chronologically. Thus, all 500 followers in the sample would have started following Musk during a specific narrow time window.

Musk appears to be imagining that he has a list starting with the earliest followers through to the most recent. (In practice, it may be easier to obtain the reverse chronological list starting with the most recent.)

***

Notice that the sampling plan involves two steps: first sample from all users, then within each selected user, sample from followers. As stated, the Musk plan is incomplete, as we have no idea what he plans to do for step 1. Aside from his own account, what other users does he include, and how many?

Even if the first step involves a random sample from all users, this unbiased sample is later tainted by the exclusion of the first 1,000 followers, because it flushes out small accounts from the sample.

The first step is unlikely to be a simple random sample; some intentional bias may be inserted. Like most social-media platforms, I'm surmising there is a highly skewed distribution of followers. Maybe 10 percent of the biggest accounts attract 90 percent of all followers. Thus, a simple random sample will contain lots of small accounts with few followers, not an ideal situation given the exclusion rule that subsequently flushes out small accounts. (Bias is not always a bad thing!)

As explained above, a sample size of 475 followers gives a good estimate of the proportion of spam accounts among Musk's followers; I'd be hesitant to generalize it to other twitter accounts. That's because Elon Musk's is no ordinary user.

Combining the spam proportion in Musk's account with similar in other selected accounts is tricky. If intentional bias is used in the first step, then a weighted average is required. Weighting does not cure the large-account bias though.

The two-step sampling seems unnecessary. Why not just sample users directly? After all, a follower must be a user.

***

The above considerations form a starter's list.

Here are further topics:

• If one wants a random sample, then every user should have equal probability of being picked. Such a sample is likely to contain a majority of small accounts.
• Some users are lurkers. They have few followers.
• Some accounts have been abandoned but not deleted.
• Newer accounts have fewer followers and following.
• One person can have multiple twitter accounts. Spammers probably have a higher average number of accounts than humans
• The more accounts a user follow, the more likely the user will appear in the random samples.
• A follower may be one of the first 1,000 followers of Elon Musk and thus excluded but the same person will be post-1000th followers of other accounts and included if randomly selected.
• Are all bot accounts "spam"? What is the definition of "spam"?
• All social media have tons of spam accounts. Is it better to answer the inverse question of how many humans use Twitter?

Reading Statistical Analytics Illustrated

A few months ago, Prof. Kottemann sent me a copy of his new book, "Statistical Analysis Illustrated: Foundations You Should Know" (link). On a recent flight, I finally found time to give it a quick read.

The book covers a central section of frequentist statistical theory, which forms the core of most traditional Statistics 101 courses. This is the sequence that runs through random sampling, sampling distributions, law of large numbers, central limit theory, confidence intervals, and hypothesis testing (tests of significance).

Kottemann adopts an approach that deemphasizes mathematical derivation, and delists any theorem while favoring verbal and visual explanation. This is an approach that I also prefer in introductory statistics courses so I flip the pages eagerly.

With the stated goal of helping readers appreciate a "way of thinking" about data, Kottemann improves upon most textbooks in the following manner:

1. He concentrates on the case of sample proportions (e.g. x% of respondents felt this way), which can be easily appreciated given their common usage in surveys. This use case avoids one of the tricky turns that confuse statistics students, namely, learning the difference between the population standard deviation, the sample standard deviation and the standard error.
2. He smartly abandons the usual bottoms-up presentation of the central sequence. In the first chapter, he goes straight into sampling distributions, confidence intervals and significance testing without having defined a random variable, a normal distribution, the law of large numbers, the standard error or the null hypothesis (all of which will appear in later chapters).
   
   
3. The visuals in the book are improved versions of typical textbook diagrams as he removed unnecessary and confusing details.

This central sequence of frequentist statistics has proven challenging to generations of students because it's not a simple, straightforward argument. It's not easy because generalizing from samples of data is a complex undertaking. Kottemann's scope is narrower than the typical Statistics 101 textbook but what he covers is essential.

Most of the writing by Kotteman is lucid. The complexity of the materials - when explained in words - only got the better of him on a few occasions. Above Figure 1.1, he writes "[This histogram] shows the frequency (out of 1,000) with which we should expect to get random samples that yield the various possible values for the percent-in-favor sample statistic." I quote it only to illustrate the type of tongue-twisters statistics teachers run into when covering this material (and so he may rephrase it in the next edition); the sentence is an outlier in a very readable book.

The hypothesis testing framework has been under constant attack from other "ways of thinking". In the last chapter of the book, Kottemann briefly covers the Bayesian way of thinking. I hope readers appreciate that everything is different, from which question is answered to what assumptions are made to how decisions are framed.

The frequentist viewpoint has been widely abused. Journal editors reject any studies that do not reach a 95% confidence level, effectively ordaining publication bias, and motivating scholars to "p-hack," which is a general term that means engineering the "p-value" to hit 95%. In practice, people may "d-hack" which is to review a large array of outcomes, and select the one that happens to reach 95%; or they may "n-hack," which is to make the sample size so big that even tiny signals are declared statistically significant. I might even add "ps-hack" which is to pre-specify a large array of outcomes to cover up "d-hacking". 

If there is one gap in Kotteman's presentation, it's in the transition from the case of proportions to "the rest of the (frequentist) iceberg". If the reader is asked to do a test of difference in means at the 10% significance level, I'm not sure s/he can figure it out.

I'd also like to see a few worked-out examples of analyzing real-life opinion polls or market research surveys, and some exercises.

If you're learning statistics, and having trouble with the central sequence from random sampling to statistical testing, or if your sole interest is understanding polls or surveys, give Kottemann's book a try. It may help unblock you.

More case-counting puzzles

The Johnson & Johnson vaccine is being treated as a third-class citizen due to some safety concerns (link). Interestingly, such concerns did not surface in the "final" analysis of the single-shot Covid-19 vaccine, published in February, 2022 in New England Journal of Medicine (link). In this report, the scientists concluded "Ad26.COV2.S was associated with mainly mild-to-moderate adverse events, and no new safety concerns were identified". Since in the interim report, the FDA said "the analysis supported a favorable safety profile with no specific safety concerns," we were supposed to believe that the J&J vaccine is completely safe.

On the efficacy side, the final report headlined 56%. This number is meaningless unless we know how VE is defined. The 56% number comes from a counting window that starts 14 days after the shot (1D+14). During the interim report, the headline number was 67% (link).

However, these numbers are not comparable for reasons I'll get into in this post. The FDA has allowed pharmas to keep shifting the goal posts so nothing can really be compared like for like. The situation illustrates clearly why statisticians won't give you an answer unless we know all the details about how things have been measured!

***

First, the interim report was conducted when only about half of the participants have reached 2 months of follow-up - note that the largest number on the time axis was 126 days.

In the final analysis, about half the participants have reached 6 months of follow-up - the largest number on the time axis is now about 280.

So, the comparable value from the final report is not the headline VE which concerns up to 280 days of follow-up but the VE at day 126... and that number is 55% (1 - (436-100)/(938-185)). According to Table 2B (final report), the error bar around this number is 20% to 70% so the trial did not enroll enough participants to give a precise estimate of VE.

Harmonizing the follow-up period is not sufficient. Between the initial and final analyses, J&J ditched the requirement that all cases be "centrally confirmed". In the interim report, cases were counted only if a central laboratory confirmed the local PCR test positives. In the final report, cases accrued based on local test results.

***

Another major development is letting the placebo group get vaccinated. After J&J's vaccine was granted EUA at the end of February 2021, all participants reaching 6 months of follow-up were effectively unblinded so that those in the placebo group could get the vaccine if they so chose. Almost everyone did.

Since enrollment ended October 2020, and the data cutoff for the final analysis occurred in July 2021, everyone could have been followed for at least 6 months, if the above plan were implemented to the letter.

But that was not what happened. Some participants were unblinded prior to reaching the 6-month point because they requested getting one of the by-then authorized Covid-19 vaccines.

As a result, only 23% of the study participants had an observation window of at least 6 months. (By design, 100% should.) In fact, half of the participants in this trial had a follow-up period of less than four months. Because the follow-up period was truncated by the placebo participants receiving the vaccination, effectively destroying the placebo group, there will be no further useful data.

Refer back to the case curve shown above, notice that the at-risk population has dropped by more than half by day 126. The further out one went in time, the fewer participants contributed to the incremental case numbers, the wider the error bar. Such dropoff does not happen in a normal clinical trial. 

Those who reached the longest follow-up period are also the earliest enrollees in the trial. In most vaccine trials, the pharmas first enrolled the lowest risk people - in the case of J&J, the earliest batch of enrollees were young people with no comorbidities. So the right hand side of the case curve lacks both validity and reliability.

***

When I wrote about the J&J trial, I pointed out that it was one of the few double-blind trials - the Pfizer and Moderna trials were single-blind. Blinding removes a source of potential bias - a kind of self-fulfilling prophecy in which the investigators or the participants behave in a way that advantages the treatment against the placebo.

We now have solid evidence that blinding did not work well, and confirmation bias was a real concern. To begin with, given that about 30% of Americans have obstinately refused to get the vaccine, we know that this subset is not represented in these vaccine trials. When the placebo groups in these trials were offered the vaccine, almost all of the participants took advantage, implying they were vaccine enthusiasts, not skeptics.

Blinding is supposed to neutralize partisan feelings about vaccines but it didn't work as planned. In Figure S1 (in the appendix), we learned that 1,222 placebo participants "received other vaccine before unblinding", compared to 670 on the vaccine arm. That's 80% more on the placebo arm than the vaccine arm. If the treatment groups were properly randomized, and if participants were properly blinded, they would not know whether they received the placebo, and so the proportion of people who got another vaccine against trial rules should be balanced.

The number of participants who withdrew during the trial was also disproportionately higher on the placebo arm (338 vs 224), about 50% more. These are huge gaps.

***

One of the big mysteries of Covid-19 vaccine trials is what counts as a case. The case-counting window is one variable I've discussed often. But there's more. Most of these early trials were designed to measure symptomatic, PCR-confirmed cases. J&J added a definition of mild, moderate and severe cases, although as I pointed out before, the definition of "mild" disease is meaningless since almost every case has been classified as moderate or severe.

In most studies, including the J&J trial, all participants are given serological tests at regular time intervals. Seropositivity indicates that someone has been infected in the past, and thus, it is presumed that said infections were asymptomatic. This device has been used by scientists to claim the vaccines also prevented asymptomatic infections.

Apparently, J&J now believes that someone without a positive result from either PCR or serological testing can also be infected. This is how I interpret one of the footnotes from Table 1 that corresponds to a concept called "any SARS-CoV-2 infection", which includes:

"undetected cases that were subsequently detected through a positive serological result (according to the clinical severity adjudication committee), which did not count as either symptomatic cases (becqause they were RT-PCR-negative) or asymptomatic cases"

I'm not sure how the adjudicators got wind of the "positive serological result" since if such a result was present, the case should have counted as "asymptomatic", which is a different line in Table 1.

(Also, the severity adjudication committee is a misnomer as the committe members concerned themselves here with asymptomatic infections that would be milder than "mild".)

What is shocking about this line of numbers is the magnitude. There were 1,038 such infections in the vaccine group and 1,699 in the placebo group. These two numbers are much higher than the counts that made it to the headline VE calculation. In particular, only 42% of the vaccine infections, and only 52% of the placebo infections, ended up as "cases". This calculation also shows that an "any" infection on the placebo group has a higher chance of being counted as a case than an "any" infection on the vaccine group.

 

 

 

 

 

 

 

 

 

Reading Gelman's take on election polling

The spector of Covid-19 has pushed U.S. elections off the headlines. Nonetheless, political scientist Andrew Gelman has just offered his take on the state of election polling (link to PDF). This article is perfect for those of us who have a keen interest in but no deep knowledge about polling as Andrew covers how such polls are conducted. Andrew, by the way, was the brain behind the election forecasts by the Economist in the 2020 U.S. presidential elections so he's not merely lecturing from the pulpit.

If you've followed recent U.S. elections, polling inaccuracy has been one of the popular themes. Funny enough, before Trump's stunning victory over Hillary Clinton, the media was trumpeting the exact opposite, how election polls were so damn accurate. Those were the glory days of Nate Silver. Recently, the narrative has flip-flopped: how pollsters could have given Clinton 90-99.9% chance of winning, how polls over-estimated Biden's victory margin, etc.

The following are notes from reading Andrew's article.

***

p. 1

Pollsters are aware that "recent pre-election polling errors have been an underrepresentation of choices favored by lower-education white voters" but "adjusting for ethnicity and education... did not solve the problem... this past November".

This gets at something important: making "adjustments" is necessary but may not be sufficient to make the problem go away. This is a very common issues I see all over the place. One has an observational dataset on people. One then shoves a pile of demographic data (age, sex, income groups, etc.) into the statisticial model, now known as the adjusted model, and voila, heaven opens up and all our sorrows have gone away! If someone asks about biases in the observational dataset, one points to the adjustment factors, and moves on. An analogous situation is peer review in a published journal. Is the research finding credible? In some corners, if it is published in a peer-reviewed journal, there should be no further questions. Reality is much more complicated. Are the right variables in the adjustment? Are the variables structured in the right way? Are there confounding variables that haven't been measured?

p. 2

the moment of truth when poll-based forecasts are compared to election outcomes.

The election prediction problem is unusual in that there is an inescapable, quick, definitive evaluation of the pollster's performance. Election forecasting is the most accountable type of data science. Most data science problems do not have this property. Sometimes, only certain types of errors are visible. In Chapter 4 of Numbers Rule Your World (link), I discussed the anti-doping labs: their false negative errors are unlikely to be discovered, while false positive errors elicit immediate outcry from the harmed athletes.

The key challenges [of polling] are (a) attaining a representative sample of potential voters, and (b) predicting turnout.

The progression is from eligible voters to potential voters to actual voters. There isn't much uncertainty in eligible voters, nor actual voters once the election is over. But turnout is uncertain, which is the link between potential voters and actual ones. Eligible voters are not usually a representative sample from the underlying population.

We cannot expect to eliminate non-sampling errors, because conditions for new elections are always changing. Unlike sampling errors, non-sampling error cannot be reduced simply by conducting more and larger surveys.

The last part debunks a Big Data myth. Collecting more data can't solve all problems. It reduces sampling error but does not eliminate non-sampling errors. Another way to think of non-sampling errors is that the people in the sample aren't representative of those in the population. It is helpful to extend our progression: eligible voters -> potential voters -> surveyed voters -> survey respondents -> actual voters. There are two sources of inaccuracy here: some actual voters may not be surveyed, or if they received the survey, may not have responded.

p. 5

If we are estimating opinion in 4 age categories, 2 categories of sex, 4 ethnicity categories, 5 education levels, and 50 states, that’s 8000 cells. Even if you are analyzing a set of polls with a total of 20,000 respondents, this will still leave with many cells with 0, 1, or 2 respondents, hardly enough to get any sort of estimate.

Each "cell" is a demographic subgroup. Polls rarely have 20,000 respondents so this problem is clearly pervasive. Andrew's statistical approach pools data from "similar" cells so there are enough respondents to come up with an estimate. Such a model is, however, vulnerable to being wrong about "similarity". Nevertheless, any method will be vulnerable to one issue or another because fundamentally, the available data are incomplete. You can, for example, produce just 8 cells based on age and sex alone, and then in each cell, you have enough respondents; but this model misses the effect of all the other variables, and therefore is inaccurate in a different way.

The adjustments only adjust for the variables included in the model.

The beauty of these models is that they are not black boxes. We know exactly what factors are being adjusted for, and the functional form. But if the form is incorrect, the adjusted model will be inaccurate. This points back to the point I made above: be skeptical when someone claims to have solved all biases by throwing a bunch of standard demographic variables into a model - be doubly skeptical if they only include one factor at a time, or only "main effects" - be triply skeptical if the outcome is affected by non-demographic factors, such as behavior.

 

p. 8 He goes into the crux of the matter:

Why were the polls as imprecise as they were? Some possible explanations are differential nonresponse, differential turnout, changes in opinion, and insincere survey responses. These last two explanations, which can also be phrased as “last-minute swings” and “shy Trump voters,” are natural explanations, but I doubt they are a major part of the story in 2020.

By "shy Trump voters", Andrew specifically meant Trump voters who lied to pollsters and claimed they would vote Democratic. His view that this was a non-factor appears to be commonly held among the political science community. I wonder if there is another kind of shy voter who appears under the heading of "differential nonresponse". What if Trump voters were systematically less likely to respond to polls?

p. 9

Some polls also adjust for partisanship, using recorded party registration, stated party identification, or stated vote in the previous election. But even the surveys that made all these adjustments were off by about 2 percentage points, on average.

There is a philosophical question in there. The residual inaccuracy can be either evidence of inadequate adjustment or of partisan bias being unimportant.

p. 10

An error of 2 or 3 percentage points is a problem for predicting very close elections but otherwise is not so consequential.

This is an important observation. The closeness of recent presidential elections makes the error bar a problem.

p. 11

He turns this around and praises the unreasonable accuracy of polls. Think about it. Surveying only some thousands of people among a population of 300 milllion gets the estimates down to a margin of error of only a few percentage points.

p. 13

In retrospect I wish that we’d expressed our 2020 Economist forecast conditionally: instead of simply stating a forecast interval and probability of each candidate winning, we could’ve graphed this interval as a function of the national polling error, thus indicating the confidence we had in our forecast at different levels of survey accuracy and making this dependence clear.

Good idea. Not sure if the public will appreciate it.

well-funded campaigns and advocacy groups... can do more effective survey adjustment using the voter file, which has information including past turnout history on nearly 200 million Americans.

I wonder how much this costs. Sounds like it could be a fortune.

it would be good to see less focus on political campaigns and more on surveys of attitudes

I really like this suggestion. I don't understand the value of election forecasting (except for the campaigns themselves).

 

The Merck Covid-19 pill: is the result statistically significant?

Two weeks ago, the FDA advisory committee narrowly voted to recommend authorizing the sale of Merck's Covid-19 pill (Molnupiravir or MOV). This is a rich story worthy of a number of posts. We start with the final result and work our way backwards.

***

Molnupiravir is said to be effective at reducing the risk of hospitalization or death of Covid-19-infected people at high risk of developing severe disease. The treatment program spans five days, two 800-mg pills per day. The study was set up as a randomized clinical trial with ~1,550 participants, randomly divided into two halves, one taking MOV pills while the other ingesting placebo pills. The primary endpoint was the difference in event rate between the two arms. An event is either hospitalization or death. Each participant has been followed up for at least 28 days (or until an event occurred).

At the end of the study, the event rate was 10% on placebo and 7% on MOV. The difference in event rate is -3%, meaning that for every 100 people taking the pills, 3 fewer people will be hospitalized or dead, on average. For each person to benefit, about 33 must take the pills. (This number, known as NNT or numbers needed to treat, looks bad because 90% will not develop severe disease whether or not they take MOV.) Note also that for every 100, 7 will be hospitalized or dead despite taking MOV.

The above result has been reported as 30% efficacy, which is the relative event rate ratio (1-7%/10%). I prefer using differences as they don't exaggerate the impact off a small baseline. This primary endpoint is not statistically significant (but it's being portrayed as significant; I will get to this point later).

Molnupiravir may have very serious side effects. None of these effects have surfaced in the trial but the trial design contained so few participants that it can only find very common effects. Only about 700 people got treated with Merck's pills. An event with risk of 1 in 1,000 has almost 50% chance of not registering a single case in 700 people. An event with risk of 1 in 10,000 has over 90% chance of not showing up in this trial.

The FDA Briefing document that was reviewed at the FDA meeting on November 30, 2021 (link) describes several potential adverse side effects, all found in earlier animal and lab studies. These include bone marrow toxicity, mutagenecity, and reproductive toxicity.

***

How this type of treatment can be put to clinical practice should have been thought through before running this experiment, but evidently, it wasn't.

Rolling out the treatment requires an accurate prediction of who are at high risk of developing severe disease. Just like we don't know what level of antibodies is high enough to prevent infection, we don't know what factors predict disease severity. During the trial, Merck went with the usual list of "risk factors" for hospitalization or death, such as older age and obesity. This just means most patients who are hospitalized with or dead from Covid-19 are older and/or obese, but that's not the same as saying most older or obese patients will develop severe Covid-19.

What we do know is that the patients present in the Merck trial definitely had unusually high risk of being hospitalized or dying. The 10% baseline event rate meant that 1 out of every 10 Covid-19 patients enrolled in the trial ended up hospitalized or dead. This rate is over 100 times higher than the unvaccinated rate of hospitalization in the U.S., according to CDC numbers cited in the FDA briefing doc.

In real life, how does one find a comparable set of patients who would have the highest chance of benefiting from these pills?

**

The FDA document also discloses a number of other headaches - all of which are direct results of the design decision to exclude certain groups from the clinical trial, namely, vaccinated people, pregnant people and immunocompromised people.

The design decision is odd because vaccinated people constitute the vast majority of Americans, and the other two groups are apparently at higher risk of developing severe disease. Instead of oversampling them in the clinical trial in order to measure its effects, they were excluded completely from the trial. Strictly speaking, there is no evidence that these pills work for anyone who has had at least one dose of Covid-19 vaccine. This meant that the FDA advisors are asked to take part in a game of guessing without any actual data to support their opinions.

In the trial, those who were seropositive at baseline did not benefit at all from taking the Merck's pills (5 events vs 2 in placebo - nice example of why I dislike relative ratios), indicating that MOV might not work for vaccinated people who have antibodies. The FDA analysts cast doubt on this result, arguing the sample size was too small - and yet, they have no trouble saying that the pills appear to work equally well across age groups, and other variables - when these analyses contain similarly small samples. (e.g. 136 who were seropositive vs 118 who were > 60 years old)

Contrast statement (a): "Given the small size of the seropositive at baseline subgroup and the small number of events in this subgroup, robust conclusions regarding MOV efficacy in seropositive participants cannot be made"

with statement (b): "Efficacy results were generally consistent across subgroups including age (>60 years), at risk medical conditions (e.g., obesity, diabetes), baseline COVID-19 severity (mild, moderate) and SARS CoV-2 variants"

This is the kind of flexible interpretation of statistical significance that frustrates your statistics professor. You either believe none of the subgroup analyses should be considered because the trial was not designed to have enough samples to read them, or you believe they should all be trusted.

***

Another place in which statistical significance was mishandled was hinted at earlier. The main result is a 3% reduction in event rate, which has a 95% confidence interval of 0.1% to 5.9%. Notice that the CI apparently missed 0% by a pip - if the CI crosses 0%, the result is statistically insignificant, leading to the conclusion that the trial did not generate sufficient evidence to prove that MOV was effective at changing the event rate.

But is it barely significant or not significant? Here are two additional issues for your consideration:

What if there were one fewer hospitalization counted in the placebo group (67 instead of 68 events)? This tilts the confidence interval to the other side of zero, making the result statistically insignificant. Talk about hanging on a thread.

Now, in Table 1 (footnote) of the FDA Briefing Doc, we learn that "participants with unknown status at Day 29 are counted as having an outcome of all-cause hospitalization or death in efficacy analysis". Said in English, this means that anyone for whom they didn't know the status by Day 29, they assumed to have been hospitalized or have died. I highlighted this line in red because this seems out of the ordinary. Usually, we assume the opposite (presumed alive vs presumed "dead").

How many people fell into the bucket of unknown outcome, turned into hospitalized or dead? Exactly one, on the placebo arm. Thus, if they had made the opposite assumption, that the unknown person was not hospitalized or dead, then the statistical significance flips!

***

The above discussion of statistical significance assumes a normal analysis schedule in which analysts perform a full analysis when the entire study population has passed the full follow-up period of 28 days. That was not what happened.

In many clinical trials, researchers are allowed to peek at early results, known as interim analyses. In this Merck study, they performed an interim analysis at the half-way point, when half the participants have completed 28 days of follow-up. (This explains the two press releases from Merck about the same trial.)

A problem with peeking at interim results is that it increases the chance of false-positive readings (i.e. wrongly declaring significance). To compensate for this risk, the required level of significance must be reduced (more stringent). We encountered this issue before when looking at Moderna's vaccine clinical trial analysis plan. (Link)

Because of early peeking, the required significance level is not the standard 5% but lower than 5%. A different way of saying this is we shouldn't look at the 95% confidence interval but something higher than 95%, which means the width of the CI is larger than the reported 95% confidence interval. In other words, if the proper threshold was applied, the reported 3% reduction is not statistically significant.

They managed to come up with a justification to overturn the usual statistical practice: "Formal statistical testing was not performed for the full population assessment, because statistical significance was demonstrated at the interim population assessment. The nominal 1-sided p-value was 0.0218."

This is yet another statement that drives your statistics professor mad. Firstly, this statement literally embodies the statistical fallacy of "testing to significance", which I explained here. Secondly, even though they didn't do formal testing, they issued an "informal" testing result, showing a p-value of 0.02, suggesting that the testing result would be significant (under the conventional of p-value < 0.05 unadjusted for early peeking). Thirdly, they showed a one-sided p-value, which makes no sense in this situation; a one-sided analysis can only be justified if there is little doubt as to the direction of the outcome, and is almost never used in clinical trials. The "nominal" two-sided p-value is over 0.04, and likely over 0.05 when adjusted for early peeking.

The situation is worse when you compare what happened in the first and second halves of the Merck study:

There are rather disturbing signals on this table. The interim analysis showed an improvement of 7% over a placebo event rate of 14%. The final analysis showed the improvement to be 3% over a placebo rate of 10%. This meant that the Merck pills performed very badly in the second half of the participants: 6% event rate vs placebo event rate of 5%.

Thus, in this particular experiment, the first half of the result is contradicted by the second half. The FDA analysts endorsed Merck's argument to ignore the second half because "statistical significance was demonstrated at the interim population assessment".

If this were my experiment, I'd investigate whether there were operational issues (or enrollment biases) that caused the placebo event rate to go from 14% in the first half to 5% in the second half, a three-fold change. (By contrast, the event rate of the treatment arm only changed slightly.)

The phenomenon can just be regression to the mean, though. That's why classical statisticians don't like early-stopping strategies. If this trial was stopped for efficacy, and the second half was not analyzable, we would have over-stated the effect by double.

***

I've exceeded the usual post length again so further comments will appear in next posts.

 

 

The booster shot data reveal why RCTs are still the gold standard

Observational studies have some common characteristics that annoy the analyst. In the previous post about Pfizer's booster studies, I described a few.

• The sample contains few examples of the people to which you're generalizing your result (e.g. people 65 or older, high-risk individuals of any age).
• The measured outcome is not the obvious, direct outcome (e.g. clinical outcome of infection) but an indirect indicator (e.g. antibody levels).

These problems create the need to extrapolate results. Such extrapolation is usually supported by unverified assumptions (aka expert opinion). Of interest, the FDA allowed Pfizer to impose a causal assumption: that if the booster achieves similar levels of antibodies as the original 2nd dose, then the booster also produces similar clinical outcomes as the original two doses. See the previous post for why modifying an indirect outcome does not invariably lead to a change in the direct outcome. (This matter is yet another real-world example about correlation and causation.)

As I have explained often on this blog, making assumptions is not a sin. All statisticians make assumptions - those who claim they make no assumptions make assumptions! Let me give a quick example. Suppose a college runs a survey asking recent graduates about their post-graduation salaries. Only 50% of the graduates responded with a valid number. Should the college impute the salaries of the non-respondents? If I were the analyst, I would impute the salaries, baking in the assumption that those who don't respond are likely to be earning lower salaries (possibly even zero). However, some statisticians will argue against using imputation. They'll claim that one should just report the average of those who responded, because it is a horrible thing to modify the data - one should only correct egregious data entry mistakes but not otherwise touch the data. Are they making no assumptions? 

They are making a huge assumption: they assume that the non-respondents have the same average salary as the respondents. What is the basis for that assumption? There is none, other than the desire to not tamper with the dataset. In fact, this assumption is almost surely wrong. As I said in Numbersense (link),

Beware of those who don't tamper with the dataset!

***

In this post, I'll cover several additional extrapolations that were used in analyzing the booster data.

The key immunogenicity results all measure antibody levels against the original virus (so-called wild type) but as we've been constantly reminded, the only virus currently in circulation is said to be the Delta variant. How did Pfizer bridge this gap?

According to the FDA Briefing document (link to PDF), "Pfizer proposes to infer effectiveness of the booster dose against the Delta variant from exploratory descriptive analyses of 50% neutralizing antibody titers against this variant evaluated among subjects from the Phase 1 portion of the study." 

"Exploratory descriptive analysis" is not regarded as a serious form of scientific evidence - the rules have changed during this pandemic emergency. I pity the statisticians working on these studies. They were assigned an impossible task. Recall that the gold-standard randomized clinical trial (RCT) has been thrown overboard, replaced by "immunobridging" analysis. The antibody levels stimulated by the booster shot are deemed comparable to those registered (by other individuals) after their second shots, and then the "non-inferiority" in antibody levels is assumed to result in the "non-inferiority" in vaccine effectiveness (a clinical outcome not directly measured).

The Delta variant introduces another complication: as it was not present around the time of the original Pfizer adult trial, researchers do not know what the VE of the first two doses is against Delta. This breaks the causal assumption of Immunobridging.

We do not have even an approximation of the VE outcome against the Delta variant for the 300-odd people in the booster trial. So if one accepts that one can expose the old blood samples to the Delta variant in the lab, and show that the antibody levels are comparable, one has to make an even stronger assumption by claiming that those antibodies will result in the same VE as measured in the Pfizer adult trial.

This line of argument unfortunately results in a contradiction. The original vaccine either works just as well against Delta as it worked against the original strain; or it is less effective, which explains recent surge in cases. Only one of these scenarios can be true.

If the first scenario were true, then the immunobridging assumption holds as VE for Delta is the same as VE for the wild type, but in this scenario, the recent surge in cases has nothing to do with Delta, and there is no need for a booster - since VE is the same.

If the second scenario were true, then the booster may be necessary, but in this scenario, the immunobridging methodology is broken as the prior value of VE no longer applies.   

Since Pfizer applied for approval for the booster, it is assuming the second scenario. One might be tempted to prefer assumption #1 because it permits the immunobridging analysis. That is a popular justification you find in academic papers, in the same vein as "computational feasibility" or "tractability". It's risky to use this logic in studies with real-world consequences. Besides, it creates a contradiction.

Beyond the logic of the analysis method, notice that the Delta analysis is performed using only Phase 1 participants, so we are talking about 11 people between 18 and 55, and 12 people between 65 and 75 years old. I'm not sure why blood from the other 300 people cannot be similarly analyzed. Without those results, the immunobridging assumption is expanded yet again to generalize from the 23 people to the 300-plus people (then to the 20K-plus in the adult trial, and finally to the U.S. population.)

A proper RCT would have yielded a much cleaner analysis of the direct clinical outcome. 

***

Another missing piece of the puzzle is the provenance of the 300 or so Phase 2/3 trial participants who were "selected" to take part in the booster study. Nowhere in the FDA Briefing document can I find the selection mechanism. Usually, this means the selection was not random from the vaccine arm of the earlier trial. If it is not random, what are the criteria? 

The 300 people reduced to about 200 that ultimately were used in any of the primary calculations. That's a drop-off of a third of the starting population, which is a high drop-off rate. (Note: Table 3 claims the evaluable immunogenicity population is 268 but the results shown in Tables 4 and 5 unexpectedly have N=210 and N=179.)

Some of the reasons for exclusion are perplexing. Let's review a few of these reasons.

Six people refused the booster shot. There is an argument that they should be excluded as they did not get the treatment under study, and there is no blood to be analyzed. 

Fifteen were dropped because they did not have "at least 1 valid and determinate immunogenicity result within 28 to 42 days after the booster shot." This sets my alarm bells ringing because this exclusion can only be applied after the primary outcome of the study is measured. It's not clear what constitutes "valid" and "determinate". It's concerning to lose 5% of the study population by effectively saying we failed to obtain the primary endpoint.

Then, they dropped a further 30 people (10% of the study population) because their clinician decided that these people committed "protocol deviations" before the 3D+30 day evaluation time.

Last but not least, they also removed 34 people from the study population (15%) based on a key clinical endpoint. This exclusion criterion is described as "evidence of infection up to 1 month after booster dose". Notice it says "after" booster dose, not before. So, it appears that the following happened: they selected about 300 people to start this study regardless of their prior infection status, various people were excluded for the reasons described above, blood samples were collected, PCR tests were performed (on the day of the shot, and then when the participants self-reported symptoms), and if anyone gets sick within 30 days of receiving the booster shot, they were kicked out of the study.

Remember the design of this study. The indirect outcome of antibody levels is used to infer the direct outcome of infection. So when someone is dropped because of infection, we should infer that the dropped person has inadequate antibody levels. These deletions induce a bias in the primary endpoint, raising the average antibody levels of those who remain in the study.

Why is the case-counting window set to 30 days instead of the 7 days used for the 2nd dose? That's not explained in the Briefing document.  

***

I may come back to the safety data some day but the sample size is so small that only really loud signals can be heard. If no adverse effects are found, one can't conclude that there are no adverse effects; one can only say that there are no adverse effects that can be detected by this study design.

 

Do vaccines have halo effects?

Today, I read an unusual clinical trial that tests a theory called "heterologous immunity". This theory postulates that humans have a type of general immunity that is not specific to the antigen. Therefore, administering existing non-Covid19-specific vaccines may give a kind of halo effect that offers protection against Covid-19. The benefits of such an approach is that we'll be able to use proven vaccines that have better safety profiles, are widely available, are cheaper for less developed nations, and have more solid years or decades of research behind them.

The study was conducted at the Federal University of Santa Catarina in Brazil. About 400 health workers at the University Hospital enrolled in the double-blind, placebo-controlled clinical trial. The vaccine arm got two shots of the MMR vaccine 8 weeks apart; the placebo group received two shots of saline solution on the same schedule.

The result is fascinating: the MMR vaccine had efficacy of 10%, that is to say, the infection rate of the vaccine arm was 90% that of the placebo arm. The VE of 10% is not statistically different from 0%.

If you've been reading this blog, you'd know that such a VE number means nothing unless you know exactly how they count cases. I'm happy to report that the above number is completely devoid of any kind of tricks. They are counting all infections, based on positive PCR test results, regardless of symptoms. They are counting every case from first shot, thus no case counting window. There is no secretive adjudication committee to inject subjective criteria into case counting.

What is fascinating is that about half of the cases in the vaccine arm were asymptomatic, compared to 20% in the placebo arm. In other words, the MMR vaccine does not prevent infection but among those infected, the health workers were more than twice as likely to experience no symptoms. You have heard this story before - that sounds like what they are saying about the mRNA vaccines.

So, the MMR vaccine - if deployed against Covid-19 - would be a prototypical "leaky" vaccine. The mRNA vaccine also appears to be leaky but not to this extent. Whether this is a bad thing depends on whether asymptomatic people have high enough viral loads to transmit the disease. What's certain is that - if they are infectious - they are even more likely to be spreading the disease since they won't know they are infected. The study did not measure viral loads, so this question is left unanswered.

***

The researchers now compute the outcome measures we have come to expect from these vaccine trials. They reported a 50% VE for symptomatic, PCR-positive Covid-19, and a 78% VE for "Covid-19 requiring treatment". 

These alternative VE measures each involve discounting some of the cases in the earlier number, typically resulting in higher VE values. To get from the 10% VE to 50% VE for symptomatic Covid-19, about 50% of the cases are removed from the vaccine arm, and 20% from the placebo arm. Then, going to 50% VE to 78% VE involves removing mild cases.

It's frustrating that the research community refuses to standardize key definitions. The so-called Covid-19 requiring treatment sounds like hospitalizations but it's more than that to this Brazilian team. To count as requiring treatment, the participant must have a positive PCR test, respiratory or clinical symptoms, and treatment such as "anti-coagulation, corticosteroid and antibiotic therapy or hospitalization".

Statistics has become collateral damage during this pandemic. I find yet another proof in this study. These researchers overlook the huge margins of error associated with the above estimates: the margin of error on the symptomatic VE is 31% to 78%, and that on the "requiring treatment" VE is - gasp - from 6% to 82%. When ignoring these, they are acting just like the average vaccine researcher.

***

These researchers - to their credit - did not apply a 2D+14 day case counting window. Doing so removes even more cases from the count, resulting in an even higher VE. If they have added the case-counting window, plus subjective adjudication, it's likely that the MMR vaccine would become competitive with the lower end of Covid19-specific vaccines.

That can be interpreted as evidence of "heterologous immunity" or "grade inflation". By grade inflation, I mean, the metric may have be defined in such a way that it sets a low bar.... just like in many colleges in the U.S., it's much harder to not get an A than to get an A.

***

Let's contemplate that a bit more. In the Brazilian trial, the average rate of infection was  25% (remember this is about the same across both arms). The median follow-up time was 5 months (unclear whether this is counted from first dose or second). Roughly speaking, the Pfizer study reported cumulative case rate in the placebo arm of 6-7% about 5 months counting from 1st dose. Why such a huge gap?

I have already mentioned two factors contributing to the wide divergence in case rates. The Brazilian number counts asymptomatic cases while the Pfizer number excluded cases by adjudication.

The authors believe that the high infection rate is due to high exposure at the hospital. If true, it provides a little information on something we have so far been blinded to: how many of the uninfected participants in any trial have been exposed to the virus, and how many simply have not gotten exposed during the study period?

The higher exposure factor is balanced by the favorable demographics: the Brazilian participants are generally young (less than 10% over 50 years old), and healthy (most have no co-morbidities). 

***

On the topic of comorbidities, the table excerpted below includes some intriguing data: (remember the allocation is 1.5 vaccine to 1 placebo)

The two co-morbidities that exhibit the highest infection rates (almost 80%) - hypertension and diabetes - are skewed towards the placebo participants while the MMR vaccine arm has more smokers, who surprisingly show a 12% infection rate, half that of the average in the trial. None of these observations are statistically significant because of small sample size. What the table shows is that in small samples, randomization does not guarantee balance on all variables.

***

The result from this study makes me wonder whether such vaccines can be used as booster shots.

 

Primer on regression adjustments 3

This is the third post in a series on using regression to correct for biases in observational data.

Let's review what we've learned so far. In the first post, I gave an example that shows regression will correct bias in a biased dataset. We looked at a sample of height data, in which women are over-represented relative to the underlying population. Without adjustment, the sample average height under-estimates the population average since women on average are shorter than men. After the regression adjustment, the gender-adjusted sample average height is almost identical to the population average.

In the second post, I expose a hidden assumption in the so-called regression-adjusted estimates, that everything is evenly distributed. In a gender adjustment (with only males and females in the population), the assumption is 50% males, 50% females. In an age-group adjustment (with 5 levels), the assumption is 20% in each level. If age is entered as a continuous variable, the age-adjusted estimate concerns persons of average age in the sample.

Needless to say, most real-world populations are not evenly balanced, nor do they contain only average persons. Therefore, if the goal is to estimate the population value, the regression adjustment may only correct the bias partially, and may even backfire, moving the sample average further from the population average. In other words, don't assume that dumping data into a regression software is a magic wand that cures all biases. And don't trust "experts" who wave this magic wand.

***

Chapter 3 of Numbers Rule Your World (link) is titled "the dilemma of being together." The chapter explores a fundamental problem in statistical analysis: the tension between capturing complex relationships in the data and finding generalizable features. The analyst resolves this question by deciding how much to aggregate the data. In regression models, this relates to how many averages do we include in the equation.

In the most naive model, the analyst summarizes the entire sample in one number, the sample average, and asserts that it is the best estimate of the population average. Thus:

[Naive] Population Average Height = Sample Average Height

Recall the statistics of our sample B of 900 people:

The naive model simply states:

[Naive] Population Average Height = 66.1

The average height of Population B is 67.1 inches, and the error of 1 inch is massive, equal to the width of a 95% confidence interval, as explained in post #1. This model makes systematic mistakes: the errors for males tend to be negative while the errors for females tend to be positive. A good model is one in which errors are as if random.

The underlying problem is that this naive model is too simple. On average, males are taller than females. So we err by summarizing everyone to one average.

To deal with this issue, we add complexity to the regression model. The so-called gender adjustment adds a gender term:

[Gender-stratified] Population Average Height = 66.6 + 2.6 if Male - 2.6 if Female

This is the regression output if I throw the data into software. It's the same equation I showed in post #2. People then take the constant term (66.6) and call that the "gender-adjusted" sample average height.

Note that the above equation is equvalent to saying:

[Gender-stratified 2] Population Average Height for Males = 69.2 while Population Average Height for Females = 64.0.

The reason for calling this "gender-stratified" should now be clear. In fact, this is a better representation of the model because the value of the constant term is not unique.

A mathematically equivalent formulation of the above model is:

[Gender-stratified 2b] Population Average Height = 69.2 - 5.2 if Female

This little wrinkle catches many software users by surprise. Depending on how the software developer codes the data, the regression outputs are different, even though they represent the same model. If you don't believe it, check that the new formula is also identical to:

[Gender-stratified 2] Population Average Height for Males = 69.2 while Population Average Height for Females = 64.0.

***

The standard way of thinking about gender adjustment is that it makes gender irrelevant. A gender-adjusted estimate removes the effect of gender, allowing us to focus on the outcome of interest.

A different - in my view, better - way of thinking about gender adjustment is that it acknowledges gender's effect on our outcome of interest. The regression adjustment involves building gender into the structure of the model.

The gender-stratified model finds that males and females have different heights: for males, we estimate, an average of 69.2 inches while for females, we estimate an average of 64.0 inches. By contrast, the naive model treats everyone as average, and we estimate the average height to be 66.1.

Gender adjustment does not prove that gender is unimportant. It's the opposite: building gender into the model means gender cannot be ignored. If gender does not matter, then we expect the gender-stratified model to reduce to the naive model. In other words, the coefficient of the gender term is zero (statistically speaking).

[Gender-stratified 3: Gender not important] Population Average Height for Males = 66.1 while Population Average Height for Females = 66.1.

And, formulated with a gender term:

[Gender-stratified 3: Gender not important] Population Average Height = 66.1 + 0 if Male - 0 if Female

So the average women in this special place is as tall as the average men.

What about the gender-adjusted sample average height of 66.6? This is the number that researchers in certain fields would have extracted from the stratified models -- recently, they have featured prominently in studies of vaccine effectiveness, including those by CDC scientists.

Mechanistically, these researchers have built gender-stratified regression models, then they ignore the gender coefficient, and report the constant term as gender-adjusted. As explained in post #2, this procedure is equivalent to averaging the gender-level height estimates, placing equal weights on male and female.

In statistics textbooks, we routinely warn readers that the coefficients of a regression model ought to be interpreted holistically. Should this rule apply to vaccine studies? If not, why not?

I'll pick this question up in the next part of this series.

***

If you missed the previous posts, you can still catch up on post #1 and post #2. 

How to count the fare after letting most passengers leave the train

We don't live in normal times and so the vaccine trials have been accelerated in order to deliver the promise of stemming the pandemic.

What does a normal clinical trial look like? In the most fundamental setting, you recruit participants, screen them for eligibility, randomly split them into two equal groups, give one group the medicine being tested, and the other group a placebo, wait a pre-determined follow-up time, and when the trial ends, you tally the outcomes in both groups and compare them.

In designing and running randomized controlled trials (RCTs), every effort is taken to make sure that the two groups of participants are identical and identically treated so that any difference in outcomes can be attributed solely to the medicine (vs placebo) and nothing else.

***

One way to accelerate trial results is to conduct an interim analysis. In the simplest setting, one conducts an analysis half way through the trial. This procedure assumes that the first half of the trial is predictive of the second half. It also assumes that the sequence of enrollment is random. 

Think of participants as boarding a train from the front to the rear. During a normal follow-up period, the train moves a fixed distance forward.  Passengers can only alight at a fixed disembark point on the platform, so at the end of the journey, the rear of the train has stopped there. In other words, every participant is tracked for the same amount of time, say one year.

For an interim analysis, the train is stopped midway to its destination. An inspector boards the train to check everyone's documents. The people at the front of the train has traversed a longer distance than the people at the back of the train. In statistical jargon, we say that the outcomes are "censored". 

At interim analysis of the vaccine trials, someone may have just received their first shot a week before the date of analysis, and so the investigators have followed them for one week only - and they can only contribute to the case counts up to 1D+7 days (7 days after first dose).

If you look at a cumulative case curve like this at interim analysis, imagine the front of the train running from left to right from Day 0 to Day 238. The interim analysis stops the train. The passengers at the front of the train have moved a further distance to the right than those who boarded last. The number of participants who can contribute to the case counts decreases as you go along the time axis from left to right.

As I have pointed out before (link 1, link 2), most vaccine trials have some form of staged enrollment, usually recruiting lower-risk participants first, and so the assumption of random sequencing of enrollment fails. These are the types of biases that appear when one deviates from the standard design. Investigators should correct for this and other biases.

At final analysis, all passengers have exited the train at the same disembark point. That means all participants have experienced the entire timeline. So the bias correction is only necessary at interim analysis but not at final analysis.  

A different adjustment - unrelated to bias - is that the bar for statistical significance is raised at interim analysis to account for additional uncertainty due to shorter follow-up. (Nevertheless, statistical significance has become collateral damage of the pandemic.)

***

That is a long setup for the topic of today's post, which is the "6-month" analysis of the Pfizer vaccine trial. A preprint of the scientific results has finally been published last week (here). The report is notable not just for what it says but what it doesn't say. My goal is to help fill out the details so you can digest the analysis with full context.

The first thing I noticed on the summary page of the report is that the investigators carefully inserted the words "up to", which includes the possibility that only a small proportion of people have reached 6 months of follow-up after the second dose. 

Enrollment in the Pfizer trial took place between end of July and end of October of 2020, and this interim "6 month" analysis froze the data on March 13, 2021. At this point, the head of the train is at about 7.5 months out while the rear of the train is at 4 months so we know for sure not everyone has reached 6 months of follow-up.

A key sentence in the paper discloses that about 55% of the participants have reached 6 months following their second doses: "During blinded and open-label periods combined, 55% of BNT162b2 [Ed: name of the Pfizer vaccine] recipients had ≥6 months follow-up post-dose 2."

It's important to realize that the above sentence did not state simply that "55% of trial participants had ≥6 months follow-up post-dose 2." The wording makes all the difference!

They didn't say the simple sentence because it's not true. The statement would have applied under normal circumstances but we live in extraordinary times. Since December, when Pfizer obtained EUA for the vaccine, a steady stream of placebo participants have been given the vaccine - they have effectively disembarked from the train before reaching the destination (link). By Pfizer's own publicity (link), they aimed to offer all placebo participants the vaccine by March 1, two weeks before the analysis date. 

As a result of the early-than-planned exits, only a fraction of the original placebo arm had ≥6 months follow-up post-dose 2. How many remained? How about 6%? This fact was disclosed in the adjoining sentence: "6% of placebo recipients had ≥6 months follow-up post-dose 2".

This is the data challenge that the analysts faced due to the decision to dismantle the placebo group. By design, 55% of the placebo group should have had 6 months of follow-up by March 13 but because of the "vaccine transition option", the number reaching that goalpost shrank from 55% to 6%... from about ~10,000 to ~1,000. To use the terminology from above, 9,000 or so persons have been censored at earlier dates so they have zero impact on the later part of the case curve. The later part of the curve is where we find "long-term efficacy".

For the interim analysis published in December, 2020, the FDA required a median follow-up period of two months following the second dose. Fast forward four months, and the middle of this train should have reached six months, triggering the six-month update. And yet, this is just theory that ignores the vaccine transition plan for the placebo group. In reality, the six-month mark is at the 6th percentile of follow-up times, not the 50th percentile. Nevertheless, passengers who left (and received the vaccine) are not allowed to re-board the train, and thus conducting the analysis later will make no difference.

Tellingly, in the VE table (Figure 2), the last line computes VE >= 4 months after dose 2. That's strange when the study is used to confirm "six month safety and efficacy." 

***

As an analyst, the crisis you're facing is that the vaccine transition option effectively destroys the careful RCT design, turning the data into observational ("real-world") data. This means the key to success is bias identification and correction. 

Unfortunately, that's not the path taken. The paper makes no mention of how many placebo participants chose to receive the vaccine, and when they did so. Timing matters a lot. It does not provide background statistics comparing those who transitioned, and those who didn't, nor any differences between those who transitioned earlier relative to those who transitioned later.

What the investigators did instead is to devise a mechanism by which they subtracted people from the vaccine arm to match the censoring of the placebo arm. The paper spent almost no time on analytical methods. We do have this sentence as a clue: "During the blinded period, 51% of participants in each group had 4 to <6 months of follow-up post dose 2; 8% of BNT162b2 recipients and 6% of placebo recipients had ≥6 months follow-up post-dose 2."

For placebo participants, the blinded period is before they were told they got saline shots during the trial and given the vaccine. The people in the vaccine arm were apparently also sent the notices (this wasn't described in the paper), and most of them responded to learn they were vaccinated.

The investigators then censored everyone in the vaccine arm who responded, meaning they de-boarded the train. That is the reason why the proportion of people in the vaccine arm with over 6 months follow up also plunged - from 55% to 8%.

The massive exodus from both arms of the trial appeared numerically similar but the reasons for exiting were night and day. Pfizer said it would be unethical to hold out their 95% efficacious vaccine from the placebo participants for the sake of science. In order to give them the vaccine, they must first be unblinded, i.e. told that they were assigned the placebo shots before. This reason does not apply to the vaccine-arm people as they have already taken both shots.

With the unblinding of the vaccine arm, the researchers threw out a lot of fine data. Instead of estimating the 6-month case rate of the vaccine arm using ~10,000 people, they did so with 10 times fewer participants. Students learn in Stat 101 that sampling error is a function of the square root of the sample size: to halve the error, you need 4 times the sample size. Reverse this. If you reduce the sample to a quarter of its size, you double the error. If you reduce it 9 times, you triple the error. That's one way to quantify the price of the vaccine transition plan. (The dismantling of the placebo arm is inevitable but it is the investigators' choice to inflict it also on the vaccine arm.)

I'd have taken a different path. The VE is based on a ratio of the case rates of the vaccine arm and the placebo arm. By preserving the original sample on the vaccine arm, I would have one good rate estimate (vaccine arm) and one bad estimate (placebo arm). In their analysis, they used two bad estimates. 

***

The challenges of observational data do not end there. It's not just about the size of the sample. It's also about the quality.

The participants who stayed on the train are unlikely to be a random sample of the original population. What would you choose to do if you were given a chance to confirm that you have been given the vaccine, and if not, to receive a vaccine that attained a 95% efficacy in the middle of a worsening pandemic? I can only guess at why a small number of participants decided not to unblind themselves. Maybe they already figured out their status. Maybe they think they don't need a vaccination, perhaps because they have lower risk, or they don't care if they are vaccinated or not. No matter what, these are the people who populate the right side of those cumulative case curves which speak to long-term efficacy. It's hard to believe that the ones who stayed on the train are identical to those who left.

I have more to say about this study. The above provides a basic understanding of why this is no ordinary study.

P.S. [8/9/2021] Here are the other posts: post 2, post 3