Big Data, Plainly Spoken (aka Numbers Rule Your World)

Open questions of the pandemic, part 3

As we head into the third year of the novel coronavirus pandemic, I've been reviewing some questions that are still open. The last two posts can be found here and here.

Today's further questions are inspired by the following chart put out recently by the White House.

On the surface, this chart provides evidence that the Covid-19 vaccine may protect people from getting hospitalized. If you're serious about causal analysis, you'd say all it depicts is a correlation. For the above chart, which plots "real-world" data, cannot be interpreted as if the data came from an RCT. In other words, the two groups (vaccinated and unvaccinated) are not identical except for vaccination status.

6. Why are unvaccinated people being hospitalized at an accelerating rate?

Focus on the blue line for the time being. This line shows the hospitalization rate for unvaccinated people, expressed in number of hospital admissions per 100,000. Between start of July and end of November, 2021, the weekly hospital admissions of unvaccinated people exploded from 6 per 100,000 to 68 per 100,000. Those are rates not counts.

This seems strange to me since I'd have expected the VE to come from a sharply reduced rate in vaccinated people while the rate among unvaccinated remains steady.

During that same period, the proportion of Americans who became fully vaccinated grew from 50% to 60%. The vaccines are said to be extremely effective, which should lower the amount of virus circulating in the population. Further, the unvaccinated proportion shrank from 45% to 30%, which is a 33% reduction rate!

The same dynamic shows up on the deaths chart as well. In other words, as more and more Americans get two doses, and fewer and fewer remain unvaccinated, the death rate among the remaining unvaccinated has skyrocketed.

Why? Why? Why?

7. How come VE is increasing while immunity is waning?

The definition of vaccine effectiveness is the relative ratio of the event rates of vaccinated versus unvaccinated. Events may be cases, hospitalizations, deaths, etc. Disregarding many methodological issues that I frequently discuss on the blog, I'm taking their analysis at face value. VE when measured by hospitalizations is then the ratio of the two curves shown. What is very odd about the chart is that the value of VE fluctuates wildly from week to week.

As a reminder, this is the overall trend of hospital admissions in the U.S. in 2021:

Hospitalized cases tumbled to a seasonal low in July, and then started rising again. In the first half of the year, the media kept reiterating that the vaccines were directly responsible for the dropping hospitalizations. Later, they said lower immunity against Delta and/or waning immunity was the reason for the return of cases. This narrative implies that the VE would be highest in July and dropping as the cases rose in the second half of 2021.

Surprise, surprise. If we compute the ratios of the rates shown in the White House chart, VE in July was the lowest (87%) and as the hospitalization rate of unvaccinated jumped 10 times, VE nudged up from 87% to 94%.

Why? Why? Why?

***

Any seasoned data analyst can tell you the very first step to good data analysis is asking the right questions. In this current series of posts, I show that the media have not been asking some of the most difficult questions. We can't pretend that we understand what's going on, unless we confront these challenges head on.

The Merck Covid-19 pill: is the result statistically significant?

Two weeks ago, the FDA advisory committee narrowly voted to recommend authorizing the sale of Merck's Covid-19 pill (Molnupiravir or MOV). This is a rich story worthy of a number of posts. We start with the final result and work our way backwards.

***

Molnupiravir is said to be effective at reducing the risk of hospitalization or death of Covid-19-infected people at high risk of developing severe disease. The treatment program spans five days, two 800-mg pills per day. The study was set up as a randomized clinical trial with ~1,550 participants, randomly divided into two halves, one taking MOV pills while the other ingesting placebo pills. The primary endpoint was the difference in event rate between the two arms. An event is either hospitalization or death. Each participant has been followed up for at least 28 days (or until an event occurred).

At the end of the study, the event rate was 10% on placebo and 7% on MOV. The difference in event rate is -3%, meaning that for every 100 people taking the pills, 3 fewer people will be hospitalized or dead, on average. For each person to benefit, about 33 must take the pills. (This number, known as NNT or numbers needed to treat, looks bad because 90% will not develop severe disease whether or not they take MOV.) Note also that for every 100, 7 will be hospitalized or dead despite taking MOV.

The above result has been reported as 30% efficacy, which is the relative event rate ratio (1-7%/10%). I prefer using differences as they don't exaggerate the impact off a small baseline. This primary endpoint is not statistically significant (but it's being portrayed as significant; I will get to this point later).

Molnupiravir may have very serious side effects. None of these effects have surfaced in the trial but the trial design contained so few participants that it can only find very common effects. Only about 700 people got treated with Merck's pills. An event with risk of 1 in 1,000 has almost 50% chance of not registering a single case in 700 people. An event with risk of 1 in 10,000 has over 90% chance of not showing up in this trial.

The FDA Briefing document that was reviewed at the FDA meeting on November 30, 2021 (link) describes several potential adverse side effects, all found in earlier animal and lab studies. These include bone marrow toxicity, mutagenecity, and reproductive toxicity.

***

How this type of treatment can be put to clinical practice should have been thought through before running this experiment, but evidently, it wasn't.

Rolling out the treatment requires an accurate prediction of who are at high risk of developing severe disease. Just like we don't know what level of antibodies is high enough to prevent infection, we don't know what factors predict disease severity. During the trial, Merck went with the usual list of "risk factors" for hospitalization or death, such as older age and obesity. This just means most patients who are hospitalized with or dead from Covid-19 are older and/or obese, but that's not the same as saying most older or obese patients will develop severe Covid-19.

What we do know is that the patients present in the Merck trial definitely had unusually high risk of being hospitalized or dying. The 10% baseline event rate meant that 1 out of every 10 Covid-19 patients enrolled in the trial ended up hospitalized or dead. This rate is over 100 times higher than the unvaccinated rate of hospitalization in the U.S., according to CDC numbers cited in the FDA briefing doc.

In real life, how does one find a comparable set of patients who would have the highest chance of benefiting from these pills?

**

The FDA document also discloses a number of other headaches - all of which are direct results of the design decision to exclude certain groups from the clinical trial, namely, vaccinated people, pregnant people and immunocompromised people.

The design decision is odd because vaccinated people constitute the vast majority of Americans, and the other two groups are apparently at higher risk of developing severe disease. Instead of oversampling them in the clinical trial in order to measure its effects, they were excluded completely from the trial. Strictly speaking, there is no evidence that these pills work for anyone who has had at least one dose of Covid-19 vaccine. This meant that the FDA advisors are asked to take part in a game of guessing without any actual data to support their opinions.

In the trial, those who were seropositive at baseline did not benefit at all from taking the Merck's pills (5 events vs 2 in placebo - nice example of why I dislike relative ratios), indicating that MOV might not work for vaccinated people who have antibodies. The FDA analysts cast doubt on this result, arguing the sample size was too small - and yet, they have no trouble saying that the pills appear to work equally well across age groups, and other variables - when these analyses contain similarly small samples. (e.g. 136 who were seropositive vs 118 who were > 60 years old)

Contrast statement (a): "Given the small size of the seropositive at baseline subgroup and the small number of events in this subgroup, robust conclusions regarding MOV efficacy in seropositive participants cannot be made"

with statement (b): "Efficacy results were generally consistent across subgroups including age (>60 years), at risk medical conditions (e.g., obesity, diabetes), baseline COVID-19 severity (mild, moderate) and SARS CoV-2 variants"

This is the kind of flexible interpretation of statistical significance that frustrates your statistics professor. You either believe none of the subgroup analyses should be considered because the trial was not designed to have enough samples to read them, or you believe they should all be trusted.

***

Another place in which statistical significance was mishandled was hinted at earlier. The main result is a 3% reduction in event rate, which has a 95% confidence interval of 0.1% to 5.9%. Notice that the CI apparently missed 0% by a pip - if the CI crosses 0%, the result is statistically insignificant, leading to the conclusion that the trial did not generate sufficient evidence to prove that MOV was effective at changing the event rate.

But is it barely significant or not significant? Here are two additional issues for your consideration:

What if there were one fewer hospitalization counted in the placebo group (67 instead of 68 events)? This tilts the confidence interval to the other side of zero, making the result statistically insignificant. Talk about hanging on a thread.

Now, in Table 1 (footnote) of the FDA Briefing Doc, we learn that "participants with unknown status at Day 29 are counted as having an outcome of all-cause hospitalization or death in efficacy analysis". Said in English, this means that anyone for whom they didn't know the status by Day 29, they assumed to have been hospitalized or have died. I highlighted this line in red because this seems out of the ordinary. Usually, we assume the opposite (presumed alive vs presumed "dead").

How many people fell into the bucket of unknown outcome, turned into hospitalized or dead? Exactly one, on the placebo arm. Thus, if they had made the opposite assumption, that the unknown person was not hospitalized or dead, then the statistical significance flips!

***

The above discussion of statistical significance assumes a normal analysis schedule in which analysts perform a full analysis when the entire study population has passed the full follow-up period of 28 days. That was not what happened.

In many clinical trials, researchers are allowed to peek at early results, known as interim analyses. In this Merck study, they performed an interim analysis at the half-way point, when half the participants have completed 28 days of follow-up. (This explains the two press releases from Merck about the same trial.)

A problem with peeking at interim results is that it increases the chance of false-positive readings (i.e. wrongly declaring significance). To compensate for this risk, the required level of significance must be reduced (more stringent). We encountered this issue before when looking at Moderna's vaccine clinical trial analysis plan. (Link)

Because of early peeking, the required significance level is not the standard 5% but lower than 5%. A different way of saying this is we shouldn't look at the 95% confidence interval but something higher than 95%, which means the width of the CI is larger than the reported 95% confidence interval. In other words, if the proper threshold was applied, the reported 3% reduction is not statistically significant.

They managed to come up with a justification to overturn the usual statistical practice: "Formal statistical testing was not performed for the full population assessment, because statistical significance was demonstrated at the interim population assessment. The nominal 1-sided p-value was 0.0218."

This is yet another statement that drives your statistics professor mad. Firstly, this statement literally embodies the statistical fallacy of "testing to significance", which I explained here. Secondly, even though they didn't do formal testing, they issued an "informal" testing result, showing a p-value of 0.02, suggesting that the testing result would be significant (under the conventional of p-value < 0.05 unadjusted for early peeking). Thirdly, they showed a one-sided p-value, which makes no sense in this situation; a one-sided analysis can only be justified if there is little doubt as to the direction of the outcome, and is almost never used in clinical trials. The "nominal" two-sided p-value is over 0.04, and likely over 0.05 when adjusted for early peeking.

The situation is worse when you compare what happened in the first and second halves of the Merck study:

There are rather disturbing signals on this table. The interim analysis showed an improvement of 7% over a placebo event rate of 14%. The final analysis showed the improvement to be 3% over a placebo rate of 10%. This meant that the Merck pills performed very badly in the second half of the participants: 6% event rate vs placebo event rate of 5%.

Thus, in this particular experiment, the first half of the result is contradicted by the second half. The FDA analysts endorsed Merck's argument to ignore the second half because "statistical significance was demonstrated at the interim population assessment".

If this were my experiment, I'd investigate whether there were operational issues (or enrollment biases) that caused the placebo event rate to go from 14% in the first half to 5% in the second half, a three-fold change. (By contrast, the event rate of the treatment arm only changed slightly.)

The phenomenon can just be regression to the mean, though. That's why classical statisticians don't like early-stopping strategies. If this trial was stopped for efficacy, and the second half was not analyzable, we would have over-stated the effect by double.

***

I've exceeded the usual post length again so further comments will appear in next posts.

 

 

Further adventures of case-counting windows

If it's not already clear, observational data is a great playground for statisticians. We sniff out where biases are hiding, and we devise methods to correct such biases. It's a lot of fun. It's also frustrating - because there are endless ways to make adjustments.

In today's post, I examine a couple of curious observations made by the Israeli researchers when they analyzed the data from vaccine boosters (link).

***

The researchers shrank the follow-up window when counting severe cases (relative to counting cases)

This decision surprised me because severe cases are a fraction of all cases so I'd want to have a longer counting window, not a shorter one.

Recall that in the analysis of cases, the Israel booster real-world study pulled data from the national database for the month of August. Because of a case-counting window that starts 12 days after the booster shot, in effect, the study only counts cases from August 11 to August 30. Most people did not get the booster on July 31st, and so the average follow-up time is probably around 10 days. (See the previous post for more.)

For severe cases, the case-counting window shrank by 4 days, as the researchers stopped counting on August 26 instead of August 30. Thus, the average follow-up time for severe illness is around 6 days. You're right - protection against severe illness was shown for less than a week.

Here is a direct quote that explains why the case-counting window was shortened for severe cases relative to all cases:

In order to minimize the problem of censoring, the rate of severe illness was calculated on the basis of cases that had been confirmed on or before August 26, 2021. This schedule was adopted to allow for a week of follow-up (until the date when we extracted the data) for determining whether severe illness had developed.

In the Appendix, the researchers offered the following chart as supporting evidence:

This chart shows the lag between testing positive and the onset of severe illness for Covid-19 cases between November and March 2021 (conditional on people subsequently getting severely ill). The median lag is 4 days, meaning among those who eventually got severely ill, half got severely ill within 4 days of testing positive, and half got severely ill after 4 days (up to almost 30 days).

I believe the logic is that for anyone who got sick in the final 4 days of August and became severely ill, the onset of severe illness would have taken place after the data cutoff for this study.

Consider the following implications:

• According to the chart above, half of the past patients took over 4 days and up to 30 days to develop severe illness. If we want to capture all severe cases, we ought to extend the follow-up period to at least 30 days - it should be 60 days if we use a one-month cohort. The methodology based on median lag time would capture only half of the severe cases.

 

• Think about the severe cases that occurred between Aug 26 and Aug 30. These severe cases are connected with infections that occurred prior to August 26, most likely during August. They are excluded from the analysis. All of these severe cases should count. The severe cases that happened on the first 4 days should be dropped, if we use their logic. (Note that the infection rate was skyrocketing during August.)

 

• Now think about someone who took the booster shot on August 16. If this person became severely ill on August 26, the severe case is not counted. But this severe case represents a lag time of 10 days from detection of infection, which according to the chart above, happens more than 10% of the time. This issue arises because the case-counting window is given a fixed ending date regardless of when the booster was administered.

***

The researchers disclosed they faced an incurable imbalance problem.

Part of the paper's Appendix is devoted to alternative analyses, one of which is a matching study. I discussed examples of matching studies before (Mayo Clinic, and Clalit). Unfortunately, this disclosure did not contain enough details to evaluate the quality of matching.

The following statement caught my eye: "Due to the very small number of severe cases, and the high censoring proportion, calculation for severe COVID-19 was not possible."

In my comments on the Clalit study (link), I mentioned a limitation of matching studies: if selection bias is severe, it may be hard to find acceptable matches, leading to dropping many unmatched cases ("high censoring proportion"). In this case, the matched population ended up being much younger than the vaccinated population.

In the Israeli booster study, the researchers attempted to find one person who hasn't taken the booster to match with each person who took the booster. Matching variables are the usual suspects like age group, second-dose date and so on. They appear to be saying that they failed to find adequate matches for a large proportion of the booster group. This is a sign of incurable covariate imbalance. This also suggests that those regression adjustments used in the main analysis are insufficient.

 

 

 

Boosters in Israel illustrate statistical issues with real-world effectivness analysis

I previously discussed the evidence that Pfizer submitted to the FDA to get approval for booster shots (here and here). The studies were small, and circumstantial. It appears that the experts were partially or completely swayed by "real world" data coming out of Israel so I looked at what is in that influential Israeli study (link to report).

The story of vaccinations in Israel is one of high hopes unfulfilled. Here is their full case curve for 2021. 

Between April and July, it appeared that the fast vaccination campaign yielded dividends; public health officials were satisfied that the vaccines beat Covid-19. Like other countries, Israel moved to remove most of the mitigation measures. Not so fast, as in July, cases started to tick up again. The rate of growth was so alarming that they became the first country to approve booster shots. Other countries, including the U.S., were curious what the outcomes would be - since the vaccine developers were not going to run proper randomized experiments to prove the value of boosters. Even after administering boosters, the case curve didn't start bending till late September. To this day, despite the general acceptance that vaccines are effective, no one can reliably predict what proportion of the population needs to get injections before infections start to decline, or how much time it would take to start seeing cases come down.

***

This Israel study (link) was published in October, and only concerned one month of data (August 2021). Approximately 1.1 million Israelis were fully vaccinated by July 30 and formed the study population. However, we do not know how many of those subsequently took the booster shot. This is one of those key omissions that have pervaded all the vaccine studies - basic check-off-the-box data that are withheld from the scientific reports.

The data came from the Ministry of Health. Data cutoff for cases was Aug 30, 2021, and these researchers invented a new case-counting window that starts at 12 days after the booster shot. Since the first booster shots analyzed by this study happened on July 30, that means no cases are counted until 12 days later (Aug 11, 2021). For those who took the booster on July 30, the study has a maximum follow-up window of 20 days. The graph above shows a linear growth in administered boosters during the study period. For someone who took the booster on Aug 15, the case-counting window opens on Aug 27 and closes on Aug 30, thus the follow-up period was 3 days. Anyone who took the booster after Aug 18 are "immortal": if they get sick after 12 days, those cases won't feature in the study data. 

This little calculation reveals that the follow-up period for the booster group is between 1 and 20 days, and that the study population is effectively three weeks, 12 days fewer than advertised (people who received boosters in those 12 days have not reached the beginning of the case counting window). That means, if we believe everything else in this study, the researchers have shown that the booster shot is effective for an average of 10 days.

The headline result of the study is that those who took the boosters are 11 times less likely to get infected than those who didn't take the boosters (note the unspoken: effective for 10 days, after ignoring anyone who gets sick during the first 12 days). For severe illness, the factor was 20 times.

As explained in my previous post, the use of these multiplicative factors (x times) is a fad. They could have expressed the same data in the usual way: the booster effectiveness was 90% for cases, and 95% for severe cases. In other words, they are claiming the same effectiveness as the original two shots.

The above calculations were "adjusted" by including main effects for age group, gender, calendar day, timing of 2nd dose, and "sector" (Arab/Jewish/Ultra-Orthodox).

***

A vaccine effectiveness metric is a relative ratio between the (severe) case rate of the treated group and that of the untreated group. In the original vaccine trials, the treated group took two shots while the untreated group took the placebo shots. In the real-world booster study, the treated group took the booster shot while the untreated group didn't; both groups previously received two doses of an mRNA vaccine.

The VE metric is merely a correlation between cases and vaccination status. In order to move from correlation to causation, statisticians want to establish that the booster group and the no-booster group are comparable; that the observed difference in case rates isn't driven by factors other than vaccination.

To understand the nature of this challenge, let's pretend that the FDA required vaccine developers to conduct a randomized controlled trial (RCT) to determine the causal effect of the booster shot on cases (The FDA didn't require it.)

We would have recruited participants from the group that has previously took two vaccine shots. Then, they are randomly divided into two groups: booster, and placebo. All participants are then followed for a period of time, and at around the two-month time point, an interim analysis is conducted. This analysis is simple, because by virtue of random assignment, the two groups have co-variate balance (same mix of age groups, genders, occupations, date of second doses, etc.), and thus, any variation in case rates is due to the booster.

Instead, we have "real world data". On the surface, we have two groups: booster and no-booster, which sounds similar to booster and placebo. But labels deceive. The lack of a random assignment device means the two groups are not apples to apples. Note that public health policy prioritizes high-risk groups, and some people have reasons to want early boosters (e.g. work on the front lines, want to stop wearing masks).

I find it rewarding to think about the study population in terms of three subgroups (bottom row of the left chart).

Start with anyone in the study population on July 30, the start of the study period. This person is in the no-booster group - until s/he decides to get the booster shot, at which point the same person migrates to the booster group. There are some who never took a booster during the month of August so they remain in the no-booster group throughout the study.

Let's consider the following three subgroups:

(NB1) People who never took a booster during August

(NB2) People who received a booster during August - they belonged to the no-booster group until the day of booster

(B) People who received a booster during August - they migrated to the booster group on the day of the booster.

(NB2) and (B) are "twins". They are the same people, counted twice, the first time as a member of the no-booster group, and the second time as a member of the booster group.

Thus the no-booster group (NB) in the study consists of (NB1) and (NB2) while the booster group is just (B). Now, let's address whether (NB) can be compared to (B).

***

Consider first (NB2) vs (B). These are twins, they are the same people counted twice. Thus, by definition, we have 100% co-variate balance, if we consider the typical demographic co-variates, such as age, gender, and occupation. Even date of second dose is also perfectly balanced since that's a variable about historical behavior prior to the study.

Nevertheless, (NB2) differs from (B) in pivotal ways. The exposure time windows are disjoint. The (NB2) twin always have earlier exposure than the (B) twin. Besides, the length of exposure is negatively correlated: the longer the follow-up time for the (NB2) twin, the shorter for the (B) twin. Thus, the booster status variable is confounded with exposure time and duration - and this is a hopeless confounding, 100% confounded.

Next, compare (NB1) with (NB2). Are the two partitions of the no-booster group similar? The answer is a resounding no. (NB2) are those who got the booster in August while (NB1), not in August. Unless one believes that the timing of the third dose is chosen completely at random, one must conclude that those two subsets are dissimilar due to selection biases.

In addition, (NB1) invariably has longer follow-up time than (NB2), and by extension, (B). The start of the follow-up window is fixed for everyone in (NB1) and (NB2) but variable for those in (B).

To summarize, (B) is identical to (NB2) on many co-variates due to twinning but differs from (NB2) pivotally on exposure metrics. (NB2) is different from (NB1) due to selection bias, thus these two subsets do not have co-variate balance; they also have different exposure durations.

When we compare (B) with (NB), we may be tricked into thinking we have good co-variate balance because of the (NB2) component. This statistical deception intensifies as more people get boosters. The exposure-related complete confounding is, to my knowledge, impossible to cure. Regression adjustments via main effects are insufficient to solve these complex statistical problems.

The differences between (B) and (NB) - in addition to booster status - include the exposure-related confounding with (NB2), the selection biases with (NB1), and the exposure-related bias with (NB1). All three are affected by the proportion of people who have taken booster shots.

Any serious causal analysis of real-world booster data must offer adjustments that deal with the above statistical issues.

P.S. [11/9/2021] In the following post, I looked at a couple of observations from this study that caught my eye.

 

Swear and gossip: answer to question 1

Last week, I posed two questions related to the Business Insider headline about swearing and gossipping in the workplace (link). Today, I provide my answer to the first question. The answer to the second question is here.

Question #1: Are the following headlines identical (i.e. interchangeable)?

Answer: They are not identical.

More precisely, the two statements can be identical under limited conditions but they are usually not interchangeable. As Scott quipped on Twitter, it depends.

A reason why we may think those headlines say the same thing is that people tend to draw the same causal conclusion from each headline: if team members swear and gossip more, their chance of success increases.

We can unpack the statements. The key difference between them is the order of presentation. The first statement starts with the base of teams that are rated as the most successful by some metric of success, and within this base, there is a high incidence of swearing and gossiping. The second statement starts with the base of teams that engage in swearing and gossiping, and within this base, there is a high incidence of success.

For those familiar with conditional probabilities, the first statement concerns the probability of swearing and gossiping given that a team is "most successful" while the second statement is about the probability of success given a team engages in swearing and gossipping. In general, P(X/Y) is not the same as P(Y/X).

According to the famous Bayes' Theorem, the ratio of those conditional probabilities is equal to the ratio of the prior probabilities so the two statements are identical if and only if the rate of success among all teams is the same as the rate of swearing and gossipping. P(swear and gossip) = P(success) is an unlikely event in real life.

***

Let's bring back the helpful diagram from the answer to Question #2, that lays out the four possible characterizations of any team.

Consider an extreme case in which all teams are either in A or D. Every team that swears and gossips is successful while every team that doesn't fails. So B and C both are empty.

In this case, A+B (teams that swear) and A+C (teams that are successful) are identical. The prior probabilities, and also the conditional probabilities are the same. We conclude that the most successful teams swear and gossip, and also that the teams that swear and gossip are successful.

In the above diagram, the conditional probability of swear and gossip given success is A /(A+C) and that of success given swear and gossip is A/(A+B). For those to be identical, B and C must contain the same number of teams. In the extreme case, B = C = empty. If we put x teams in B, then we must put x teams in C, and then we have to subtract x teams from A and from D to make things even.

There are a small number of scenarios in which this can happen, and a much larger number of scenarios in which the two probabilities aren't the same.

***

Consider the two statements in this format:

Formula (I) comes from the original headline while formula (II) corresponds to the other headline.

Formula (II) predicts success (... leads to success) while formula (I) describes success (a successful team does ...). Formula (II) does not imply formula (I).

In the extreme case, every team swears and gossips, and so it follows that the most successful teams swear and gossip. Since everyone does it, swearing and gossiping will not predict success. Formula (II) does not imply formula (I).

In order to learn Formula (II), we must add other factors that affect team success.

(In the above, I excluded the possibility that success may cause swearing. Allowing this relationship complicates the causal interpretation even more.)

***

The confusion between descriptive and predictive (or causal) statements is endemic. For example, in coronavirus news, we keep hearing statements such as "Most of the hospitalized people are  unvaccinated." This is very different from the related statement of "Most of the unvaccinated people are hospitalized."

If we think of vaccination status as a cause of an effect, and hospitalization as an outcome, then "Most of the hospitalized people are unvaccinated" is a description of people with a given outcome. It is not a predictive statement supporting a claim that vaccination status predicts some outcome. In addition, the first statement does not imply the second statement - that "most unvaccinated people are hospitalized."

 

Swear and gossip: answer to question 2

Today, I give my answer to Question #2 in the test presented a few days ago, here.

***

Question #2: Do the following headlines contradict or can both be true at once?

When we are presented with just the first statement that the most successful teams swear and gossip, we are tempted to draw a causal conclusion. We think swearing and gossiping is the cause of success. This motivates the advice given by the headline writer. (Okay, so the writer did not say out loud encourage your team members to swear and gossip more... but is there any other possible interpretation?)

If the cause--effect relationship holds, we may expect the least successful teams to not engage in swear and gossip, which means the truth of the first statement denies the truth of the second, and vice versa. (Even this is not certain since there are other factors affecting team success.)

If we are told both statements are true at the same time, I don't think any of us would conclude that swearing and gossiping determines team success. That's why it's important to know if one statement implies the truth/falsehood of the other.

Answer: Both statements can be true at once. One does not imply the other.

More precisely, one may imply the other but often doesn't. As Scott quipped on Twitter, it depends.

One approach is to lay out the 2x2 grid of all four possible states of being. Like this:

Each team is classified as either successful or not (based on some metric of success). Each team is also classified as either heavy or light on the swear-gossip axis. The sum of the four numbers is the total number of teams.

The statement that "the most successful teams swear and gossip" suggests that A is substantially larger than C, or that A is a large proportion of (A+C).

The statement that "the least successful teams swear and gossip" suggests that B is substantially larger than D, or that B is a large proportion of (B+D).

The question is whether the values of A and C constrain the values of B and D in such a way that if the first statement is true, then the second statement must be false. To invalidate this, we can show scenarios under which both statements are true.

Imagine we have 100 teams. There are only two polite teams that don't swear and gossip, split evenly between successful and not successful. Next, we have 50 teams in cell A (successful and swearing) and 48 teams in cell B (not successful and swearing). Now, among the successful teams, 50/51 swear and gossip while among the unsuccessful teams, 48/49 swear and gossip. Both statements are true. The first statement does not deny the second.

We have a whole class of scenarios that work similarly. The split between A and B can be 49/49, 48/50, 40/58, 55/43, etc.

***

For those who have been exposed to contingency tables like the one above, you may be tempted to fill in the four numbers by multiplying the probability of success and the probability of swearing and gossipping. When you do this, you're assuming independence.

Independence means that knowing whether team members swear and gossip provides no information on team success. This is the opposite of the study's intention. The researchers must believe that team success is dependent on the level of swearing and gossipping.

To model dependence, it's not sufficient to know just the probability of success and the probability of swearing and gossipping. For each level of swearing and gossipping, we assume a different rate of success.

Through this exercise, we learn that three of the four values (A, B, C, D) are free. The only constraint is that they must sum to the total number of teams. In other words, having specified A and C in the first statement, we have only nailed down the sum of B and D but the split between B and D is not constrained. Thus, both statements can be, and often are, true at once.

***

In Numbersense (link), I described a real-life example of this fallacy.

Years ago, the Gates Foundation was putting a lot of money behind the "small schools movement". This was based on some studies showing that the top X schools in district Y have a high proportion of small schools (with small average class size).

The situation can be shown in the following table:

Notice that this is the same table as above, with "small class size" substituting for "swear and gossip".

The Foundation was shown just the first column, from which they concluded that small class size was the cause of high test scores (A being a large proportion of A+C). The statistician, Howard Wainer, looked at the entire distribution and demonstrated that the bottom X schools also have a high proportion of small schools (B being a large proportion of B+D). This corresponds to the second column.

If shown both columns, you'd not conclude that class size drives test scores. If you only know the first column, you'd be tempted toward that conclusion.

If you only know the second column, you'd be tempted to draw the opposite conclusion that small class size depresses test scores!

That's why data analysts want to see all of the data. That's why biased datasets can be extremely misleading. You can make errors of direction, not just magnitude.

***

There is an underlying statistical reason that makes Wainer's discovery inevitable in many real-world datasets. Extreme values (highest or lowest test scores) are associated with small class sizes, because small samples have higher variability.

Imagine a large school with 10,000 seniors, with average GPA of 3.75. Imagine taking a random sample of 1,000 students, and reporting the sample average GPA. It's not going to be exactly 3.75, off by some small amount. Now take a sample of 500 students, and the average deviation of the sample mean from 3.75 will be higher. Now do 100 students. The smaller the sample size, the more likely the sample average runs further away from 3.75.

In the business teams example, I would not be surprised (if they did not control for team size) that the most successful (and least successful) teams are smaller teams.

This is an example of statistical gravity: if the dataset consists of teams of varying sizes, the large teams will be found in the middle of the distribution, and the smaller teams will be found towards the tails.

 

 

Swear and gossip: a test of your statistical acumen

Business Insider has been flogging this headline since late August, and it was still on the front page last week:

I don't have a premium subscription so I don't know what evidence they have for this. I'm imagining that someone collected data, zeroed in on the teams that are deemed successful, and computed the percentage of team members who "swear and gossip". How do they know they "swear and gossip"? I have no idea - the data may have come from one of those Big Brother surveillance technologies.

***

I love this headline because it can be turned into a test of one's statistical acumen.

Question #1: Are the following headlines identical (i.e. interchangeable)?

What's your first instinct? Now, think about it a bit, and see if you change your mind. I'll present some hints in the second half of this post, and the answer in a future post.

 

Question #2: Do the following headlines contradict or can both be true at once?

What's your answer? Come back later in the week to check what my answers are.

For those needing some hints, keep reading.

***

For Question #1, one perspective is whether the order of presentation matters. Is the relationship symmetric? Imagine throwing a dice twice, and getting 1 then 5 in that order. If the question is the sum of the values, then the order does not matter. One followed by 5 is the same as 5 following 1, both adding to 6, in the context of this question. However, if the question is the values are increasing, then order matters. One followed by 5 and 5 followed by 1 are evaluated differently.

One approach that helps with either question involves writing down the entire set of possibilities. When you have two binary variables, there are four possible states. Are the two headlines talking about the same things?

Identity requires that each statement implies the other. Check both directions. If it's not true, then find counter-examples in which one statement is true while the other one is false. Question #2 asks about non-contradiction which is less stringent than identity. Identical statements cannot contradict each other but non-identity does not imply contradiction.

Remember we are doing statistics, not pure math. Almost nothing in statistics apply universally. Think of counter-examples not as a single example but a class of examples sharing the same characteristics.

***

Feel free to use the comments to discuss with other readers.

 

The booster shot data reveal why RCTs are still the gold standard

Observational studies have some common characteristics that annoy the analyst. In the previous post about Pfizer's booster studies, I described a few.

• The sample contains few examples of the people to which you're generalizing your result (e.g. people 65 or older, high-risk individuals of any age).
• The measured outcome is not the obvious, direct outcome (e.g. clinical outcome of infection) but an indirect indicator (e.g. antibody levels).

These problems create the need to extrapolate results. Such extrapolation is usually supported by unverified assumptions (aka expert opinion). Of interest, the FDA allowed Pfizer to impose a causal assumption: that if the booster achieves similar levels of antibodies as the original 2nd dose, then the booster also produces similar clinical outcomes as the original two doses. See the previous post for why modifying an indirect outcome does not invariably lead to a change in the direct outcome. (This matter is yet another real-world example about correlation and causation.)

As I have explained often on this blog, making assumptions is not a sin. All statisticians make assumptions - those who claim they make no assumptions make assumptions! Let me give a quick example. Suppose a college runs a survey asking recent graduates about their post-graduation salaries. Only 50% of the graduates responded with a valid number. Should the college impute the salaries of the non-respondents? If I were the analyst, I would impute the salaries, baking in the assumption that those who don't respond are likely to be earning lower salaries (possibly even zero). However, some statisticians will argue against using imputation. They'll claim that one should just report the average of those who responded, because it is a horrible thing to modify the data - one should only correct egregious data entry mistakes but not otherwise touch the data. Are they making no assumptions? 

They are making a huge assumption: they assume that the non-respondents have the same average salary as the respondents. What is the basis for that assumption? There is none, other than the desire to not tamper with the dataset. In fact, this assumption is almost surely wrong. As I said in Numbersense (link),

Beware of those who don't tamper with the dataset!

***

In this post, I'll cover several additional extrapolations that were used in analyzing the booster data.

The key immunogenicity results all measure antibody levels against the original virus (so-called wild type) but as we've been constantly reminded, the only virus currently in circulation is said to be the Delta variant. How did Pfizer bridge this gap?

According to the FDA Briefing document (link to PDF), "Pfizer proposes to infer effectiveness of the booster dose against the Delta variant from exploratory descriptive analyses of 50% neutralizing antibody titers against this variant evaluated among subjects from the Phase 1 portion of the study." 

"Exploratory descriptive analysis" is not regarded as a serious form of scientific evidence - the rules have changed during this pandemic emergency. I pity the statisticians working on these studies. They were assigned an impossible task. Recall that the gold-standard randomized clinical trial (RCT) has been thrown overboard, replaced by "immunobridging" analysis. The antibody levels stimulated by the booster shot are deemed comparable to those registered (by other individuals) after their second shots, and then the "non-inferiority" in antibody levels is assumed to result in the "non-inferiority" in vaccine effectiveness (a clinical outcome not directly measured).

The Delta variant introduces another complication: as it was not present around the time of the original Pfizer adult trial, researchers do not know what the VE of the first two doses is against Delta. This breaks the causal assumption of Immunobridging.

We do not have even an approximation of the VE outcome against the Delta variant for the 300-odd people in the booster trial. So if one accepts that one can expose the old blood samples to the Delta variant in the lab, and show that the antibody levels are comparable, one has to make an even stronger assumption by claiming that those antibodies will result in the same VE as measured in the Pfizer adult trial.

This line of argument unfortunately results in a contradiction. The original vaccine either works just as well against Delta as it worked against the original strain; or it is less effective, which explains recent surge in cases. Only one of these scenarios can be true.

If the first scenario were true, then the immunobridging assumption holds as VE for Delta is the same as VE for the wild type, but in this scenario, the recent surge in cases has nothing to do with Delta, and there is no need for a booster - since VE is the same.

If the second scenario were true, then the booster may be necessary, but in this scenario, the immunobridging methodology is broken as the prior value of VE no longer applies.   

Since Pfizer applied for approval for the booster, it is assuming the second scenario. One might be tempted to prefer assumption #1 because it permits the immunobridging analysis. That is a popular justification you find in academic papers, in the same vein as "computational feasibility" or "tractability". It's risky to use this logic in studies with real-world consequences. Besides, it creates a contradiction.

Beyond the logic of the analysis method, notice that the Delta analysis is performed using only Phase 1 participants, so we are talking about 11 people between 18 and 55, and 12 people between 65 and 75 years old. I'm not sure why blood from the other 300 people cannot be similarly analyzed. Without those results, the immunobridging assumption is expanded yet again to generalize from the 23 people to the 300-plus people (then to the 20K-plus in the adult trial, and finally to the U.S. population.)

A proper RCT would have yielded a much cleaner analysis of the direct clinical outcome. 

***

Another missing piece of the puzzle is the provenance of the 300 or so Phase 2/3 trial participants who were "selected" to take part in the booster study. Nowhere in the FDA Briefing document can I find the selection mechanism. Usually, this means the selection was not random from the vaccine arm of the earlier trial. If it is not random, what are the criteria? 

The 300 people reduced to about 200 that ultimately were used in any of the primary calculations. That's a drop-off of a third of the starting population, which is a high drop-off rate. (Note: Table 3 claims the evaluable immunogenicity population is 268 but the results shown in Tables 4 and 5 unexpectedly have N=210 and N=179.)

Some of the reasons for exclusion are perplexing. Let's review a few of these reasons.

Six people refused the booster shot. There is an argument that they should be excluded as they did not get the treatment under study, and there is no blood to be analyzed. 

Fifteen were dropped because they did not have "at least 1 valid and determinate immunogenicity result within 28 to 42 days after the booster shot." This sets my alarm bells ringing because this exclusion can only be applied after the primary outcome of the study is measured. It's not clear what constitutes "valid" and "determinate". It's concerning to lose 5% of the study population by effectively saying we failed to obtain the primary endpoint.

Then, they dropped a further 30 people (10% of the study population) because their clinician decided that these people committed "protocol deviations" before the 3D+30 day evaluation time.

Last but not least, they also removed 34 people from the study population (15%) based on a key clinical endpoint. This exclusion criterion is described as "evidence of infection up to 1 month after booster dose". Notice it says "after" booster dose, not before. So, it appears that the following happened: they selected about 300 people to start this study regardless of their prior infection status, various people were excluded for the reasons described above, blood samples were collected, PCR tests were performed (on the day of the shot, and then when the participants self-reported symptoms), and if anyone gets sick within 30 days of receiving the booster shot, they were kicked out of the study.

Remember the design of this study. The indirect outcome of antibody levels is used to infer the direct outcome of infection. So when someone is dropped because of infection, we should infer that the dropped person has inadequate antibody levels. These deletions induce a bias in the primary endpoint, raising the average antibody levels of those who remain in the study.

Why is the case-counting window set to 30 days instead of the 7 days used for the 2nd dose? That's not explained in the Briefing document.  

***

I may come back to the safety data some day but the sample size is so small that only really loud signals can be heard. If no adverse effects are found, one can't conclude that there are no adverse effects; one can only say that there are no adverse effects that can be detected by this study design.

 

Reading the evidence for the booster shot

The FDA Advisory Panel met in mid September to discuss Pfizer's application for a booster shot. This led to a partial authorization (link). The key statement is:

Today, the U.S. Food and Drug Administration amended the emergency use authorization (EUA) for the Pfizer-BioNTech COVID-19 Vaccine to allow for use of a single booster dose, to be administered at least six months after completion of the primary series in:

• individuals 65 years of age and older;
• individuals 18 through 64 years of age at high risk of severe COVID-19; and 
• individuals 18 through 64 years of age whose frequent institutional or occupational exposure to SARS-CoV-2 puts them at high risk of serious complications of COVID-19 including severe COVID-19

In today's post, I read the FDA's Briefing Document (link to PDF), released as part of the September meeting in order to understand the science behind this decision. This briefing represents the FDA's interpretation of data submitted by Pfizer.

***

The most striking discovery is that the FDA no longer requires a randomized clinical trial (RCT) to authorize vaccines. As explained in my research methods talk (link), RCTs - long considered the gold standard for data-driven medical science - have six key components: random sample of underlying population, randomization of treatment, double-blinding, control, placebo, pre-specification.

With the FDA's blessing, Pfizer offered data from two observational studies; these studies do not have most of those key components that make RCTs the standard.

The first study is a Phase 1 trial, consisting of a total of 23 people. All of them were given a third (i.e. booster) shot about 7 to 9 months after their 2nd shot.

The second study involved about 300 participants who enrolled in the big Pfizer adult trial. (Here is the link to results from that trial.) All of them were given a third (i.e. booster) shot, about 6 months after their 2nd shot.

In all, the FDA looked at two studies that enrolled fewer than 350 people, without a control group, without placebo, and not blinded. The investigators wished to generalize from about 300 to about 300 million people in the U.S. I'm unsure as to why they can't find 300 other participants from the big trial so that they could randomize treatment into extra shot versus placebo.

The FDA advisory committee apparently were not fully convinced by the data but they nevertheless authorized the booster shot for 65 plus and others at high risk. This is a curious decision.

The second study with 300 participants contained zero people aged 65 and above. The age cutoff was 55 years old. So the only older participants came from the first study, and there were N=12 such people labelled as 65 to 85 years old. Table 2 in the Briefing document revealed that despite the labeling, the 12 people's ages ranged from 65 to 75 years old so there was no data submitted for people above 75 years old. This is an example of tokenism, which I described here.

Further, the Phase 1 trial excluded all people who are at high risk of Covid-19 infection, and so all the evidence we have for 65 plus are from 12 healthy, low-risk individuals aged 65 to 75. Table 2 confirms that this dozen did not represent the diversity of Americans - not surprising given the tiny sample size. They are 100% white, 0% obese, and 0% with comorbidities.

The evidence for the second recommendation for 18-64 who are at high risk of severe Covid-19 is similarly stretched. Presumably, indicators of risk of severe disease include comorbidities and minorities. The second study only included people aged 18-55 so the only data available for the 55-64 age group came from the first study - however, none of these people are at high risk of Covid-19, let alone severe Covid-19, by virtue of the Phase 1 study's exclusion criterion.

As for those in the 18-55 age range, the second study enrolled 306, of which only 55 have comorbidities, only 28 are black Americans, and only 2 are native Americans. In short, the FDA authorization is based on observational studies that have really small sample sizes, which get attenuated further when the recommendations are restricted to subgroups. The advisors extrapolated these results to subgroups that have few or zero data based on their expert opinion. (I'm not denigrating expertise, just pointing out how it is applied.)

***

The FDA has months ago told the pharmaceutical companies they no longer have to run RCTs. They said that future "modified" vaccines can be approved based on "immunobridging" analysis. In this Briefing Document, the FDA further extends the rule to booster vaccines that are not modified from the original, which is what the Pfizer booster shot is.

The logic of immunobridging is to show "non-inferiority" in antibody levels after taking the booster shot compared to after taking the two-dose series. Instead of comparing vaccinated to placebo as in an RCT, such analyses compare people who took the booster shot to people who took the first two shots. If the antibody levels are non-inferior - as defined by statistical criteria, then it is presumed that the booster shot will provide the same level of protection as the original two doses, i.e. the famous 95% vaccine efficacy.

Pfizer submitted data that showed the antibody levels after the booster shot were non-inferior, using thresholds to which the FDA has agreed.

***

What is "immunobridging"? This bridge is what I'd call a "causal assumption".

You see, neither Pfizer nor the FDA can define the relationship between antibody levels and getting infected. During the Advisory Committee meeting, a Pfizer scientist acknowledged, "We actually looked at our breakthrough cases in our placebo-controlled phase 3 study, and have compared the antibody titers where we had the opportunity in individuals that got the disease versus the ones that didn’t, and we were also unable to really come up with an antibody threshold. So I think there’s probably a much more complex story and not easily just addressed with neutralizing antibodies."

What they were hoping for is a simple rule that says antibody levels > X means vaccinated person is protected. The "much more complex story" implies that the rule must involve not just antibody levels but many other factors.

We therefore know that antibody levels is necessary but not sufficient to induce immunity. The causal model requires other variables, some of which may be unmeasured. The chosen solution is to make a causal assumption: just assume that antibody levels is sufficient for protection. This is the meaning of the following sentence in the summary section of the FDA Briefing Document (link to PDF):

Effectiveness of the booster dose against the reference strain is being inferred based on immunobridging to the 2-dose primary series, as assessed by SARS- CoV-2 neutralizing antibody titers elicited by the vaccine.

Two steps are involved here: the 95% VE estimated in the adult trial is assumed to be directly caused by the vaccine producing enough antibodies to fight off infection - while assuming no other factors play a role; then, the booster shot is assumed to produce 95% VE if it induces "non-inferior" antibody levels compared to the original two shots.

***

The next question is how long will immunity coming from the booster shot last. The short answer is 1-2 months; it could be more but we don't have the data since this is another interim analysis with a brief follow-up period.

The second study has the bulk of the people (about 300). The follow-up periods range from 1.1 to 2.8 months post booster shot. But we should subtract 1 month because there is a case-counting window that would start 30 days after the booster shot. Effectively, the period of time during which these 300 people can accumulate cases is as low as 3 days, and up to 1.8 months (54 days). (The first study with 23 participants has the same maximum follow-up time.)

Where does the 30 days come from? This is analogous to the 14-day period in which cases are removed if they occur in vaccinated people. The antibody measurements were taken 30 days after the booster shot and compared to measurements taken at 2D+30 days in the previous adult trial. Since we are inferring protection based on antibody levels, we can't infer protection for any time prior to 3D+30 days by measuring antibodies at 3D+30 days. Effectively, for the third dose, the case-counting window has been pushed from 14 days after the shot to 30 days after the shot. Anyone who gets sick within one month after getting the booster will not be considered "breakthrough".

***

This post just scratches the surface of all the issues that arise when we don't have RCTs to evaluate medical interventions. These observational studies are small, do not have randomly selected participants, do not have a placebo/holdout arm, and are not blinded. As a result, any observed outcome could have been caused by a host of factors, including unknown or unmeasured variables. This forces data analysts to make assumptions. The quality of the analysis depends heavily on the quality of these assumptions.

 

 

 

 

Why do we have biased datasets?

Why do we have biased datasets?

The answer is simple. It's usually because biases have been actively injected into the data collection processes.

***

As U.S. colleges reopened to in-person teaching this semester, on the strength of a successful vaccination campaign, some schools continue to enforce testing. Last year, I covered how many colleges - such as Cornell and Georgia Tech, succeeded in keeping on-campus infections down by running a strict testing and tracing program (link, link). Staff and students were tested once or twice a week.

This term, the testing policies have been modified. At some colleges, people are still getting tested weekly. Make that some people. Specifically, these colleges require people who do not show proof of vaccination to get tested weekly. Meanwhile, fully vaccinated people are tested, only when they decide to - which means, when their symptoms have become so severe that they present themselves to a testing clinic.

This is a perfect example of injecting bias into one's data collection process. More testing leads to more reported cases. This bias is due to asymptomatic cases and mild cases, and as described before, is (inadequately) revealed by the positivity rate (what proportion of test results come back positive). Compulsory weekly testing adds asymptomatic and mild cases - as well false positive cases - to the tally. This is actually a good thing as Cornell and other schools demonstrated last year.

The new testing policies at some colleges mandate one set of rules for the unvaccinated, and another set of rules for the vaccinated. Because only the unvaccinated are subject to weekly testing, the case count for the vaccinated will include only severe cases while the case count for the unvaccinated will include everything.

***

Such biased data directly result in biased statistics, which lead to errant decision-making.

The differential testing policy guarantees that most of the reported cases happen to unvaccinated people - even if the vaccine were useless. Assume weekly surveillance testing last year found a run rate of 100 cases. A good working assumption is that half the Covid-19 cases are asymptomatic so 50 asymptomatic cases. Now this year, assume 80% on campus are vaccinated, and one's vaccination status is independent of infection risk. Then if the run rate stays the same, then 80 cases will be among the vaccinated subpopulation while 20 cases will be among the unvaccinated. However, testing policy has changed so that all 20 unvaccinated cases will be detected while only the very sick get tested among the vaccinated. Conservatively, we assume 10% of infections become severe. That means 8 of the 80 vaccinated cases will enter the database, accounting for 8/28 = about 30% of all reported cases.

This mix of cases is then turned into a naive estimate of vaccine effectiveness: the unvaccinated has a (70/20)/(30/80) = 9 times higher chance of getting infected. The trouble is that this result is entirely driven by the reporting bias due to differential testing. No assumption of vaccine efficacy was made in the above calculation.

If we add an assumption that the vaccine is 50% effective at stopping infections, then the 80 cases among vaccinated - the run rate from last year - should have turned into 40 cases this year. Then, only 10%, or 4 cases, would be detected because only very sick people present themselves to be tested. In this scenario, the total number of reported cases is lower, and 4 out of 24 reported cases (17%) occur among the vaccinated. The imputed "vaccine effectiveness" (using the naive methodology) shows that the unvaccinated has a (83/20)/(17/80) = 20 times higher chance of getting infected relative to the vaccinated.

A relative ratio of 9 times corresponds to VE of 89% while 20 times is 95%. In other words, bias in the data due to differential testing explains 89% of the 95% naive vaccine effectiveness. The decision-maker operates on the real-world evidence of 95% VE, unaware that the bulk of this number is explained by the misguided differential testing policy.

***

The potential for harm goes beyond counting cases. There have been some reports that claim that vaccinated people who got sick carry even higher viral loads than unvaccinated people.

Once again, I hypothesize that the differential testing policy may explain some if not all of this effect. The average vaccinated person who gets tested has a more severe case of Covid-19 than the average unvaccinated person.

***

The reason why one should resist all biased data collection processes is that it introduces additional factors that can explain some or even all of our outcome metrics, making it harder to prove that the thing we are interested in (vaccination, in this case) is the true driver of those outcomes.

***

What about those colleges that simply require all community members to get fully vaccinated? In this case, they will only test severely sick people (under the assumption that the vaccine is an invitation to the virus.). There are no unvaccinated people to draw comparisons to, and so it may appear that the above problem has been avoided.

Not so. For these colleges are likely to trumpet a comparison of case rates between this year and the previous year, and conclude that vaccinations work. However, there are other differences between the two years. One glaring difference is the amount of testing, which has declined dramatically. Last year's case count included asymptomatic and mild cases (and false positives) while this year's case count don't. Therefore, even in the absence of vaccinations, we should expect the reported case count to fall significantly.

***

I looked up what Cornell and Georgia Tech are doing in terms of testing. Cornell conducts weekly surveillance testing "regardless of vaccination status". This is excellent.

Georgia Tech makes participation in surveillance testing voluntary, which all but guarantees that their dataset is biased by preferentially selecting people who have experienced symptoms or severe illness, and they also say "You may participate in regular testing even if you have been fully vaccinated, but I especially encourage those who have not been vaccinated to get tested weekly," which means the symptomatic bias is more severe among the vaccinated subpopulation than the unvaccinated.