Big Data, Plainly Spoken (aka Numbers Rule Your World)

Book review: Don't Trust Your Gut

Seth Stephens-Davidowitz has a new book out early this year, "Don't Trust Your Gut", which he kindly sent me for review. The book is Malcolm Gladwell meets Tim Ferriss - part counter intuition, part self help. Seth tackles big questions: how to find love? how to raise kids? how to get rich? how to be happier? He invariantly believes that big data reveal universal truths on such matters.

In 2013, I speculated about the challenges faced by big data analysts in this piece for Significance. Big Data has become a real thing in the decade since its publication. Seth's book interests me as a progress report on the state of "big data analytics".

If there is a common core of this style of analytics, as demonstrated by Seth's examples, I'd include the following elements:

a) Big Data is less defined by the amount of data, but by several characteristics I previously labeled OCCAM (see this post).

The data are typically collected by passive observation (e.g. tax records, dating app usage, artist exhibit schedules). Meaningful controls are absent (e.g. no non-app users, no failed artists). The dataset is believed to be complete. The data aren't specifically collected for the analysis (an important exception is the happiness data collected from apps for that specific purpose). Several datasets are merged to investigate correlations.

b) Much - though not all - of the analyses use the most rudimentary statistics, such as statistical averages.

This can be appropriate, if one insists one has all the data, or "essentially" all. An unstated axiom is that the sheer quantity of data crowds out any bias. This is not a new belief: as long as Google has existed, marketing analysts have always claimed that Google search data are fully representative of all searches since Google dominates the market. (Seth's previous book covers his analyses of Google search results.)

c) If the analyst incorporates model adjustments, these adjusted models are treated as full cures of all statistical concerns.

The last few chapters on activities that cause happiness or unhappiness report numerous results from adjusted models of underlying data collected from 60,000 users of specially designed mobile apps. The researchers broke down 3 million logged events by 40 activity types, hour of day, day of week, season of year, location, among other factors. For argument's sake, let's say the users came from 100 places, ignore demographic segmentation, and apply zero exclusions. Then, the 3 million points fell into 40*24*7*4*100 = 2.7 million cells... unevenly but if evenly, each cell has an average of 1.1 events. That means many cells contain zero events. By the magic of model adjustments, all cells have non-zero estimated activities and associated happiness quotients. The estimates in many cells reflect an underlying model that hasn't been confirmed with data - and the credibility of these estimates rests with the reader's trust in the model structure.

I observed a similar phenonmenon when reading the well-known observational studies of Covid-19 vaccine effectiveness. Many of these studies adjust for age, an obvious confounder. Having included the age term, which quite a few studies proclaimed to be non-significant, the researchers spoke as if their models are free of any age bias. Taking this line of logic further, such a belief implies that no one should bother with running expensive randomized clinical trials because a self-selected trial coupled with post-hoc regression adjustments is a perfect substitute!

d) A blurred line barely delineates using data as explanation and as prescription.

Take, for example, the revelation that people who own real estate businesses have the highest chance of being a top 0.1% earner in the U.S., relative to other industries. This descriptive statistic is turned into a life hack, that people who want to get rich should start real-estate businesses. Nevertheless, being able to explain past data is different from being able to predict the future.

A kind of feedback loop is also at play. The average odds of getting rich surely depends on how many people are entering the industry. So if people read Seth's book, and rush into real estate, the data may no longer rank this industry as best.

e) Most of the featured big-data research aim to discover universal truths that apply to everyone.

For example, an eye-opening chart in the book shows that women who were rated bottom of the barrel in looks have half the chance of getting a response in a dating app when they messaged men in the most attractive bucket... but the absolute response was still about 30%. This produces the advice to send more messages to presumably "unattainable" prospects.

Such a conclusion assumes that the least attractive women are identical to the average women on factors other than attractiveness. It's possible that such women who approach the most attractive-looking men have other desirable assets that the average woman does not possess.

It's an irony because with "big data", it should be possible to slice and dice the data into many more segments, moving away from the world of "universal truths," which are statistical averages, said differently. In the last chapter of the book, Seth describes one study that tackles this problem.

***

I wish Seth had devoted more space to explaining model adjustments and statistical issues, and discussing whether the remedies are sufficient. His target audience is clearly people with minimal mathematical training, and I understand his decision not to open such cans of worms. The references at the end should satisfy those interested in exploring further.

As with Gladwell, I recommend reading this genre with a critical eye. Think of these books as offering fodder to exercise your critical thinking. Don't Trust Your Gut is a light read, with some intriguing results of which I was not previously aware. I enjoyed the book, and have kept pages of notes about the materials. The above comments should give you a guide should you want to go deeper into the analytical issues.

I think there is a lot more that can be done with big data, we are just seeing the tip of the iceberg. So I agree with Seth that the potential is there. Seth is more optimistic about the current state than I am.

The role of prior infection in the vaccine trials

In the final analysis of the J&J vaccine trial (link), the study authors suggested that their vaccine benefits those who have previously been infected with Covid-19:

We observed that participants with previous asymptomatic infection (defined by serologic positivity for SARS-CoV-2 N protein and an absence of history of symptomatic Covid-19) can benefit from immunization with a Covid-19 vaccine. In a post hoc analysis, previous infection alone provided 90.4% protection against symptomatic infection, and after administration of Ad26.COV2.S in seropositive participants, 97.7% protection was observed in a comparison with seronegative placebo recipients.

Attentive readers should be raising their eyebrows as they digest this paragraph. The headline vaccine efficacy (VE) reported in this study was 56%. Against this backdrop, the above claim of 98% efficacy for people who have already caught Covid-19 before (aka seropositive) is quite simply astounding.

[The authors, nevertheless, chose their words carefully, as the statement does not actually make a causal claim, but merely promotes such an interpretation.]

***

Let's unpack the underlying data.

The randomized trial design included the usual pair of treatment groups (vaccine and placebo). However, FDA allowed excluding many subgroups post-randomization in the statistical analysis. One such exclusion are people who were seropositive at baseline (actually, it appeared that they may have excluded also anyone who got infected before 14 days after the shot, an additional complication I will ignore in what follows).

We can subdivide the trial participants into four subgroups, as shown in the following table:

Roughly 10% of the vaccine and placebo groups were deemed seropositive (previously infected) at baseline and removed from the headline analysis. The "post-hoc" analysis brings the seropositive subgroups back into consideration.

The headline analysis ignores the first row of the table, and reports on the difference between the case rates between the placebo and vaccine groups in the second row (expressed as a relative ratio). As mentioned already, the VE is 56%, which means that the case rate of the vaccinated was roughly 44% that of the placebo group. Since we are allowed a causal interpretation in the context of randomized trials, the vaccine roughly cuts the case rate by half, from 5.5% to 2.5%.

If the VE were to be 98% instead of 56%, the case rate of 5.5% would have to drop to 0.1%. This is the first place where the paper's aforementioned claim can easily be misinterpreted. Because the VE is expressed as a relative ratio, a 98% drop is not the same as a 98% drop - because the base rates are not equal.

In this example, the base rates cannot be more different. The average case rate for sero-positive (i.e. with prior infection) is only one-tenth of that for sero-negative people. As shown below, on the placebo arm, those with prior infections had a case rate of 0.6% (despite not receiving the vaccine) compared to 5.5% if they did not have a prior infection.

As the study authors acknowledged, natural infection acts like a vaccine with 90% efficacy. Well, they used more scholarly words:

Previous infection alone, in an analysis involving seropositive and seronegative placebo recipients, was found to provide 90.4% (95% CI, 83.2 to 95.1) protection against moderate to severe–critical Covid-19.

Now, where did the 98% claim come from? It turns out the authors were making a comparison between two unlikely subgroups: (seropositive + vaccine) vs (seronegative + placebo).

This comparison is odd in a number of ways. Two conditions are varied at the same time. The 98% VE represents the aggregate effects of the vaccine plus prior infection. The study makes no attempt to assign relative importance between these two changes. In theory, the effect of the vaccine can range from all of the 98% to none of it.

From a practical view, each person either has caught Covid-19 before or hasn't. Unlike the vaccine decision, serological status is typically not a manipulable treatment (unless someone deliberately attends a "Covid party".)

If you're unvaccinated with a prior infection, then your decision is whether or not to get vaccinated. The appropriate comparison is (seropositive + placebo) vs (seropositive + vaccine). This is the first row of the table.

If you're unvaccinated without prior Covid-19, then your decision is - also - whether or not to get the vaccine. The appropriate comparison is (seronegative + placebo) vs (seronegative + vaccine). This question is the primary endpoint in the trial.

The vaccine efficacy for seropositive people is 76% (first row). This is quite a bit higher than the 56% for seronegative people (second row). Remember though that 76% represents a reduction of case rate by 76% - off a base of 0.6% while the 56% represents a reduction of case rate by 56% - off a base of 5.5% so they are not directly comparable values. Besides, at the time of the trial, seronegatives outnumber seropositives 9 to 1.

In glorifying numbers > 90%, the study authors picked the larger case rate from the placebo column and the smaller case rate from the vaccine column to produce a meaningless 98% number. A fuller analysis shows that the vaccine is decently effective for those who have been infected but the improvement is on top of a tiny case rate.

***

If we are serious about learning the effect of vaccination on those who have already been infected, what experiment should we run?

It's straightforward. Recruit people who had previous infections. Randomly divide them into two groups, give one group the vaccine and the other group a placebo. In such a trial, you are effectively comparing (seropositive + vaccine) and (seropositive + placebo). This is precisely the first row of the table shown above.

The only difference is that the sample size of seropositive people was too small during the original vaccine trial. There were simply not enough people with Covid-19 when the trial was initiated.

***

At this juncture of the pandemic, experts have acknowledged that these Covid-19 vaccines do not stop infection or transmission - they say the purpose of these vaccines is preventing hospitalization and deaths.

In the vaccine trials, basically no one with prior infection subsequently was hospitalized for or died from Covid-19. In the J&J trial, only 15 participants out of over 42,000 were both seropositive at baseline, and subsequently counted as cases. Therefore, we have no data on how the vaccine affected the chance of hospitalization or death among the seropositive subgroup (but we know it's not a big number).

The 98% VE number mentioned in the opening quote of this blog is a metric about preventing infection, not severity of disease. Thus, we have been served incongruent messages that (a) previously uninfected people should not worry about getting infected (and only worry about getting severe disease) while (b) previously infected people should worry about getting infected, even though their case rate is 10 times lower than the previously uninfected.

 

More case-counting puzzles

The Johnson & Johnson vaccine is being treated as a third-class citizen due to some safety concerns (link). Interestingly, such concerns did not surface in the "final" analysis of the single-shot Covid-19 vaccine, published in February, 2022 in New England Journal of Medicine (link). In this report, the scientists concluded "Ad26.COV2.S was associated with mainly mild-to-moderate adverse events, and no new safety concerns were identified". Since in the interim report, the FDA said "the analysis supported a favorable safety profile with no specific safety concerns," we were supposed to believe that the J&J vaccine is completely safe.

On the efficacy side, the final report headlined 56%. This number is meaningless unless we know how VE is defined. The 56% number comes from a counting window that starts 14 days after the shot (1D+14). During the interim report, the headline number was 67% (link).

However, these numbers are not comparable for reasons I'll get into in this post. The FDA has allowed pharmas to keep shifting the goal posts so nothing can really be compared like for like. The situation illustrates clearly why statisticians won't give you an answer unless we know all the details about how things have been measured!

***

First, the interim report was conducted when only about half of the participants have reached 2 months of follow-up - note that the largest number on the time axis was 126 days.

In the final analysis, about half the participants have reached 6 months of follow-up - the largest number on the time axis is now about 280.

So, the comparable value from the final report is not the headline VE which concerns up to 280 days of follow-up but the VE at day 126... and that number is 55% (1 - (436-100)/(938-185)). According to Table 2B (final report), the error bar around this number is 20% to 70% so the trial did not enroll enough participants to give a precise estimate of VE.

Harmonizing the follow-up period is not sufficient. Between the initial and final analyses, J&J ditched the requirement that all cases be "centrally confirmed". In the interim report, cases were counted only if a central laboratory confirmed the local PCR test positives. In the final report, cases accrued based on local test results.

***

Another major development is letting the placebo group get vaccinated. After J&J's vaccine was granted EUA at the end of February 2021, all participants reaching 6 months of follow-up were effectively unblinded so that those in the placebo group could get the vaccine if they so chose. Almost everyone did.

Since enrollment ended October 2020, and the data cutoff for the final analysis occurred in July 2021, everyone could have been followed for at least 6 months, if the above plan were implemented to the letter.

But that was not what happened. Some participants were unblinded prior to reaching the 6-month point because they requested getting one of the by-then authorized Covid-19 vaccines.

As a result, only 23% of the study participants had an observation window of at least 6 months. (By design, 100% should.) In fact, half of the participants in this trial had a follow-up period of less than four months. Because the follow-up period was truncated by the placebo participants receiving the vaccination, effectively destroying the placebo group, there will be no further useful data.

Refer back to the case curve shown above, notice that the at-risk population has dropped by more than half by day 126. The further out one went in time, the fewer participants contributed to the incremental case numbers, the wider the error bar. Such dropoff does not happen in a normal clinical trial. 

Those who reached the longest follow-up period are also the earliest enrollees in the trial. In most vaccine trials, the pharmas first enrolled the lowest risk people - in the case of J&J, the earliest batch of enrollees were young people with no comorbidities. So the right hand side of the case curve lacks both validity and reliability.

***

When I wrote about the J&J trial, I pointed out that it was one of the few double-blind trials - the Pfizer and Moderna trials were single-blind. Blinding removes a source of potential bias - a kind of self-fulfilling prophecy in which the investigators or the participants behave in a way that advantages the treatment against the placebo.

We now have solid evidence that blinding did not work well, and confirmation bias was a real concern. To begin with, given that about 30% of Americans have obstinately refused to get the vaccine, we know that this subset is not represented in these vaccine trials. When the placebo groups in these trials were offered the vaccine, almost all of the participants took advantage, implying they were vaccine enthusiasts, not skeptics.

Blinding is supposed to neutralize partisan feelings about vaccines but it didn't work as planned. In Figure S1 (in the appendix), we learned that 1,222 placebo participants "received other vaccine before unblinding", compared to 670 on the vaccine arm. That's 80% more on the placebo arm than the vaccine arm. If the treatment groups were properly randomized, and if participants were properly blinded, they would not know whether they received the placebo, and so the proportion of people who got another vaccine against trial rules should be balanced.

The number of participants who withdrew during the trial was also disproportionately higher on the placebo arm (338 vs 224), about 50% more. These are huge gaps.

***

One of the big mysteries of Covid-19 vaccine trials is what counts as a case. The case-counting window is one variable I've discussed often. But there's more. Most of these early trials were designed to measure symptomatic, PCR-confirmed cases. J&J added a definition of mild, moderate and severe cases, although as I pointed out before, the definition of "mild" disease is meaningless since almost every case has been classified as moderate or severe.

In most studies, including the J&J trial, all participants are given serological tests at regular time intervals. Seropositivity indicates that someone has been infected in the past, and thus, it is presumed that said infections were asymptomatic. This device has been used by scientists to claim the vaccines also prevented asymptomatic infections.

Apparently, J&J now believes that someone without a positive result from either PCR or serological testing can also be infected. This is how I interpret one of the footnotes from Table 1 that corresponds to a concept called "any SARS-CoV-2 infection", which includes:

"undetected cases that were subsequently detected through a positive serological result (according to the clinical severity adjudication committee), which did not count as either symptomatic cases (becqause they were RT-PCR-negative) or asymptomatic cases"

I'm not sure how the adjudicators got wind of the "positive serological result" since if such a result was present, the case should have counted as "asymptomatic", which is a different line in Table 1.

(Also, the severity adjudication committee is a misnomer as the committe members concerned themselves here with asymptomatic infections that would be milder than "mild".)

What is shocking about this line of numbers is the magnitude. There were 1,038 such infections in the vaccine group and 1,699 in the placebo group. These two numbers are much higher than the counts that made it to the headline VE calculation. In particular, only 42% of the vaccine infections, and only 52% of the placebo infections, ended up as "cases". This calculation also shows that an "any" infection on the placebo group has a higher chance of being counted as a case than an "any" infection on the vaccine group.

 

 

 

 

 

 

 

 

 

The fun and frustration of seeking context

This is a third post related to an exploratory analysis of electricity consumption data from California, published at StackOverflow. The previous two posts are here and here.

***

Exploratory analysis always contains twists and turns. The linear presentation shown at StackOverflow likely reflects an ex-post simplification that dropped deadends, U-turns, etc.

There are two points in this journey at which I'd have taken a different turn.

The first moment relates to the peak consumption day in 2020:

The analyst discovered that on August 19, between 4 and 6 pm, there was a peak in usage. The analyst then looked for context (a good idea!). This leads to the following sentences:

A quick Google search for “California Aug 19th 2020” shows that the region was suffering from wildfires, so perhaps people kept windows closed and the AC on instead of opening their windows to the cooler nighttime air. September 6 also shows up among the highest values, and a search indicates a likely cause: a record-breaking heat wave in California that hit the national news while the fires continued to burn.

If one wants to make this a key conclusion of the analysis, much more work is needed to validate the hypothesis. Heatwaves and wildfires are events that affect a region for a duration of time. It would be interesting to see the trend in energy use in that whole time window. Also, a top N google search is risky when trying to explain extreme values. Extreme values are frequently caused by extreme events, and it's not clear that such an extreme event may be picked up as one of the top N by search engines.

It's the phrase "quick Google search" that doesn't sit well with me. This part of the analysis moves from descriptive insights to cause--effect, and a "quick" search is not going to do the job!

***

The second moment at which I'd have taken a different turn concerns this scatter plot:

The analyst has now switched to a Texas dataset. The picture plots the relationship between electricity consumption and temperature. Each dot in the scatter plot is an hour.

I have seen a lot of similar analyses and they all suffer from an ecological fallacy. It's the problem of over-aggregation, ironically, it's the topic of the StackOverflow blog post, but on the spatial dimension rather than the temporal dimension.

Each temperature value is the average temperature across all of Texas, and each energy consumption value is the average usage across all Texans - for a specific hour of a specific day.

If there is a pattern that affects all (or most) of Texas localities in the same direction (and similar magnitude), then this chart will faithfully show the pattern. However, most interesting patterns are likely going to affect different localities in different directions/magnitudes, and so the above chart would have "aggregated away the signal", to use their words.

Just imagine a single dot which is a single hour of a day, and focus on the temperature dimension only. At any moment in time, the dot represents the average temperature across all weather stations in Texas, but the variability of temperature would be quite large. Some localities will experience higher than normal temperatures while others will show lower than normal. The weights of these localities on average Texas energy consumption is a factor lurking behind these numbers. The point is that the above scatter plot is not useful.

The lesson of the StackOverflow blog post is that instead of this aggregated scatter plot, the analyst should disaggregate the spatial dimension, and look for interesting patterns there.

 

How definitions drive outcomes

 

The planning phase of analysis does not get half the attention it deserves. In reviewing the final report of J&J's vaccine trial, I found an example of how definitions have definitive impact on outcomes.

The one-line conclusion of the report was "A single dose of Ad26.COV2.S provided 52.9% protection against moderate to severe–critical Covid-19." Unlike the Pfizer and Moderna trials, the primary endpoint for J&J has always been stated as "moderate and severe--critical" cases. This outcome lends itself to the narrative around vaccines not necessarily preventing infections but protecting against severe disease.

Here is an excerpt of Table 1 from the paper:

The one-line conclusion is displayed in the first row of the table, right column. To be specific, this endpoint is adjudicated, symptomatic, PCR-confirmed Covid-19 cases that were classified as not mild occurring at least 28 days since the vaccination.

The second section of the excerpt show how cases are classified by severity: mild, moderate, and severe--critical.

Surprisingly, "mild" cases constitute <2% of the cases. Thus, the distinction between "mild" and "not mild" is a distinction without a difference. Given our lived experience that the vast majority of Covid-19 cases are mild, this statistic suggests that the definition of "mild" is mistaken.

This misspecification is non-trivial, especially if one believes the hypothesis that the vaccine reduces disease severity. With this definition of severity, an analysis including or excluding "mild" cases will yield the same conclusion. If many more of the "moderate" cases were reclassified as "mild", we would expect the adjustment to affect vaccine and placebo arms differentially, and thus the VE would be different.

***

In my view, the result of the J&J vaccine trial should be read as protection against disease of any severity. This brings the endpoint in line with those used by Pfizer and Moderna.

Finally, if the FDA wanted to demonstrate that these vaccines reduce disease severity, they should have required different endpoints that directly measure it.

 

Open questions of the pandemic, part 3

As we head into the third year of the novel coronavirus pandemic, I've been reviewing some questions that are still open. The last two posts can be found here and here.

Today's further questions are inspired by the following chart put out recently by the White House.

On the surface, this chart provides evidence that the Covid-19 vaccine may protect people from getting hospitalized. If you're serious about causal analysis, you'd say all it depicts is a correlation. For the above chart, which plots "real-world" data, cannot be interpreted as if the data came from an RCT. In other words, the two groups (vaccinated and unvaccinated) are not identical except for vaccination status.

6. Why are unvaccinated people being hospitalized at an accelerating rate?

Focus on the blue line for the time being. This line shows the hospitalization rate for unvaccinated people, expressed in number of hospital admissions per 100,000. Between start of July and end of November, 2021, the weekly hospital admissions of unvaccinated people exploded from 6 per 100,000 to 68 per 100,000. Those are rates not counts.

This seems strange to me since I'd have expected the VE to come from a sharply reduced rate in vaccinated people while the rate among unvaccinated remains steady.

During that same period, the proportion of Americans who became fully vaccinated grew from 50% to 60%. The vaccines are said to be extremely effective, which should lower the amount of virus circulating in the population. Further, the unvaccinated proportion shrank from 45% to 30%, which is a 33% reduction rate!

The same dynamic shows up on the deaths chart as well. In other words, as more and more Americans get two doses, and fewer and fewer remain unvaccinated, the death rate among the remaining unvaccinated has skyrocketed.

Why? Why? Why?

7. How come VE is increasing while immunity is waning?

The definition of vaccine effectiveness is the relative ratio of the event rates of vaccinated versus unvaccinated. Events may be cases, hospitalizations, deaths, etc. Disregarding many methodological issues that I frequently discuss on the blog, I'm taking their analysis at face value. VE when measured by hospitalizations is then the ratio of the two curves shown. What is very odd about the chart is that the value of VE fluctuates wildly from week to week.

As a reminder, this is the overall trend of hospital admissions in the U.S. in 2021:

Hospitalized cases tumbled to a seasonal low in July, and then started rising again. In the first half of the year, the media kept reiterating that the vaccines were directly responsible for the dropping hospitalizations. Later, they said lower immunity against Delta and/or waning immunity was the reason for the return of cases. This narrative implies that the VE would be highest in July and dropping as the cases rose in the second half of 2021.

Surprise, surprise. If we compute the ratios of the rates shown in the White House chart, VE in July was the lowest (87%) and as the hospitalization rate of unvaccinated jumped 10 times, VE nudged up from 87% to 94%.

Why? Why? Why?

***

Any seasoned data analyst can tell you the very first step to good data analysis is asking the right questions. In this current series of posts, I show that the media have not been asking some of the most difficult questions. We can't pretend that we understand what's going on, unless we confront these challenges head on.

The Merck Covid-19 pill: is the result statistically significant?

Two weeks ago, the FDA advisory committee narrowly voted to recommend authorizing the sale of Merck's Covid-19 pill (Molnupiravir or MOV). This is a rich story worthy of a number of posts. We start with the final result and work our way backwards.

***

Molnupiravir is said to be effective at reducing the risk of hospitalization or death of Covid-19-infected people at high risk of developing severe disease. The treatment program spans five days, two 800-mg pills per day. The study was set up as a randomized clinical trial with ~1,550 participants, randomly divided into two halves, one taking MOV pills while the other ingesting placebo pills. The primary endpoint was the difference in event rate between the two arms. An event is either hospitalization or death. Each participant has been followed up for at least 28 days (or until an event occurred).

At the end of the study, the event rate was 10% on placebo and 7% on MOV. The difference in event rate is -3%, meaning that for every 100 people taking the pills, 3 fewer people will be hospitalized or dead, on average. For each person to benefit, about 33 must take the pills. (This number, known as NNT or numbers needed to treat, looks bad because 90% will not develop severe disease whether or not they take MOV.) Note also that for every 100, 7 will be hospitalized or dead despite taking MOV.

The above result has been reported as 30% efficacy, which is the relative event rate ratio (1-7%/10%). I prefer using differences as they don't exaggerate the impact off a small baseline. This primary endpoint is not statistically significant (but it's being portrayed as significant; I will get to this point later).

Molnupiravir may have very serious side effects. None of these effects have surfaced in the trial but the trial design contained so few participants that it can only find very common effects. Only about 700 people got treated with Merck's pills. An event with risk of 1 in 1,000 has almost 50% chance of not registering a single case in 700 people. An event with risk of 1 in 10,000 has over 90% chance of not showing up in this trial.

The FDA Briefing document that was reviewed at the FDA meeting on November 30, 2021 (link) describes several potential adverse side effects, all found in earlier animal and lab studies. These include bone marrow toxicity, mutagenecity, and reproductive toxicity.

***

How this type of treatment can be put to clinical practice should have been thought through before running this experiment, but evidently, it wasn't.

Rolling out the treatment requires an accurate prediction of who are at high risk of developing severe disease. Just like we don't know what level of antibodies is high enough to prevent infection, we don't know what factors predict disease severity. During the trial, Merck went with the usual list of "risk factors" for hospitalization or death, such as older age and obesity. This just means most patients who are hospitalized with or dead from Covid-19 are older and/or obese, but that's not the same as saying most older or obese patients will develop severe Covid-19.

What we do know is that the patients present in the Merck trial definitely had unusually high risk of being hospitalized or dying. The 10% baseline event rate meant that 1 out of every 10 Covid-19 patients enrolled in the trial ended up hospitalized or dead. This rate is over 100 times higher than the unvaccinated rate of hospitalization in the U.S., according to CDC numbers cited in the FDA briefing doc.

In real life, how does one find a comparable set of patients who would have the highest chance of benefiting from these pills?

**

The FDA document also discloses a number of other headaches - all of which are direct results of the design decision to exclude certain groups from the clinical trial, namely, vaccinated people, pregnant people and immunocompromised people.

The design decision is odd because vaccinated people constitute the vast majority of Americans, and the other two groups are apparently at higher risk of developing severe disease. Instead of oversampling them in the clinical trial in order to measure its effects, they were excluded completely from the trial. Strictly speaking, there is no evidence that these pills work for anyone who has had at least one dose of Covid-19 vaccine. This meant that the FDA advisors are asked to take part in a game of guessing without any actual data to support their opinions.

In the trial, those who were seropositive at baseline did not benefit at all from taking the Merck's pills (5 events vs 2 in placebo - nice example of why I dislike relative ratios), indicating that MOV might not work for vaccinated people who have antibodies. The FDA analysts cast doubt on this result, arguing the sample size was too small - and yet, they have no trouble saying that the pills appear to work equally well across age groups, and other variables - when these analyses contain similarly small samples. (e.g. 136 who were seropositive vs 118 who were > 60 years old)

Contrast statement (a): "Given the small size of the seropositive at baseline subgroup and the small number of events in this subgroup, robust conclusions regarding MOV efficacy in seropositive participants cannot be made"

with statement (b): "Efficacy results were generally consistent across subgroups including age (>60 years), at risk medical conditions (e.g., obesity, diabetes), baseline COVID-19 severity (mild, moderate) and SARS CoV-2 variants"

This is the kind of flexible interpretation of statistical significance that frustrates your statistics professor. You either believe none of the subgroup analyses should be considered because the trial was not designed to have enough samples to read them, or you believe they should all be trusted.

***

Another place in which statistical significance was mishandled was hinted at earlier. The main result is a 3% reduction in event rate, which has a 95% confidence interval of 0.1% to 5.9%. Notice that the CI apparently missed 0% by a pip - if the CI crosses 0%, the result is statistically insignificant, leading to the conclusion that the trial did not generate sufficient evidence to prove that MOV was effective at changing the event rate.

But is it barely significant or not significant? Here are two additional issues for your consideration:

What if there were one fewer hospitalization counted in the placebo group (67 instead of 68 events)? This tilts the confidence interval to the other side of zero, making the result statistically insignificant. Talk about hanging on a thread.

Now, in Table 1 (footnote) of the FDA Briefing Doc, we learn that "participants with unknown status at Day 29 are counted as having an outcome of all-cause hospitalization or death in efficacy analysis". Said in English, this means that anyone for whom they didn't know the status by Day 29, they assumed to have been hospitalized or have died. I highlighted this line in red because this seems out of the ordinary. Usually, we assume the opposite (presumed alive vs presumed "dead").

How many people fell into the bucket of unknown outcome, turned into hospitalized or dead? Exactly one, on the placebo arm. Thus, if they had made the opposite assumption, that the unknown person was not hospitalized or dead, then the statistical significance flips!

***

The above discussion of statistical significance assumes a normal analysis schedule in which analysts perform a full analysis when the entire study population has passed the full follow-up period of 28 days. That was not what happened.

In many clinical trials, researchers are allowed to peek at early results, known as interim analyses. In this Merck study, they performed an interim analysis at the half-way point, when half the participants have completed 28 days of follow-up. (This explains the two press releases from Merck about the same trial.)

A problem with peeking at interim results is that it increases the chance of false-positive readings (i.e. wrongly declaring significance). To compensate for this risk, the required level of significance must be reduced (more stringent). We encountered this issue before when looking at Moderna's vaccine clinical trial analysis plan. (Link)

Because of early peeking, the required significance level is not the standard 5% but lower than 5%. A different way of saying this is we shouldn't look at the 95% confidence interval but something higher than 95%, which means the width of the CI is larger than the reported 95% confidence interval. In other words, if the proper threshold was applied, the reported 3% reduction is not statistically significant.

They managed to come up with a justification to overturn the usual statistical practice: "Formal statistical testing was not performed for the full population assessment, because statistical significance was demonstrated at the interim population assessment. The nominal 1-sided p-value was 0.0218."

This is yet another statement that drives your statistics professor mad. Firstly, this statement literally embodies the statistical fallacy of "testing to significance", which I explained here. Secondly, even though they didn't do formal testing, they issued an "informal" testing result, showing a p-value of 0.02, suggesting that the testing result would be significant (under the conventional of p-value < 0.05 unadjusted for early peeking). Thirdly, they showed a one-sided p-value, which makes no sense in this situation; a one-sided analysis can only be justified if there is little doubt as to the direction of the outcome, and is almost never used in clinical trials. The "nominal" two-sided p-value is over 0.04, and likely over 0.05 when adjusted for early peeking.

The situation is worse when you compare what happened in the first and second halves of the Merck study:

There are rather disturbing signals on this table. The interim analysis showed an improvement of 7% over a placebo event rate of 14%. The final analysis showed the improvement to be 3% over a placebo rate of 10%. This meant that the Merck pills performed very badly in the second half of the participants: 6% event rate vs placebo event rate of 5%.

Thus, in this particular experiment, the first half of the result is contradicted by the second half. The FDA analysts endorsed Merck's argument to ignore the second half because "statistical significance was demonstrated at the interim population assessment".

If this were my experiment, I'd investigate whether there were operational issues (or enrollment biases) that caused the placebo event rate to go from 14% in the first half to 5% in the second half, a three-fold change. (By contrast, the event rate of the treatment arm only changed slightly.)

The phenomenon can just be regression to the mean, though. That's why classical statisticians don't like early-stopping strategies. If this trial was stopped for efficacy, and the second half was not analyzable, we would have over-stated the effect by double.

***

I've exceeded the usual post length again so further comments will appear in next posts.

 

 

Further adventures of case-counting windows

If it's not already clear, observational data is a great playground for statisticians. We sniff out where biases are hiding, and we devise methods to correct such biases. It's a lot of fun. It's also frustrating - because there are endless ways to make adjustments.

In today's post, I examine a couple of curious observations made by the Israeli researchers when they analyzed the data from vaccine boosters (link).

***

The researchers shrank the follow-up window when counting severe cases (relative to counting cases)

This decision surprised me because severe cases are a fraction of all cases so I'd want to have a longer counting window, not a shorter one.

Recall that in the analysis of cases, the Israel booster real-world study pulled data from the national database for the month of August. Because of a case-counting window that starts 12 days after the booster shot, in effect, the study only counts cases from August 11 to August 30. Most people did not get the booster on July 31st, and so the average follow-up time is probably around 10 days. (See the previous post for more.)

For severe cases, the case-counting window shrank by 4 days, as the researchers stopped counting on August 26 instead of August 30. Thus, the average follow-up time for severe illness is around 6 days. You're right - protection against severe illness was shown for less than a week.

Here is a direct quote that explains why the case-counting window was shortened for severe cases relative to all cases:

In order to minimize the problem of censoring, the rate of severe illness was calculated on the basis of cases that had been confirmed on or before August 26, 2021. This schedule was adopted to allow for a week of follow-up (until the date when we extracted the data) for determining whether severe illness had developed.

In the Appendix, the researchers offered the following chart as supporting evidence:

This chart shows the lag between testing positive and the onset of severe illness for Covid-19 cases between November and March 2021 (conditional on people subsequently getting severely ill). The median lag is 4 days, meaning among those who eventually got severely ill, half got severely ill within 4 days of testing positive, and half got severely ill after 4 days (up to almost 30 days).

I believe the logic is that for anyone who got sick in the final 4 days of August and became severely ill, the onset of severe illness would have taken place after the data cutoff for this study.

Consider the following implications:

• According to the chart above, half of the past patients took over 4 days and up to 30 days to develop severe illness. If we want to capture all severe cases, we ought to extend the follow-up period to at least 30 days - it should be 60 days if we use a one-month cohort. The methodology based on median lag time would capture only half of the severe cases.

 

• Think about the severe cases that occurred between Aug 26 and Aug 30. These severe cases are connected with infections that occurred prior to August 26, most likely during August. They are excluded from the analysis. All of these severe cases should count. The severe cases that happened on the first 4 days should be dropped, if we use their logic. (Note that the infection rate was skyrocketing during August.)

 

• Now think about someone who took the booster shot on August 16. If this person became severely ill on August 26, the severe case is not counted. But this severe case represents a lag time of 10 days from detection of infection, which according to the chart above, happens more than 10% of the time. This issue arises because the case-counting window is given a fixed ending date regardless of when the booster was administered.

***

The researchers disclosed they faced an incurable imbalance problem.

Part of the paper's Appendix is devoted to alternative analyses, one of which is a matching study. I discussed examples of matching studies before (Mayo Clinic, and Clalit). Unfortunately, this disclosure did not contain enough details to evaluate the quality of matching.

The following statement caught my eye: "Due to the very small number of severe cases, and the high censoring proportion, calculation for severe COVID-19 was not possible."

In my comments on the Clalit study (link), I mentioned a limitation of matching studies: if selection bias is severe, it may be hard to find acceptable matches, leading to dropping many unmatched cases ("high censoring proportion"). In this case, the matched population ended up being much younger than the vaccinated population.

In the Israeli booster study, the researchers attempted to find one person who hasn't taken the booster to match with each person who took the booster. Matching variables are the usual suspects like age group, second-dose date and so on. They appear to be saying that they failed to find adequate matches for a large proportion of the booster group. This is a sign of incurable covariate imbalance. This also suggests that those regression adjustments used in the main analysis are insufficient.

 

 

 

Boosters in Israel illustrate statistical issues with real-world effectivness analysis

I previously discussed the evidence that Pfizer submitted to the FDA to get approval for booster shots (here and here). The studies were small, and circumstantial. It appears that the experts were partially or completely swayed by "real world" data coming out of Israel so I looked at what is in that influential Israeli study (link to report).

The story of vaccinations in Israel is one of high hopes unfulfilled. Here is their full case curve for 2021. 

Between April and July, it appeared that the fast vaccination campaign yielded dividends; public health officials were satisfied that the vaccines beat Covid-19. Like other countries, Israel moved to remove most of the mitigation measures. Not so fast, as in July, cases started to tick up again. The rate of growth was so alarming that they became the first country to approve booster shots. Other countries, including the U.S., were curious what the outcomes would be - since the vaccine developers were not going to run proper randomized experiments to prove the value of boosters. Even after administering boosters, the case curve didn't start bending till late September. To this day, despite the general acceptance that vaccines are effective, no one can reliably predict what proportion of the population needs to get injections before infections start to decline, or how much time it would take to start seeing cases come down.

***

This Israel study (link) was published in October, and only concerned one month of data (August 2021). Approximately 1.1 million Israelis were fully vaccinated by July 30 and formed the study population. However, we do not know how many of those subsequently took the booster shot. This is one of those key omissions that have pervaded all the vaccine studies - basic check-off-the-box data that are withheld from the scientific reports.

The data came from the Ministry of Health. Data cutoff for cases was Aug 30, 2021, and these researchers invented a new case-counting window that starts at 12 days after the booster shot. Since the first booster shots analyzed by this study happened on July 30, that means no cases are counted until 12 days later (Aug 11, 2021). For those who took the booster on July 30, the study has a maximum follow-up window of 20 days. The graph above shows a linear growth in administered boosters during the study period. For someone who took the booster on Aug 15, the case-counting window opens on Aug 27 and closes on Aug 30, thus the follow-up period was 3 days. Anyone who took the booster after Aug 18 are "immortal": if they get sick after 12 days, those cases won't feature in the study data. 

This little calculation reveals that the follow-up period for the booster group is between 1 and 20 days, and that the study population is effectively three weeks, 12 days fewer than advertised (people who received boosters in those 12 days have not reached the beginning of the case counting window). That means, if we believe everything else in this study, the researchers have shown that the booster shot is effective for an average of 10 days.

The headline result of the study is that those who took the boosters are 11 times less likely to get infected than those who didn't take the boosters (note the unspoken: effective for 10 days, after ignoring anyone who gets sick during the first 12 days). For severe illness, the factor was 20 times.

As explained in my previous post, the use of these multiplicative factors (x times) is a fad. They could have expressed the same data in the usual way: the booster effectiveness was 90% for cases, and 95% for severe cases. In other words, they are claiming the same effectiveness as the original two shots.

The above calculations were "adjusted" by including main effects for age group, gender, calendar day, timing of 2nd dose, and "sector" (Arab/Jewish/Ultra-Orthodox).

***

A vaccine effectiveness metric is a relative ratio between the (severe) case rate of the treated group and that of the untreated group. In the original vaccine trials, the treated group took two shots while the untreated group took the placebo shots. In the real-world booster study, the treated group took the booster shot while the untreated group didn't; both groups previously received two doses of an mRNA vaccine.

The VE metric is merely a correlation between cases and vaccination status. In order to move from correlation to causation, statisticians want to establish that the booster group and the no-booster group are comparable; that the observed difference in case rates isn't driven by factors other than vaccination.

To understand the nature of this challenge, let's pretend that the FDA required vaccine developers to conduct a randomized controlled trial (RCT) to determine the causal effect of the booster shot on cases (The FDA didn't require it.)

We would have recruited participants from the group that has previously took two vaccine shots. Then, they are randomly divided into two groups: booster, and placebo. All participants are then followed for a period of time, and at around the two-month time point, an interim analysis is conducted. This analysis is simple, because by virtue of random assignment, the two groups have co-variate balance (same mix of age groups, genders, occupations, date of second doses, etc.), and thus, any variation in case rates is due to the booster.

Instead, we have "real world data". On the surface, we have two groups: booster and no-booster, which sounds similar to booster and placebo. But labels deceive. The lack of a random assignment device means the two groups are not apples to apples. Note that public health policy prioritizes high-risk groups, and some people have reasons to want early boosters (e.g. work on the front lines, want to stop wearing masks).

I find it rewarding to think about the study population in terms of three subgroups (bottom row of the left chart).

Start with anyone in the study population on July 30, the start of the study period. This person is in the no-booster group - until s/he decides to get the booster shot, at which point the same person migrates to the booster group. There are some who never took a booster during the month of August so they remain in the no-booster group throughout the study.

Let's consider the following three subgroups:

(NB1) People who never took a booster during August

(NB2) People who received a booster during August - they belonged to the no-booster group until the day of booster

(B) People who received a booster during August - they migrated to the booster group on the day of the booster.

(NB2) and (B) are "twins". They are the same people, counted twice, the first time as a member of the no-booster group, and the second time as a member of the booster group.

Thus the no-booster group (NB) in the study consists of (NB1) and (NB2) while the booster group is just (B). Now, let's address whether (NB) can be compared to (B).

***

Consider first (NB2) vs (B). These are twins, they are the same people counted twice. Thus, by definition, we have 100% co-variate balance, if we consider the typical demographic co-variates, such as age, gender, and occupation. Even date of second dose is also perfectly balanced since that's a variable about historical behavior prior to the study.

Nevertheless, (NB2) differs from (B) in pivotal ways. The exposure time windows are disjoint. The (NB2) twin always have earlier exposure than the (B) twin. Besides, the length of exposure is negatively correlated: the longer the follow-up time for the (NB2) twin, the shorter for the (B) twin. Thus, the booster status variable is confounded with exposure time and duration - and this is a hopeless confounding, 100% confounded.

Next, compare (NB1) with (NB2). Are the two partitions of the no-booster group similar? The answer is a resounding no. (NB2) are those who got the booster in August while (NB1), not in August. Unless one believes that the timing of the third dose is chosen completely at random, one must conclude that those two subsets are dissimilar due to selection biases.

In addition, (NB1) invariably has longer follow-up time than (NB2), and by extension, (B). The start of the follow-up window is fixed for everyone in (NB1) and (NB2) but variable for those in (B).

To summarize, (B) is identical to (NB2) on many co-variates due to twinning but differs from (NB2) pivotally on exposure metrics. (NB2) is different from (NB1) due to selection bias, thus these two subsets do not have co-variate balance; they also have different exposure durations.

When we compare (B) with (NB), we may be tricked into thinking we have good co-variate balance because of the (NB2) component. This statistical deception intensifies as more people get boosters. The exposure-related complete confounding is, to my knowledge, impossible to cure. Regression adjustments via main effects are insufficient to solve these complex statistical problems.

The differences between (B) and (NB) - in addition to booster status - include the exposure-related confounding with (NB2), the selection biases with (NB1), and the exposure-related bias with (NB1). All three are affected by the proportion of people who have taken booster shots.

Any serious causal analysis of real-world booster data must offer adjustments that deal with the above statistical issues.

P.S. [11/9/2021] In the following post, I looked at a couple of observations from this study that caught my eye.

 

Swear and gossip: answer to question 1

Last week, I posed two questions related to the Business Insider headline about swearing and gossipping in the workplace (link). Today, I provide my answer to the first question. The answer to the second question is here.

Question #1: Are the following headlines identical (i.e. interchangeable)?

Answer: They are not identical.

More precisely, the two statements can be identical under limited conditions but they are usually not interchangeable. As Scott quipped on Twitter, it depends.

A reason why we may think those headlines say the same thing is that people tend to draw the same causal conclusion from each headline: if team members swear and gossip more, their chance of success increases.

We can unpack the statements. The key difference between them is the order of presentation. The first statement starts with the base of teams that are rated as the most successful by some metric of success, and within this base, there is a high incidence of swearing and gossiping. The second statement starts with the base of teams that engage in swearing and gossiping, and within this base, there is a high incidence of success.

For those familiar with conditional probabilities, the first statement concerns the probability of swearing and gossiping given that a team is "most successful" while the second statement is about the probability of success given a team engages in swearing and gossipping. In general, P(X/Y) is not the same as P(Y/X).

According to the famous Bayes' Theorem, the ratio of those conditional probabilities is equal to the ratio of the prior probabilities so the two statements are identical if and only if the rate of success among all teams is the same as the rate of swearing and gossipping. P(swear and gossip) = P(success) is an unlikely event in real life.

***

Let's bring back the helpful diagram from the answer to Question #2, that lays out the four possible characterizations of any team.

Consider an extreme case in which all teams are either in A or D. Every team that swears and gossips is successful while every team that doesn't fails. So B and C both are empty.

In this case, A+B (teams that swear) and A+C (teams that are successful) are identical. The prior probabilities, and also the conditional probabilities are the same. We conclude that the most successful teams swear and gossip, and also that the teams that swear and gossip are successful.

In the above diagram, the conditional probability of swear and gossip given success is A /(A+C) and that of success given swear and gossip is A/(A+B). For those to be identical, B and C must contain the same number of teams. In the extreme case, B = C = empty. If we put x teams in B, then we must put x teams in C, and then we have to subtract x teams from A and from D to make things even.

There are a small number of scenarios in which this can happen, and a much larger number of scenarios in which the two probabilities aren't the same.

***

Consider the two statements in this format:

Formula (I) comes from the original headline while formula (II) corresponds to the other headline.

Formula (II) predicts success (... leads to success) while formula (I) describes success (a successful team does ...). Formula (II) does not imply formula (I).

In the extreme case, every team swears and gossips, and so it follows that the most successful teams swear and gossip. Since everyone does it, swearing and gossiping will not predict success. Formula (II) does not imply formula (I).

In order to learn Formula (II), we must add other factors that affect team success.

(In the above, I excluded the possibility that success may cause swearing. Allowing this relationship complicates the causal interpretation even more.)

***

The confusion between descriptive and predictive (or causal) statements is endemic. For example, in coronavirus news, we keep hearing statements such as "Most of the hospitalized people are  unvaccinated." This is very different from the related statement of "Most of the unvaccinated people are hospitalized."

If we think of vaccination status as a cause of an effect, and hospitalization as an outcome, then "Most of the hospitalized people are unvaccinated" is a description of people with a given outcome. It is not a predictive statement supporting a claim that vaccination status predicts some outcome. In addition, the first statement does not imply the second statement - that "most unvaccinated people are hospitalized."