Big Data, Plainly Spoken (aka Numbers Rule Your World)

Note on a Simpson's paradox in real-world vaccine effectiveness studies

You have heard it in the media: real-world studies of high-risk people such as nursing home residents or healthcare workers offer definitive proof that the vaccines' true efficacy is <insert a percentage above 90%>!!

These were the people who got the vaccines first, they are at greatest exposure to the virus, more of them are fully vaccinated (two shots), they have the highest risk of death, they are easier to track down. Ergo, the result is rock-solid, beyond criticism!

I wish someone had the foresight, and did a quick randomized trial while delivering vaccines to these groups, then we'll have a good dataset to draw some useful conclusions.

Alas, no one did (as far as we know) - which means all these studies are real-world observational studies. I discussed two examples of these studies (Mayo Clinic, Israel's Clalit health fund), both analyzing everyone in their customer database using the matching technique. Matching is used to create a synthetic control group (after the fact) in the hope of emulating a randomized control group as in a randomized, controlled trial (RCT). After matching, we apply methods commonly used to analyze RCTs, as if the conditions of RCT have been satisfied.

Today, I'm examining a Danish study of the Pfizer vaccine in two high-risk groups, nursing home residents, and healthcare workers. (link). This is an example of those real-world studies of high-risk groups that have so excited the punditry.

Counting People No Longer Works

Let's start with a simple calculation, and I'll explain why it fails. The study period is from December 27, 2020 to February 18, 2021, just under 2 months starting from the day Denmark initiated vaccinations. Based on the Figure 1 at the back of the paper, I estimated that during this period, there were 1,675 confirmed cases in Denmark among the nursing home population. The researchers said 382 cases were discovered among the vaccinated nursing-home residents, so there were 1,294 cases confirmed for unvaccinated residents. (The actual numbers used in their calculations are slightly lower because of exclusions but this doesn't matter for what I'm about to say.)

During the study period, 37,172 nursing home residents got at least one Pfizer shot so the case rate for vaccinated is 382/37,172 = 1%. For the unvaccinated group, I use the total number of nursing home residents who have not been infected as of December 27, which was 39,050. So the case rate for unvaccinated is 1,294/39,050 = 3.3%. The ratio of those case rates results in a vaccine effectiveness (VE) of 70%.

This calculation sounds vanilla but has a serious problem. Each nursing home resident who is counted as vaccinated is also counted as unvaccinated - because on December 27, 2020, the first day of vaccinations, everyone is unvaccinated. Then, over the course of time, those who have not been infected are eligible to get vaccinated. Upon vaccination, the nursing home resident transitions from being counted as unvaccinated to being counted as vaccinated. By the end of the study period, only about 5% of the residents are still unvaccinated. So 95 percent of this population are counted twice, and in each case, the persons spend the first part of the timeline as an unvaccinated and then the second part of the timeline as vaccinated.

Moreover, the proportion of time spent as unvaccinated varies depending on the day of vaccination. To deal with this issue, we change the denominator of the case rate. Instead of cases per person, we use cases per person-week. Each person is still counted twice, but in any given week, a specific resident is counted just once, either on the vaccinated arm or the unvaccinated arm but not both.

There were roughly 148K person-weeks accumulated on the vaccinated arm, and 164K on the unvaccinated arm, leading to new case rate estimates of 0.26% and 0.79% (interpreted as per 100 person-weeks). This leads to VE of 67%.

(In the second half of this post, I'll show why this 67% number is compatible with a completely useless vaccine. Stay tuned.)

Matching No Longer Works

For any real-world study, we ask why we can believe that the group labeled unvaccinated and the group labeled vaccinated are identical except for their vaccination status. In prior studies I featured on this blog, the researchers rely on the matching technique to assert this claim. Matching is infeasible here because the entire study is centered on two high-risk groups, for whom the progress of vaccination was rapid. Among the nursing home residents, about 5 percent were vaccinated during the first week, reaching almost 95% by the end of the study period.

The Danish researchers treated this issue differently from the Clalit researchers.

In the Israeli study, the investigators matched each vaccinated person with an unvaccinated one on a rolling basis through the study period, and they ended follow-up on both members of each pair when the unvaccinated person received the first shot. This matching and censoring scheme creates many undesirable side effects (for details, see my prior post):

• the follow-up time is severely limited because most matched, unvaccinated patients quickly became vaccinated, stopping the count on both members
   
• cases on the vaccinated arm are not counted because of the early stopping described above
   
• the number available for matching shrinks rapidly throughout the study period, leaving the majority of high-risk subgroups unmatched, and dropped from the study.

By contrast, in the Danish study, few are dropped from the study (only those who have been infected prior to the first day of vaccination). Each person is followed through the entire study period (until they get infected, died or become untrackable). For anyone who eventually got vaccinated, the person contributes to unvaccinated person-weeks until the day of vaccination, after which the person contributes to vaccinated person-weeks. They don't have extra early-stopping rules triggered by something that happens to an unvaccinated person so every case will be counted (except when they play around with case-counting windows, which I'll cover in a future post).

But there is one big problem with this design, which the researchers recognized and tried to resolve.

Vaccination Timing Bias

Because of the design described above, cases counted as unvaccinated typically occur earlier in the study period than cases counted as vaccinated. If you think about any given individual, the time progression is always from unvaccinated to vaccinated. So earlier infection dates are correlated with being unvaccinated, and later dates with being vaccinated.

This is particularly problematic because vaccinations were happening in the backdrop of a general decline in cases post-holiday, at least partially related to a national lockdown that started in Denmark a week before vaccinations began. To deal with this bias, the Danish study added a "calendar-time adjustment" to their regression model.

Unfortunately, such an adjustment is better than nothing but isn't sufficient. In the rest of this post, I'll walk you through a numerical example to show how this timing bias is a real-world example of what statisticians call a "Simpson's paradox". Then, in a next post, I discuss what a regression adjustment does, why it doesn't get rid of the problem, and also address other residual biases.

Simpson's Paradox

For this demo, I start with an assumption that the vaccine is useless (VE = 0). Then, I show that using the usual way that we compute vaccine effectiveness, we will conclude that the vaccine is extremely effective. This "paradox" is of the Simpson's type, well known to statistics students.

As mentioned already, the Danish study attempts to correct for this issue. This demo explains why they must do something about it. It also provides an estimate of the magnitude of this bias (huge).

As background, we start with some real data taken from Figure 1 of the paper: the progress of vaccinations during the study period, and the weekly trend of cases in nursing homes.

Note in particular that the case rate (gray) dropped markedly over the course of the study.

Assuming VE = 0 means each week, the split of cases between the vaccinated group and the unvaccinated group is equal to the split of people between the two groups. This assumption makes the group-level case rates equal each week. The VE in any week is zero since VE is based on the ratio of group-level case rates. Like this:

So far, so good. Now, we compute a cumulative VE using the standard method. We sum the cases and the person-weeks over all weeks, and then divide the two to get an overall VE for the entire study period. This is exactly the calculation I did in a previous section, which led to a VE of 67%.

I hope your head is where mine is: WTF!

We forced VE to be 0% every week of the study period (by assumption), and yet the cumulative VE for the entire period turns into 67%. This is known as a "Simpson's paradox". The aggregated statistic leads to a startlingly different conclusion than the disaggregated (weekly) statistics.

This is the reason why the Danish researchers introduced a "calendar time adjustment" to their analysis. They realized that they must do something to overcome this paradox.

What's Lurking Behind the Scene

We expected the cumulative VE to be 0% since it's just the average of a bunch of zeroes. The problem is it's not a simple average but a weighted average. The weights reflect the progression of vaccination through the study period.

The unvaccinated group is front-loaded, and shrinks over time while the vaccinated group is back-loaded, and expands. So the aggregate case rate for the unvaccinated group is weighted toward the first few weeks while the aggregate case rate for the vaccinated group is weighted toward the last few weeks. Remember that the overall weekly case rate was high in the first two weeks, and then started declining. This explains why the unvaccinated group with a much higher cumulative case rate than the vaccinated group despite our assumption that the vaccine is useless.

In the above diagram, you see that for the unvaccinated group (pink), the largest weights coincide with the several weeks with very high case rates while for the vaccinated group (blue), the largest weights fall on the most recent weeks with close to zero case rates. Simpson's paradox is a situation in which aggregating the data removes too much information, and distorts the interpretation.

The magnitude of this distortion is extreme! What should have been 0% shows up as 67%.

(In the preprint, I only have the proportion vaccinated at the start and end of the study. In the demo, I drew a straight line, which means I assumed an even pace through those weeks. The reality is that the earlier weeks were probably slower than the later weeks, which makes the distortion even worse.)

***

The situation of rapid vaccination and declining overall case rate are prevalent everywhere so every real-world study, particularly ones focused on high-risk groups, faces this problem.

This analysis does not show the vaccine is useless. It shows that if the researcher does not correct for this timing bias, the estimate of vaccine effectiveness is extremely optimistic.

***

My second book, Numbersense, opens with an example of Simpson's paradox, and then it keeps getting better. Get your copy here.

In the next post, I explore how the Danish researchers attempted to fix this bias.

 

 

 

 

 

 

 

 

 

 

Real-world studies: how to interpret them

Recently, two types of data analyses about vaccine effectiveness have flooded the airwaves.

One type is being termed "re-analysis" of data from randomized controlled trials (RCTs). They used to be called "post-hoc", or "not pre-specified", or "exploratory." Serious analysts do engage in this type of analyses but we treat the findings as purely suggestive, requiring confirmation by another study. It's bad practice to treat them as capable of overturning or correcting analyses of RCTs by the original research team. Beware when the analyst is not arranging further studies to confirm such findings.

Another currently popular type of analysis is labeled "real-world studies." Technically called "observational studies," or "causal studies", these are worth the effort. As noted in my previous post, such studies should not be expected to correct or overturn findings from an RCT either. They are most useful in situations when we cannot run an experiment, e.g. if the US Postal Service wants to measure the impact of a price hike, it cannot randomly apply increases to some citizens and not others.

When we can run either RCTs or observational studies, the "real-world studies" benefit from larger sample sizes, especially for subgroups, and a more realistic context. It's a double-edged sword though. When the investigators cannot control the context, there will be more variability in the study population ("statistical noise"). The observed difference between test and control may be explained by biases, both known and unknown ("confounding").

Imagine an observational study that aims to estimate the real-world effectiveness of the Moderna vaccine. The simplest first analysis is to compare the case rate of the vaccinated people to the case rate of the unvaccinated people. This is hopelessly flawed because in a real-world study, we must not assume "all else equal." People who have received the vaccines at this point are apples and oranges to those who haven't. We already know about racial inequities but the differences go beyond that: geographical, cultural, economic, religious, access to technology, access to transportation, age, comorbidities, occupation, etc. may all explain some of the observed difference between the vaccinated and the not vaccinated.

The point of observational study methods is to statistically adjust the data. In Numbersense (link), I spend quite a few chapters explaining statistical adjustments (e.g. marketing and economic data). Critics complain that "massaging the data" is a horrible thing to do but for those of us invested in these methods, it is the only reasonable thing to do. Not adjusting the data when biases are present is to wilfully publish biased and therefore incorrect results.

In this post, I will walk you through an example of an observational study, conducted by the Mayo Clinic in partnership with nference (link). I chose this study because I'm impressed by its quality, and satisfied with the level of transparency into how they massaged the data.

What a real-world study tells us

Let's start with their headline finding:

They provide a "preliminary assessment of real-world vaccination efficacy in 62,138 individuals from the Mayo Clinic... between December 1st 2020 and February 8th 2021... Administration of two COVID-19 vaccine doses was 88.7% effective in preventing SARS-CoV-2 infection (68.4-97.1%) with onset at least 36 days after the first dose."

To a statistician, this is a heavily loaded statement, with every word chosen precisely, its meaning to be revealed in the rest of the paper. Here are the footnotes to save you time:

1. "preliminary": This is an interim analysis in which the individuals under analysis have not all reached the end of the desired observation window. The length of follow-up differs by individual.
   
   
2. "real-world": it's an observational study in which individuals did not get vaccinated at random, and so we cannot analyze this dataset using methods for analyzing RCTs. What determines one's vaccination status is self-selection plus government prioritization rules.
   
   
3. "62,138 individuals": we did not generalize our finding beyond the analysis set of patients in our hospital system. You are on your own if you should extrapolate our finding outside this group.
   
   
4. "December 1st 2020 and February 9th 2021": see point #3. Ditto for generalizing across time.
   
   
5. "administration" [of two doses]: the 88.7% number was arrived at after removing a few cases from the vaccine arm because those people have not received their second shots despite being past the target time for them. Without removing those cases, the effectiveness was 83%.
   
   
6. "onset at least 36 days after the first dose": as with other research teams, they made their own decision as to when to start counting cases. 36 days after the first dose is roughly 2 weeks after the second dose of Pfizer, and 1 week after the first dose of Moderna. (This just happens to flip the official case-counting windows from their respective RCTs, in which Pfizer started counting 1 week after the second dose, and Moderna, 2 weeks.)

How a real-world study assembles the control group

The key to any observational study is how they define the control group (i.e. the unvaccinated group).

They start with a vaccinated group. These are patients in the Mayo Clinic system who has received at least the first shot of either Pfizer or Moderna during their study period, which was 12/1/2020 to 2/8/2021. Each of these patients has a vaccination date, PCR test dates, and whether each test came back positive or negative. For the moment, we will concern ourselves with just the vaccination date (Day 0, day of the first shot), and any positive test result after Day 0 to indicate infected. 

To form the control group, they start with all patients in the system who has not been vaccinated as of 2/8/2021 (subject to exclusions which I'll talk about later).

The researchers collect demographic data and zip codes on all patients in both groups. They then assign "propensity scores" to each individual. These scores project the probability that an individual of given zip code, age, sex, race, ethnicity, and number of prior PCR tests has taken a first shot by 2/8/2021. For example, the model that generates propensity scores might say someone in zip code 55443, age 25, male, white, non-Hispanic and 1 prior test has a 20% chance of having been vaccinated (and 80% chance of not).

The originators of the propensity-scoring method demonstrated that we can form well-balanced control groups in the following way: within each zip code, take each vaccinated individual and find an unvaccinated individual that has the same propensity score. Within this basic framework, there are many variants. What does it mean by "same score"? How many controls to match to a single vaccinated person? (The Mayo Clinic does 1-to-1 matches.) Which individual is selected if several viable candidates are available? (They use a "greedy" method.)

What is the point of such adjustments?

The following excerpt of Table 1 from the paper illustrates the core idea of "covariate balance."

Each of the vaccinated and unvaccinated groups consists of 31,069 people because of 1-to-1 matching. The distribution of age groups, sex, race, etc. are all well-balanced between the two groups. Both groups are approximiately 63% female, 21% 75 years or older, etc. (Sidenote: I flagged the age groups because I was surprised that only 23% of the vaccinations went to 75 years and older, equal to the proportion that went to 18-25.)

This balance condition emulates the randomization in an RCT. But don't mistakenly think they are equivalent. If truly randomized, we can see balance on any variable known or unknown to the investigators. The balance for propensity scores holds for variables used in the adjustment - precisely because we made it happen - but there is no guarantee that there will be balance for any variables not used.

So, it's very important to make sure researchers use as many variables as possible, and include all relevant ones. This is one of these nuggets of advice that sound better than practiced. We're making an assumption that we know all the factors that matter. But we soldier on.

One of the curious variables used by the Mayo-nference team is the number of prior PCR tests (any time before 2/8/2021). They assert that this variable is a good proxy for the level of exposure to the coronavirus.

Matching the case-counting window

Now, we get to a tricky problem that I have discussed before. We have pairs of patients, one vaccinated and one unvaccinated, which we will treat as if randomized. The vaccinated one has a Day 0, which is the day of the first shot. What about the unvaccinated person? Establishing Day 0 is really important - given the desire of these investigators to pinpoint when an infection occurred.

In the Mayo Clinic study, they assign to the unvaccinated person a "study enrollment date" (pretend), which is Day 0 of the paired vaccinated individual. Now, they can start counting cases after Day 0 in both groups. This then lead to a vaccine effectiveness (VE) number.

As the above quote suggests, their headline number ignores all cases that occur prior to Day 36, meaning 2D+14 (2D = 2nd dose) for Pfizer and 2D+7 for Moderna, and Day 36 from the study enrollment date for the unvaccinated. The VE is 83%.

Like every other team doing re-analyses or real-world studies, these investigators computed VE for all reasonable case-counting windows, and selected the one with the maximum VE. They show their homework in Table 3. Here is an excerpt:

The top part of the table shows calculations of VE for cumulative case-counting windows of the type Day X to end of study while the bottom part of the table shows calculations of VE for incremental weekly windows of the type Day X - Day X+7.

Note that there are two rows for "Day 36 onwards". The second row has VE 89%, slightly higher than 83% (but of course, the margin of errors are plus/minus 15%). I'll let the authors explain their reasoning:

Starting 36 days ... vaccine efficacy of 83.4%... Importantly, we found that two of the six infections in the vaccinated cohort on or after day 36 occurred in individuals who had received only one vaccine dose, even though all vaccinated individuals should have received two doses by this time point... Among the properly vaccinated individuals... efficacy of 88.7%.

This style of reasoning is now state of the art. They are not alone in using it. Every recent study I've read is doing the same. I've said a lot about this issue in a prior post (link) so I won't repeat this here. The one addition is that this final adjustment only applies to the vaccinated group because the unvaccinated group has no such thing as missing a second shot (unlike in the RCT, when the control group got placebo shots).

Can this study's finding be generalized outside the study population?

Before ending this post, I want to address one other issue - why the research team was very careful in not generalizing their finding outside the analysis set of people.

There are two challenges to such a generalization: the vaccinated group in the analysis set has to be representative of all vaccinated Mayo Clinic patients; and then the Mayo Clinic vaccinated patients has to be representative of the wider vaccinated population.

(Note that because of matching, the unvaccinated group does not represent the general population. It mirrors the vaccinated group in the analysis set. That is the point.)

The analysis set is actually not representative because of a series of exclusions. They exclude:

• anyone who did not take at least one PCR test prior to February 8, 2021
• anyone who have not previously granted proper research authorization
• anyone under 18
• anyone who tested positive before December 1, 2020
• anyone living in zip codes with fewer than 25 people vaccinated at one of Mayo's facilities

In addition, for the vaccinated patients, they exclude:

• anyone who tested positive prior to their first shot (the data appear to show that people are given shots despite prior infections)
• anyone who got their first shot on the last day of the study period, thus having no follow-up period

The zip-code exclusion is particularly instructive. This creates biases. For example, less populous zip codes may be excluded. Also, it's likely that zip codes that have lower exposure may have fewer people interested in getting vaccinations.

It's very possible that sparsely-populated zip codes pose obstacles to finding matches. This is what is called incurable imbalance. If we can't find a matched control, we have to drop the vaccinated case. Dropping such cases makes the vaccinated group in the analysis set deviate from the vaccinated group in the system. (But it improves the balance of the people who are included.)

One of the key pieces of information you're looking for in an observational study is how many cases were dropped for lack of matches. This affects whether you can generalize the result. Sometimes, lack of matches is masquerading as an exclusion, as in the example above. If those small zip codes are excluded up top, they aren't looking for a match later but the effect is the same; we would have dropped them for lack of matches.

***

If you have come this far with me, I'm very impressed. I ran twice the length as the usual post so I'm going to move the rest into a future post. I will be addressing several concerns I have with this analysis for those who want to jump in the deep end. I believe this paper will stand the test of time as one of the better efforts at real-world studies.

 

P.S. [3/4/2021] The next post is published here. I delve into several technical issues so you can understand the limits to what real-world studies can tell us. It's an exciting area of research that blends art and science.

 

 

 

 

New year, more numbersense

Happy New Year. It's ever more important to use your numbersense when you hear data about the pandemic.

<1> The current count of "vaccinations" of Pfizer or Moderna - in the U.S., Israel, and anywhere else - is zero. No one is "vaccinated" until s/he receives the second shot. (Well, those who got two shots in the vaccine trials can be counted.)

<2> The dashboards are inflicting actual harm on the vaccination campaign, spreading the illusion that one dose is a complete vaccination. The world is soon to face a crisis of half-baked vaccinations.

<3> The government officials and celebrities must now get their second shots on camera to remind people to get them.

<4> In the Euro-zone, the second wave surge are being tamed after imposing more severe "lockdown" measures (mandatory night curfews, closing restaurants and bars, restricting movement, etc.) starting late October.

<5> Mortality trends are still highly predictable, demonstrating "statistical gravity". Current death rates reflect cases from up to a month ago. In the coming weeks, Italy's line would have fallen further, Germany's risen a little, and U.K.'s spiked. (Cross-reference the previous chart)

<6> The data in three key countries do not support the theory that the death rate from Covid-19 has declined. (Remember the U.S. and U.K. lines for the second wave have not yet peaked.)

<7> Rates don't tell the full story. Here are the cumulative death tolls, which show failure to control the spread is tragic.

<8> Death is a choice. Be smart.

 

Netflix, the data pioneer, gets flagged for data offence

The headline says it all. The Hollywood Reporter reported that "'The Witcher' Breaks Netflix Series Viewership Records -- with One Big Catch."

This should come as no surprise to my readers. It proves yet again you can use data to prove anything you want. Apparently, Netflix switched how it counts "viewers" in 2020.

Those not knee-deep in data work might not appreicate how measuring viewership can be so controversial. How hard is it to count how many people viewed a show?

In this case, Netflix announced 76 million households watched the series during the first four weeks of release. This seemingly simple statement includes a load of unspoken assumptions:

• Does each houshold have to watch all four weeks or one out of four weeks? (Nothing prevents one from requiring each household to watch two or three out of four weeks, either!)
  
  
• How much of each show must be watched in order to count as a "view"? (Hint: if the show is 60 minutes, absolutely no one will count 60 minutes. If you want to guess, select a number closer to 0 than to 60. Correct answer is disclosed below.)
  
  
• How many episodes must be be "viewed" to consider the series "viewed"? (The Witcher Season 1 has eight episodes, all released at the same time. I don't think you need a hint for this one.)
  
  
• Should each device and each sub-account be aggregated or not?

The above list isn't even comprehensive; there are even more subtle issues:

• What is being measured is not "viewing" but "streaming". The video player sends back regular signals to Netflix saying the player is still streaming a particular show. To measure actual viewing, the TV would have to track the viewers' eyes. (This is technically feasible although I think these companies aren't doing this.)
  
  
• Do repeat minutes count? If the rule is 10 minutes of at least two episodes, how do we deal with someone who watched 10 minutes of the same episode, twice?
  
  
• How to deal with fast forward, skipping, etc.?

***

As you look at those questions, think about whether there is a principled way to make these decisions. How does one argue that one out of four weeks is not a view but two out of four is? How does one argue that 10 minutes constitute viewing but 5 minutes don't?

It's pretty clear why you can use data to prove anything you want.

Still, what Netflix did is jaw-dropping. The previous definition of a view is "watching at least 70 percent of one episode of a series or 70 percent of a feature film". The new definition of a view - watching at least 2 minutes of one episode. You are reading this right; the 76 million households have "viewed" one Watcher episode for at least 2 minutes.

The first definition is already quite generous. The rule of the game is to get enough people to turn on the first episode, and it doesn't matter that they never return. Since Netflix has the famous recommendation engine, it's got the tool to drive traffic to the first episode of pretty much anything. The audience have to stick around for 70 percent of the running time, though.

How does Netflix justify dropping the bar from 70 percent of one episode to 2 minutes?

From the Hollywood Reporter:

Our new methodology is similar to the BBC iPlayer in their rankings based on 'requests' for the title, 'most popular' articles on The New York Times, which include those who opened the articles, and YouTube view counts," the company notes in its earnings report. "This way, short and long titles are treated equally, leveling the playing field for all types of our content including interactive content, which has no fixed length."

Netflix is right about Youtube. Youtube counts 30 seconds of run time as "viewing". Also, Youtube is autoplay by default. Once you start playing one video, it performs a version of "doom scrolling" so it keeps playing the next clip. I'm surprised people actually suggest that such "autoplay" views are not counted as views. Doing so would disqualify a ton of legitimate viewing as not views - and Youtube makes money from generating views!

The most objectionable part of the quote is "short and long titles are treated equally". According to Netflix, treating someone who watched 2 minutes of a 10-minute clip the same as someone who watched 2 minutes of a 240-minute movie demonstrates fairness.

You know it's a problem when the agreeable press not only noticed the data offense but also decided to stick it in the headline rather than burying it two paragraphs from the end of the article.

***

The issue of definitions is everywhere in data analysis. Covid19 tests and deaths. Flight delays. Mean tweets. Twitter users. Facebook video view time. Wind chill.

In Chapter 2 of Numbersense (link), I featured some doctors who believed that changing the definition of obesity will solve the obesity problem in the U.S.

P.S. [10/20/2020:

Mark Palko alerted me to his take on this news also. I think what he's saying is Netflix knows (of course) how much time each device "viewed" each minute of each episode, it just chooses not to publicize that data. Great point!

Here is another post by Mark that covers how Netflix uses "deceptive messaging and suspicious statistics". 

]

Coronavirus has not gone away, neither have these Covid fallacies

It's a good time to be reminded of the fallacies that continue to pollute global dialogue about the pandemic.

 

Comparing Covid fatality rate to seasonal flu fatality rate

A few research groups are pursuing the hypothesis that Covid-19 is just a type of seasonal flu. These groups hope to show that the "true" fatality rate of Covid-19 is on a par with the seasonal flu. They have pushed for antibody testing, under the theory that many people have been infected but did not get symptoms. The much heralded study that claimed prevalence of over 15 percent is the study by the NY State Department of Health, much trumpeted by Governor Cuomo (about which no one involved either scientifically or politically has the guts to publish an official report. What are they hiding?)

This project was doomed from the start. It should be clear by now that data collected during any epidemic are subject to significant measurement errors, which are real scientific challenges, and exacerbated by political interference - perhaps expressing the desire to look good. But you already know this - and I'm not retreading this point today.

Even if we assume that the infection fatality rate of Covid-19 has been accurately measured, it is still inappropriate to compare that to the fatality rate of seasonal flu.

The fatality rate for seasonal flu is conditioned by widespread flu vaccination (at least in the U.S. where everyone is urged to take the flu shot) while there is currently no vaccine for SARS-CoV-2. This fact alone dooms the project. One would have to go back to the period before the flu vaccine was invented to find the right comparable.

What's more, how accurate is the flu's fatality rate? It also faces measurement challenges because the flu is endemic. Most flu infections are not reported, detected or confirmed. Undercounting is as much an issue with flu as it is with Covid-19. [Epis reading this, please correct me if the fatality rate of seasonal flu published by the CDC are not counts of actual confirmed cases but estimations based on modeling that imputes undetected cases.]

 

Hoping for herd immunity as well as flu-like fatality

Those who believe Covid is a type of flu also tend to advocate the strategy of "herd immunity." If enough people are infected, the remaining population will be naturally protected. I'm only interested in the analytical methods so I don't judge anyone on their beliefs. It's still possible Covid-19 becomes an endemic disease like the flu. What doesn't work is believe both the flu hypothesis and the herd immunity strategy.

Herd immunity requires immunity: someone who was previously infected develop antibodies that confer immunity to the virus upon recovery.

Most of the epi models adopt the SIR structure, for which immunity is built in. The R compartment consists of "recovered" (and dead) people. Recovered people and dead people are treated the same mathematically because neither can spread the disease to anyone else. That's only true if infected people who recover are not infectious.

And yet, no expert has been willing to say definitively that having antibodies for SARS-CoV-2 makes one immune to Covid-19.

Further, you'll learn that epidemiologists do not use SIR models for seasonal flu. Instead, the appropriate model is SIS. The SIS structure has no R compartment; anyone who recovers is susceptible again. That of course is our experience with the flu. Having caught it does not mean you won't catch it again - one might even catch the flu multiple times within one season.

If Covid-19 is indeed just like the flu, then immunity is irrelevant. Herd immunity is even more pointless.

 

Saying young people have a low risk of dying from Covid

This statement is literally true. What is missing is context. Young people have a low risk of dying, period. I'm sure if the overall death rate of flu is 0.1%, the rate for young people is much smaller than that. (Remember this post.) What matters is the excess risk over the normal risk of dying for each age group. Dave Spiegelhalter at Cambridge has done a lot of this work for the UK. Here is an example of the calculations involved. The bottom line is: having Covid-19 is like packing a full year's of risk into a few weeks, for all age groups.

That framing apparently confused some people so he re-wrote the headline later to say "Covid roughly doubles your risk of dying this year". That's because the normal risk of dying excludes Covid-19 and the risk of dying from Covid-19 is roughly equal (ages 20-49) or higher (50 and up) than the normal risk.

Thus, even for younger people, getting infected with Covid-19 doubles your chance of dying. It's one thing to say double the death rate is fine by me, it's another thing to say that the risk of Covid death is "low".

 

Claiming monitoring X could have predicted the outbreak

A slate of studies are making claims that sewage, parking lot volume, search engine traffic and so on are leading indicators, and could have alerted us to the outbreak before the first case was detected.

Studies that look for the virus in sewage are worth doing. They help epidemiologists nail down the spread of the virus at the early stage of the epidemic.

Such research goes off track when these investigators over-reach and claim predictive prowess. After the novel virus has been discovered, it is a straigthforward matter to find prior sewage samples, and test them for the presence of the virus. If we also know that the virus-infected sewage came from which buildings or households, we'd have learned a lot about how the virus spread. But this explanatory model does not predict.

It is utterly wrong to conclude that sewage monitoring is a leading indicator of viral spread. Dial time back to November before the first case was confirmed in Wuhan, China. We assume the local sewage company was testing the dirty stuff. But for what? At that time, no one knew anything about this coronavirus, there were no tests available, no virus samples to validate any tests. Is the lab supposed to test for every possible virus including all animal viruses that have non-zero chance of leaping to humans?

It's not just the sewage (link). It's also the parking lot images and search engine traffic (link). So, from here on out, if more people are searching for "diarrhea", we'll predict that a new virus is circulating in the population? Even if true, which virus is this?

This type of studies do not, and will never, become the basis of a predictive model. It is malpractice that scientific journals have allowed researchers to make such false conclusions.

 

Talking about metrics as if they aren't manipulated

All metrics are perverted. This is a theme in my second book, Numbersense (link), which starts with a chapter filled with every manner of statistical shenanighens practiced by deans of reputable colleges and universities. It is then followed by a chapter on rampant obesity in the U.S., which some medical researchers claimed was a result of "bad measurement". (This blog is a preview of that chapter.) Latter chapters focus on economic metrics like unemployment rate and inflation rate. You get the idea.

In the short history of the Covid pandemic, people started with case statistics. Then, they claimed that death statistics are less manipulated than case statistics, when they learned about testing. Then, they claimed that testing statistics are less manipulated, until they realized governments determined who got tested. Some governments counted tests shipped, not test results. Then, they claimed hospitalization statistics are less manipulated, until they learned that hospitals sent sick patients to nursing homes.

If you read Numbersense (link), you may recall some students with low GPAs or test scores are asked to spend their first year at other schools or part-time programs, so they disappear from the statistics submitted to U.S. News for the school rankings. Does it shock you that hospitals sent patients to nursing homes?

Only the naive takes a "see no evil" view. In the age of Big Data, taking such a view is simply irresponsible - as there is too much leeway to twist the data or analysis to whatever story one wants to spin.

 

The first major study using Covid tracking app data stumbles out of the gate

The tracking app studies are on rush order, just like all sorts of Covid-related preprints that have come under scrutiny. I learned about this collaboration between King’s College (UK) and an app developer Zoe Global through a news article proclaiming that 13 percent of the UK have already been infected with the novel coronavirus. This is the catch of the moment, being able to declare that you know what proportion of the population has already been infected.

I discussed the Stanford study the other day, which has received robust criticism from the statistical community (e.g. Gelman). This Covid Symptom Tracker study is much less convincing as a way to measure population prevalence. Its potential value is for rationing test kits to those most likely to test positive but notice that the objective of targeting the most afflicted conflicts with the objective of measuring prevalence, which is a feature of the general population.

***

Methodology

The UK team used an indirect path to estimating the prevalence of the SARS-CoV-2 virus. They launched a mobile app, which users downloaded and by which they submitted symptoms on a daily basis. A tiny subset of these users reported having taken a test, and submitted their test results. Using those users with known test results, the team then built predictive models relating the symptoms to the test results. They claimed that loss of smell and taste (anosmia) was the top predictor, even higher than fever or cough. They then used the model to score the large chunk of app users who submitted symptoms but not test results. The model predicted 13 percent of this group to test positive (should they get tested). The team then generalized this result to the general UK population. They (or the press) made a host of breathtaking claims: a) their predictive model can replace blood tests (“detect infections….without the need of extensive biospecimen testing”); b) the single predictor (anosmia) should be “used as a screening tool to help identify potential mild cases who could be instructed to self-isolate”; c) the 54,000 users predicted by the model to test positive “are likely to be infected by the virus”, which led to suggestions that official case counts drastically undercount infections.

Comparison to Stanford Study

This method of estimating population prevalence of the virus differs from the Stanford study in major ways. The Stanford study involved actual blood tests combined with survey data collected on the day of testing. The Symptom Tracker study collected survey data over time via an app interface, while the published analysis focused only on the first five days after the app launched. The Stanford study used actual results from antibody tests while the Symptom Tracker study asked users to report their test results from previously taken PCR (diagnostic) tests. The former looks for people who have recovered from a previous infection while the latter confirms people who were infected at the time they were tested.

***

I took plenty of notes while reading the preprint submitted to Nature (link). I grouped my comments into three buckets that reverse the analytical workflow. First, I discuss their conclusions while assuming that the analysis and data collection are accepted. Then, I point out major issues with the analytical framework. Last but not least, I raise concerns about how the data are collected and processed.

[P.S. This post became so long that I have to break it up into multiple parts.]

***

The fruit is dry, and the researchers kept squeezing

#1

Despite the headline number of 1.6 million downloads, 75% of those users never made it past the starting block. The data driving the predictive model contain about 1,700 users, those who reported both their symptoms and results from prior testing to the app. The model is then applied to another 412,000 users who reported symptoms but not test results. The authors referred these 412,000 people as a “test set”. This is an abuse of terminology. A test set must come with the answers. In this case, a proper test set should include the test results for every individual scored by the predictive model. The analyst can then compare the predicted result to the actual result in order to measure the model’s accuracy. No one knows a single test result for the 412,000 individuals in the “test set”.

#2

The oversight of #1 led the researchers to make the gravest error. They believed their own model. In stating their conclusion, they strayed from 13 percent predicted to test positive to 13 percent “likely to be infected by the virus”. This generalization presumes 100% accuracy for model prediction as well as 100% accuracy for testing.

#3

The press took the liberty to generalize further. Instead of 13 percent of users of the Covid Symptom Tracker app, they speak about 13 percent of the UK population. That step of logic requires that the app users are a random sample of the general public, which is obviously false. [I will address data collection and processing in a separate post.]

#4

We next look at accuracy metrics for the 1,700 users used to develop the predictive model. The reported sensitivity is 54% and the specificity is 86%. I’ll round these to 55% and 85% for our discussion. We are familiar with the concepts of false positives and false negatives because of Covid testing (if not, see my previous post). In the set of 1,700 users, the researchers said, one in three reported positive test results. So, the model’s job is to fish out the 567 positives from the pool of 1,700. Sensitivity of 55 percent means the model correctly predicted 312 of those 567 users. Specificity of 85 percent means the false positive rate is 15 percent, so out of the 1133 users with negative test results, the model incorrectly predicted 170 to be positive. Taken together, the model issued 312+170 = 482 positive predictions out of 1,700 users, roughly 28 percent.

The so-called positive predictive value for the predictive model is 312/482 = 65 percent. Out of every 100 people predicted positive by the model, we expect 65 to be correct while 35 are false positives.

From a predictive accuracy perspective, this is pretty good. If we grab a random sample of those 1,700 users, we will find a third of them to be positive (self-reporting) but if we select from the subset that are predicted positive, we will find almost two-thirds to be positive.

But the model (like any other) is nowhere close to perfect. Thirty-five percent of those predicted to test positive will test negative, if they take the test. The model also makes false negative errors. It flags 28 percent of the 1,700 as positives so even if every predicted positive is a true positive, the model cannot capture all self-reported positives, which is one-third of the group.

If you have the Numbersense book (link), you can review all of this material in Chapter 5, where I applied the same analysis to Target’s model for predicting which shopper might be pregnant.

#5

The percent truly positive is not the same as the percent predicted positive because of false positives and false negatives. A prediction cannot be equated to being “infected with the virus”.

The authors of the preprint did qualify their statement with “likely to be infected”. How likely is likely? Don’t hold your breath, the chance of testing positive can be just above 50/50.

To see this, you have to understand model scoring. The main model is a logistic regression, which means that each of the 412,000 individuals is given a score, to be interpreted as the predicted probability of testing positive for the coronavirus should they take the PCR test. The more relevant symptoms someone has, the higher the score. Since the score is a probability, its value ranges from 0% to 100%.

The analyst must translate these probabilities into a binary outcome (will the individual test positive or not?). This study did the most standard thing one can do - picking 50% as the cutoff score. All individuals scoring above 50 percent are predicted to test positive, and those scoring below 50 percent are predicted to test negative.

With a 50% cutoff probability, the marginal cases within the 13 percent predicted to test positive will have a chance of testing positive of around 50 percent. It’s quite, hmm, bold to say these people are “likely to test positive”.

#6

Now, let me explain why the percent predicted positive isn’t just somewhat useless; the percent predicted positive has nothing to say about population prevalence.

It turns out that for a given predictive model, the analyst decides what percent to predict positive. It’s not a passive outcome of the model. The analyst chooses its value by trading off false positives and false negatives.

If the cutoff is increased from 50% to 75%, then the proportion predicted positive will drop below 13 percent, but we will have more confidence that those predictions are correct – the people at the margins now have at least 75% chance of testing positive. On the other hand, we can expand the proportion of predicted positives above 13 percent, and then, we will be less confident of those predictions – some of the predicted positives will have less than 50 percent chance of testing positive. The point being that the modeler decides what proportion of predictions are positive, and thus, the methodology of this Symptom Tracker study can justify any level of “prevalence”. It’s really this bad.

#7

I previously told Wired one of the evils of “triage” or targeted testing is that it biases the samples available to build predictive models that are necessary to conduct triage. In other words, we’ve set up a vicious cycle. This predictive model is being proposed to implement triage testing – that is, let the model decide who should get tested. (It’s hard to lay bad ideas to rest! The other being “herd immunity”.)

We have a real-life demonstration of the problem in this preprint. As shown in #4, this predictive model guesses that 28 percent of the 1,700 users who had prior test results would have tested positive. Then, the model is used to score 412,000 users who have not taken the test. It guesses 13 percent of this group to test positive. Why not 28 percent?

If both groups of users have the same characteristics (say, both are random samples of the general population), then we should expect both proportions to be 28 percent.

However, in the UK, they severely ration testing, and only test people with severe symptoms. In Table 1 of the preprint, you can see that people in the 412,000 group reported lower incidence of almost all the symptoms listed, compared to the 1,700 who already were tested (even lower than the ones who tested negative).

Did the model realize that the untested people are less likely to test positive, and therefore predicted a lower proportion to test positive? I wish our models could be so smart. But no way! So why did the model flag only 13 percent of th 412,000?

I'm going to let you stew on this one a bit, and reveal the reason in my next post. It might help to use the Target pregnancy prediction model that I featured in the Numbersense book (link).

That model predicts the chance that a female shopper is pregnant because pregnant women spends money. Assume that I built the model using a list of women shoppers. The model makes predictions based on past purchases of certain products.

Now, I apply this model to score a list of male shoppers. This means I feed in data on whether these shoppers purchased those products included in the model. What scores (probabilities) will the men get? Does the model know the training data contain all women, and the "test set" contain all men? Please comment below if you need help thinking through this.

#8 (a minor point)

In the preprint, the 13 percent comes with a ludicrously tiny margin of error: [12.97%, 13.15%]. Typically, a small margin of error instills confidence that what was observed was unlikely to occur due to random chance. In this instance, the tiny margin of error is purely a result of throwing 412,000 names into the “test set”. When the sample size is so huge, the model almost always flags 13 percent as positive.

The uncertainty of the study’s conclusion arises from the modeling not the scoring of the model. So, if you want to understand the model’s accuracy, you have to go back to the discussion in #4. Those accuracy metrics have their own margin of errors, which are in the plus or minus 5 to 10 percent range.

***

What about the other big talking point coming out of this study - that loss of smell and taste is more important than even cough as a predictor of having SARS-CoV-2?

#9

Triage testing also dooms this finding (I know, it's sad to see). Because of pre-selection, the entire set of 1,700 people with test results have high incidence of all the symptoms. Thirty-three percent of them experienced loss of smell and taste. For persistent cough, it’s almost 50 percent.

Here’s the problem: if half the people have cough, cough isn't a good predictor of anything. Remember that only a third of this group self-reported positive test results. Any model that uses cough as a single indicator splits the group 50/50, which immediately leads to 28 false positives.

It’s no coincidence that 33 percent experienced loss of smell and taste, compared to 34 percent (to be exact) who self-reported positive test results. The reason why anosmia was the “best predictor” is because its incidence in the training data approximates the frequency of the positive outcome.

***

For data scientists and analysts, the big lesson is don’t feed biased data into model building.

 

[P.S. I have further comments on the data pre-processing and the data collection of the Covid Symptom Tracker study.]

Know your data 26: Confidence in numbers dropping

Now that you clicked on that, I'm here to tell you confidence - in - numbers dropping is not the same as confidence in numbers - dropping.

That's my reaction when I went to the Covid Tracking Project looking for California's testing numbers. Here's what I found:

Did someone explain what happened on Wednesday? The total number of test results jumped by 50% in one day, with most of those being negatives. The proportion of positive results went down from 11% to 7.5% in one day.

This type of weirdness makes data analysts nervous. It's hard to trust the data, and it destroys any ability to analyze trends.

Excess deaths: why we want the cake and why we can't eat it yet

In my early March preview of statistical issues relating to coronavirus, I mentioned "people are counted as infected only if they are tested", and I also asked "how are deaths confirmed?"

For a while, the media ignored how meaningless the case counts were. Eventually, they got it, and that was great. But some experts convinced them that the savior is the death count. And now, in the last week, the reporters finally learned that death counts are subject to similar problems of definition.

The Economist just launched a tracker to look at excess all-cause mortality, using data from EuroMOMO. If during this pandemic, the total number of deaths from any cause is higher than normal, then it provides evidence (though not irrefutable) that Covid-19 is not a new name for influenza. Of course, we are finding excess deaths everywhere.

The next puzzle is that the number of deaths attributed to Covid-19 on death certificates is frequently lower than the excess deaths from all causes. This raises the possibility that Covid-19 deaths are under-reported. By itself, this is not necessarily true, because it is also possible that deaths from other causes are being under-counted.

However, it has also emerged that in many countries and regions, the published counts of fatalities so far have only included deaths occurring at hospitals for whom the deceased had returned a positive test for the virus. A good number of deaths occurred in care homes, hospices, and even private residences, and these have been ignored. In fact, many hospitals refused to admit patients who do not report severe symptoms, and told them to isolate at home.

The excess all-cause mortality data take a bit more time to collect, and perhaps not by accident, as these statistics come to light, many countries announced that they are revising how they count Covid-19 deaths, essentially admitting they were under-reporting.

***

The lesson related to data integrity is being taken up very slowly. Even as the Economist published the valuable tracker of all-cause mortality, they wrote:

... people have become grimly familiar with the death tolls... these numbers give a better indicator of a country's trajectory than do counts of confirmed cases, which largely measure how many people have been tested. Nonetheless, official covid-19 death tolls still under-count...

This is better than most but still repeats several fallacies:

a) Given the unknown fatality rate of Covid-19, the death count is a proportion of the case count. If data integrity is not at issue, the death count is a worse indicator of a trajectory than the case count because it just gives the same information after a time delay. If death tolls are better than confirmed cases, it's because we believe the data on deaths are less bad than the data on cases.

b) The emerging evidence shows that the unadjusted death counts are wildly inaccurate, so I'm not sure you can say the data on deaths are "the lesser evil".  In Lombardy, the Economist estimated the "true toll was about 120% higher" than the confirmed deaths. In Spain, the true number might be 60% higher (off a much larger base).

c) Confirmed cases is not a simple function of how many people got tested. More importantly, it is measuring who got tested. I know I have been hounding this point to death. In statistics, we care not only about the size of the sample but the composition of the sample! We could test lot of people with no symptoms and still have few confirmed cases.

d) The under-counting of deaths is still partially due to lack of testing. It is true that Covid-19 testing is not required in those all-cause mortality statistics but without testing, the count of deaths is clearly less reliable, just like the count of cases.

e) Deaths can also be over-counted, especially without testing to confirm coronavirus. How well can a doctor tell if the patient who died of pnuemonia had Covid-19 or some other illness?

f) If it's broke, you can't fix it. Countries change the definition of deaths mid-way but won't (and can't) change the past. This renders the death count series useless, unless you're willing to adjust the data. This was a key point I made about new metrics of obesity in Chapter 2 of Numbersense (link).

***

So, if you've been reading this blog, you know interpretation of statistics is no simple business.

While compiling these excess mortality numbers is necessary and important, here is something you should know, and I'm not going to sugarcoat it.

Imagine wearing glasses or sunglasses in the summer. You walk into an air-conditioned shop. You must wait for the fog to clear.

These excess mortality numbers are shrouded in a fog right now. It is actually somewhat meaningless to discuss them. This is because of a substitution effect. Some of the excess deaths we're seeing are accelerated deaths. (Of course, all excess deaths are accelerated deaths.) What proportion of the excess deaths are relevant will be up for debate once we wait long enough to see clearly.

A store usually runs an annual sale on New Year's Day only. This year, the store starts the discounts on the day before. After New Year's Eve, the store reports 50% jump in sales versus last year ("excess"). Was the promotion particularly effective this year? You don't know until after New Year's Day. When you compare the total sales of both days against the two days last year, you'll find that the jump is less than 50%. Some customers just showed up one day earlier.