Big Data, Plainly Spoken (aka Numbers Rule Your World)

Coronavirus has not gone away, neither have these Covid fallacies

It's a good time to be reminded of the fallacies that continue to pollute global dialogue about the pandemic.

 

Comparing Covid fatality rate to seasonal flu fatality rate

A few research groups are pursuing the hypothesis that Covid-19 is just a type of seasonal flu. These groups hope to show that the "true" fatality rate of Covid-19 is on a par with the seasonal flu. They have pushed for antibody testing, under the theory that many people have been infected but did not get symptoms. The much heralded study that claimed prevalence of over 15 percent is the study by the NY State Department of Health, much trumpeted by Governor Cuomo (about which no one involved either scientifically or politically has the guts to publish an official report. What are they hiding?)

This project was doomed from the start. It should be clear by now that data collected during any epidemic are subject to significant measurement errors, which are real scientific challenges, and exacerbated by political interference - perhaps expressing the desire to look good. But you already know this - and I'm not retreading this point today.

Even if we assume that the infection fatality rate of Covid-19 has been accurately measured, it is still inappropriate to compare that to the fatality rate of seasonal flu.

The fatality rate for seasonal flu is conditioned by widespread flu vaccination (at least in the U.S. where everyone is urged to take the flu shot) while there is currently no vaccine for SARS-CoV-2. This fact alone dooms the project. One would have to go back to the period before the flu vaccine was invented to find the right comparable.

What's more, how accurate is the flu's fatality rate? It also faces measurement challenges because the flu is endemic. Most flu infections are not reported, detected or confirmed. Undercounting is as much an issue with flu as it is with Covid-19. [Epis reading this, please correct me if the fatality rate of seasonal flu published by the CDC are not counts of actual confirmed cases but estimations based on modeling that imputes undetected cases.]

 

Hoping for herd immunity as well as flu-like fatality

Those who believe Covid is a type of flu also tend to advocate the strategy of "herd immunity." If enough people are infected, the remaining population will be naturally protected. I'm only interested in the analytical methods so I don't judge anyone on their beliefs. It's still possible Covid-19 becomes an endemic disease like the flu. What doesn't work is believe both the flu hypothesis and the herd immunity strategy.

Herd immunity requires immunity: someone who was previously infected develop antibodies that confer immunity to the virus upon recovery.

Most of the epi models adopt the SIR structure, for which immunity is built in. The R compartment consists of "recovered" (and dead) people. Recovered people and dead people are treated the same mathematically because neither can spread the disease to anyone else. That's only true if infected people who recover are not infectious.

And yet, no expert has been willing to say definitively that having antibodies for SARS-CoV-2 makes one immune to Covid-19.

Further, you'll learn that epidemiologists do not use SIR models for seasonal flu. Instead, the appropriate model is SIS. The SIS structure has no R compartment; anyone who recovers is susceptible again. That of course is our experience with the flu. Having caught it does not mean you won't catch it again - one might even catch the flu multiple times within one season.

If Covid-19 is indeed just like the flu, then immunity is irrelevant. Herd immunity is even more pointless.

 

Saying young people have a low risk of dying from Covid

This statement is literally true. What is missing is context. Young people have a low risk of dying, period. I'm sure if the overall death rate of flu is 0.1%, the rate for young people is much smaller than that. (Remember this post.) What matters is the excess risk over the normal risk of dying for each age group. Dave Spiegelhalter at Cambridge has done a lot of this work for the UK. Here is an example of the calculations involved. The bottom line is: having Covid-19 is like packing a full year's of risk into a few weeks, for all age groups.

That framing apparently confused some people so he re-wrote the headline later to say "Covid roughly doubles your risk of dying this year". That's because the normal risk of dying excludes Covid-19 and the risk of dying from Covid-19 is roughly equal (ages 20-49) or higher (50 and up) than the normal risk.

Thus, even for younger people, getting infected with Covid-19 doubles your chance of dying. It's one thing to say double the death rate is fine by me, it's another thing to say that the risk of Covid death is "low".

 

Claiming monitoring X could have predicted the outbreak

A slate of studies are making claims that sewage, parking lot volume, search engine traffic and so on are leading indicators, and could have alerted us to the outbreak before the first case was detected.

Studies that look for the virus in sewage are worth doing. They help epidemiologists nail down the spread of the virus at the early stage of the epidemic.

Such research goes off track when these investigators over-reach and claim predictive prowess. After the novel virus has been discovered, it is a straigthforward matter to find prior sewage samples, and test them for the presence of the virus. If we also know that the virus-infected sewage came from which buildings or households, we'd have learned a lot about how the virus spread. But this explanatory model does not predict.

It is utterly wrong to conclude that sewage monitoring is a leading indicator of viral spread. Dial time back to November before the first case was confirmed in Wuhan, China. We assume the local sewage company was testing the dirty stuff. But for what? At that time, no one knew anything about this coronavirus, there were no tests available, no virus samples to validate any tests. Is the lab supposed to test for every possible virus including all animal viruses that have non-zero chance of leaping to humans?

It's not just the sewage (link). It's also the parking lot images and search engine traffic (link). So, from here on out, if more people are searching for "diarrhea", we'll predict that a new virus is circulating in the population? Even if true, which virus is this?

This type of studies do not, and will never, become the basis of a predictive model. It is malpractice that scientific journals have allowed researchers to make such false conclusions.

 

Talking about metrics as if they aren't manipulated

All metrics are perverted. This is a theme in my second book, Numbersense (link), which starts with a chapter filled with every manner of statistical shenanighens practiced by deans of reputable colleges and universities. It is then followed by a chapter on rampant obesity in the U.S., which some medical researchers claimed was a result of "bad measurement". (This blog is a preview of that chapter.) Latter chapters focus on economic metrics like unemployment rate and inflation rate. You get the idea.

In the short history of the Covid pandemic, people started with case statistics. Then, they claimed that death statistics are less manipulated than case statistics, when they learned about testing. Then, they claimed that testing statistics are less manipulated, until they realized governments determined who got tested. Some governments counted tests shipped, not test results. Then, they claimed hospitalization statistics are less manipulated, until they learned that hospitals sent sick patients to nursing homes.

If you read Numbersense (link), you may recall some students with low GPAs or test scores are asked to spend their first year at other schools or part-time programs, so they disappear from the statistics submitted to U.S. News for the school rankings. Does it shock you that hospitals sent patients to nursing homes?

Only the naive takes a "see no evil" view. In the age of Big Data, taking such a view is simply irresponsible - as there is too much leeway to twist the data or analysis to whatever story one wants to spin.

 

The first major study using Covid tracking app data stumbles out of the gate

The tracking app studies are on rush order, just like all sorts of Covid-related preprints that have come under scrutiny. I learned about this collaboration between King’s College (UK) and an app developer Zoe Global through a news article proclaiming that 13 percent of the UK have already been infected with the novel coronavirus. This is the catch of the moment, being able to declare that you know what proportion of the population has already been infected.

I discussed the Stanford study the other day, which has received robust criticism from the statistical community (e.g. Gelman). This Covid Symptom Tracker study is much less convincing as a way to measure population prevalence. Its potential value is for rationing test kits to those most likely to test positive but notice that the objective of targeting the most afflicted conflicts with the objective of measuring prevalence, which is a feature of the general population.

***

Methodology

The UK team used an indirect path to estimating the prevalence of the SARS-CoV-2 virus. They launched a mobile app, which users downloaded and by which they submitted symptoms on a daily basis. A tiny subset of these users reported having taken a test, and submitted their test results. Using those users with known test results, the team then built predictive models relating the symptoms to the test results. They claimed that loss of smell and taste (anosmia) was the top predictor, even higher than fever or cough. They then used the model to score the large chunk of app users who submitted symptoms but not test results. The model predicted 13 percent of this group to test positive (should they get tested). The team then generalized this result to the general UK population. They (or the press) made a host of breathtaking claims: a) their predictive model can replace blood tests (“detect infections….without the need of extensive biospecimen testing”); b) the single predictor (anosmia) should be “used as a screening tool to help identify potential mild cases who could be instructed to self-isolate”; c) the 54,000 users predicted by the model to test positive “are likely to be infected by the virus”, which led to suggestions that official case counts drastically undercount infections.

Comparison to Stanford Study

This method of estimating population prevalence of the virus differs from the Stanford study in major ways. The Stanford study involved actual blood tests combined with survey data collected on the day of testing. The Symptom Tracker study collected survey data over time via an app interface, while the published analysis focused only on the first five days after the app launched. The Stanford study used actual results from antibody tests while the Symptom Tracker study asked users to report their test results from previously taken PCR (diagnostic) tests. The former looks for people who have recovered from a previous infection while the latter confirms people who were infected at the time they were tested.

***

I took plenty of notes while reading the preprint submitted to Nature (link). I grouped my comments into three buckets that reverse the analytical workflow. First, I discuss their conclusions while assuming that the analysis and data collection are accepted. Then, I point out major issues with the analytical framework. Last but not least, I raise concerns about how the data are collected and processed.

[P.S. This post became so long that I have to break it up into multiple parts.]

***

The fruit is dry, and the researchers kept squeezing

#1

Despite the headline number of 1.6 million downloads, 75% of those users never made it past the starting block. The data driving the predictive model contain about 1,700 users, those who reported both their symptoms and results from prior testing to the app. The model is then applied to another 412,000 users who reported symptoms but not test results. The authors referred these 412,000 people as a “test set”. This is an abuse of terminology. A test set must come with the answers. In this case, a proper test set should include the test results for every individual scored by the predictive model. The analyst can then compare the predicted result to the actual result in order to measure the model’s accuracy. No one knows a single test result for the 412,000 individuals in the “test set”.

#2

The oversight of #1 led the researchers to make the gravest error. They believed their own model. In stating their conclusion, they strayed from 13 percent predicted to test positive to 13 percent “likely to be infected by the virus”. This generalization presumes 100% accuracy for model prediction as well as 100% accuracy for testing.

#3

The press took the liberty to generalize further. Instead of 13 percent of users of the Covid Symptom Tracker app, they speak about 13 percent of the UK population. That step of logic requires that the app users are a random sample of the general public, which is obviously false. [I will address data collection and processing in a separate post.]

#4

We next look at accuracy metrics for the 1,700 users used to develop the predictive model. The reported sensitivity is 54% and the specificity is 86%. I’ll round these to 55% and 85% for our discussion. We are familiar with the concepts of false positives and false negatives because of Covid testing (if not, see my previous post). In the set of 1,700 users, the researchers said, one in three reported positive test results. So, the model’s job is to fish out the 567 positives from the pool of 1,700. Sensitivity of 55 percent means the model correctly predicted 312 of those 567 users. Specificity of 85 percent means the false positive rate is 15 percent, so out of the 1133 users with negative test results, the model incorrectly predicted 170 to be positive. Taken together, the model issued 312+170 = 482 positive predictions out of 1,700 users, roughly 28 percent.

The so-called positive predictive value for the predictive model is 312/482 = 65 percent. Out of every 100 people predicted positive by the model, we expect 65 to be correct while 35 are false positives.

From a predictive accuracy perspective, this is pretty good. If we grab a random sample of those 1,700 users, we will find a third of them to be positive (self-reporting) but if we select from the subset that are predicted positive, we will find almost two-thirds to be positive.

But the model (like any other) is nowhere close to perfect. Thirty-five percent of those predicted to test positive will test negative, if they take the test. The model also makes false negative errors. It flags 28 percent of the 1,700 as positives so even if every predicted positive is a true positive, the model cannot capture all self-reported positives, which is one-third of the group.

If you have the Numbersense book (link), you can review all of this material in Chapter 5, where I applied the same analysis to Target’s model for predicting which shopper might be pregnant.

#5

The percent truly positive is not the same as the percent predicted positive because of false positives and false negatives. A prediction cannot be equated to being “infected with the virus”.

The authors of the preprint did qualify their statement with “likely to be infected”. How likely is likely? Don’t hold your breath, the chance of testing positive can be just above 50/50.

To see this, you have to understand model scoring. The main model is a logistic regression, which means that each of the 412,000 individuals is given a score, to be interpreted as the predicted probability of testing positive for the coronavirus should they take the PCR test. The more relevant symptoms someone has, the higher the score. Since the score is a probability, its value ranges from 0% to 100%.

The analyst must translate these probabilities into a binary outcome (will the individual test positive or not?). This study did the most standard thing one can do - picking 50% as the cutoff score. All individuals scoring above 50 percent are predicted to test positive, and those scoring below 50 percent are predicted to test negative.

With a 50% cutoff probability, the marginal cases within the 13 percent predicted to test positive will have a chance of testing positive of around 50 percent. It’s quite, hmm, bold to say these people are “likely to test positive”.

#6

Now, let me explain why the percent predicted positive isn’t just somewhat useless; the percent predicted positive has nothing to say about population prevalence.

It turns out that for a given predictive model, the analyst decides what percent to predict positive. It’s not a passive outcome of the model. The analyst chooses its value by trading off false positives and false negatives.

If the cutoff is increased from 50% to 75%, then the proportion predicted positive will drop below 13 percent, but we will have more confidence that those predictions are correct – the people at the margins now have at least 75% chance of testing positive. On the other hand, we can expand the proportion of predicted positives above 13 percent, and then, we will be less confident of those predictions – some of the predicted positives will have less than 50 percent chance of testing positive. The point being that the modeler decides what proportion of predictions are positive, and thus, the methodology of this Symptom Tracker study can justify any level of “prevalence”. It’s really this bad.

#7

I previously told Wired one of the evils of “triage” or targeted testing is that it biases the samples available to build predictive models that are necessary to conduct triage. In other words, we’ve set up a vicious cycle. This predictive model is being proposed to implement triage testing – that is, let the model decide who should get tested. (It’s hard to lay bad ideas to rest! The other being “herd immunity”.)

We have a real-life demonstration of the problem in this preprint. As shown in #4, this predictive model guesses that 28 percent of the 1,700 users who had prior test results would have tested positive. Then, the model is used to score 412,000 users who have not taken the test. It guesses 13 percent of this group to test positive. Why not 28 percent?

If both groups of users have the same characteristics (say, both are random samples of the general population), then we should expect both proportions to be 28 percent.

However, in the UK, they severely ration testing, and only test people with severe symptoms. In Table 1 of the preprint, you can see that people in the 412,000 group reported lower incidence of almost all the symptoms listed, compared to the 1,700 who already were tested (even lower than the ones who tested negative).

Did the model realize that the untested people are less likely to test positive, and therefore predicted a lower proportion to test positive? I wish our models could be so smart. But no way! So why did the model flag only 13 percent of th 412,000?

I'm going to let you stew on this one a bit, and reveal the reason in my next post. It might help to use the Target pregnancy prediction model that I featured in the Numbersense book (link).

That model predicts the chance that a female shopper is pregnant because pregnant women spends money. Assume that I built the model using a list of women shoppers. The model makes predictions based on past purchases of certain products.

Now, I apply this model to score a list of male shoppers. This means I feed in data on whether these shoppers purchased those products included in the model. What scores (probabilities) will the men get? Does the model know the training data contain all women, and the "test set" contain all men? Please comment below if you need help thinking through this.

#8 (a minor point)

In the preprint, the 13 percent comes with a ludicrously tiny margin of error: [12.97%, 13.15%]. Typically, a small margin of error instills confidence that what was observed was unlikely to occur due to random chance. In this instance, the tiny margin of error is purely a result of throwing 412,000 names into the “test set”. When the sample size is so huge, the model almost always flags 13 percent as positive.

The uncertainty of the study’s conclusion arises from the modeling not the scoring of the model. So, if you want to understand the model’s accuracy, you have to go back to the discussion in #4. Those accuracy metrics have their own margin of errors, which are in the plus or minus 5 to 10 percent range.

***

What about the other big talking point coming out of this study - that loss of smell and taste is more important than even cough as a predictor of having SARS-CoV-2?

#9

Triage testing also dooms this finding (I know, it's sad to see). Because of pre-selection, the entire set of 1,700 people with test results have high incidence of all the symptoms. Thirty-three percent of them experienced loss of smell and taste. For persistent cough, it’s almost 50 percent.

Here’s the problem: if half the people have cough, cough isn't a good predictor of anything. Remember that only a third of this group self-reported positive test results. Any model that uses cough as a single indicator splits the group 50/50, which immediately leads to 28 false positives.

It’s no coincidence that 33 percent experienced loss of smell and taste, compared to 34 percent (to be exact) who self-reported positive test results. The reason why anosmia was the “best predictor” is because its incidence in the training data approximates the frequency of the positive outcome.

***

For data scientists and analysts, the big lesson is don’t feed biased data into model building.

 

[P.S. I have further comments on the data pre-processing and the data collection of the Covid Symptom Tracker study.]

Know your data 26: Confidence in numbers dropping

Now that you clicked on that, I'm here to tell you confidence - in - numbers dropping is not the same as confidence in numbers - dropping.

That's my reaction when I went to the Covid Tracking Project looking for California's testing numbers. Here's what I found:

Did someone explain what happened on Wednesday? The total number of test results jumped by 50% in one day, with most of those being negatives. The proportion of positive results went down from 11% to 7.5% in one day.

This type of weirdness makes data analysts nervous. It's hard to trust the data, and it destroys any ability to analyze trends.

Excess deaths: why we want the cake and why we can't eat it yet

In my early March preview of statistical issues relating to coronavirus, I mentioned "people are counted as infected only if they are tested", and I also asked "how are deaths confirmed?"

For a while, the media ignored how meaningless the case counts were. Eventually, they got it, and that was great. But some experts convinced them that the savior is the death count. And now, in the last week, the reporters finally learned that death counts are subject to similar problems of definition.

The Economist just launched a tracker to look at excess all-cause mortality, using data from EuroMOMO. If during this pandemic, the total number of deaths from any cause is higher than normal, then it provides evidence (though not irrefutable) that Covid-19 is not a new name for influenza. Of course, we are finding excess deaths everywhere.

The next puzzle is that the number of deaths attributed to Covid-19 on death certificates is frequently lower than the excess deaths from all causes. This raises the possibility that Covid-19 deaths are under-reported. By itself, this is not necessarily true, because it is also possible that deaths from other causes are being under-counted.

However, it has also emerged that in many countries and regions, the published counts of fatalities so far have only included deaths occurring at hospitals for whom the deceased had returned a positive test for the virus. A good number of deaths occurred in care homes, hospices, and even private residences, and these have been ignored. In fact, many hospitals refused to admit patients who do not report severe symptoms, and told them to isolate at home.

The excess all-cause mortality data take a bit more time to collect, and perhaps not by accident, as these statistics come to light, many countries announced that they are revising how they count Covid-19 deaths, essentially admitting they were under-reporting.

***

The lesson related to data integrity is being taken up very slowly. Even as the Economist published the valuable tracker of all-cause mortality, they wrote:

... people have become grimly familiar with the death tolls... these numbers give a better indicator of a country's trajectory than do counts of confirmed cases, which largely measure how many people have been tested. Nonetheless, official covid-19 death tolls still under-count...

This is better than most but still repeats several fallacies:

a) Given the unknown fatality rate of Covid-19, the death count is a proportion of the case count. If data integrity is not at issue, the death count is a worse indicator of a trajectory than the case count because it just gives the same information after a time delay. If death tolls are better than confirmed cases, it's because we believe the data on deaths are less bad than the data on cases.

b) The emerging evidence shows that the unadjusted death counts are wildly inaccurate, so I'm not sure you can say the data on deaths are "the lesser evil".  In Lombardy, the Economist estimated the "true toll was about 120% higher" than the confirmed deaths. In Spain, the true number might be 60% higher (off a much larger base).

c) Confirmed cases is not a simple function of how many people got tested. More importantly, it is measuring who got tested. I know I have been hounding this point to death. In statistics, we care not only about the size of the sample but the composition of the sample! We could test lot of people with no symptoms and still have few confirmed cases.

d) The under-counting of deaths is still partially due to lack of testing. It is true that Covid-19 testing is not required in those all-cause mortality statistics but without testing, the count of deaths is clearly less reliable, just like the count of cases.

e) Deaths can also be over-counted, especially without testing to confirm coronavirus. How well can a doctor tell if the patient who died of pnuemonia had Covid-19 or some other illness?

f) If it's broke, you can't fix it. Countries change the definition of deaths mid-way but won't (and can't) change the past. This renders the death count series useless, unless you're willing to adjust the data. This was a key point I made about new metrics of obesity in Chapter 2 of Numbersense (link).

***

So, if you've been reading this blog, you know interpretation of statistics is no simple business.

While compiling these excess mortality numbers is necessary and important, here is something you should know, and I'm not going to sugarcoat it.

Imagine wearing glasses or sunglasses in the summer. You walk into an air-conditioned shop. You must wait for the fog to clear.

These excess mortality numbers are shrouded in a fog right now. It is actually somewhat meaningless to discuss them. This is because of a substitution effect. Some of the excess deaths we're seeing are accelerated deaths. (Of course, all excess deaths are accelerated deaths.) What proportion of the excess deaths are relevant will be up for debate once we wait long enough to see clearly.

A store usually runs an annual sale on New Year's Day only. This year, the store starts the discounts on the day before. After New Year's Eve, the store reports 50% jump in sales versus last year ("excess"). Was the promotion particularly effective this year? You don't know until after New Year's Day. When you compare the total sales of both days against the two days last year, you'll find that the jump is less than 50%. Some customers just showed up one day earlier.