Big Data, Plainly Spoken (aka Numbers Rule Your World)

P esteem, bosom peril, flag to clamor, and anti-tech tech

What drug am I on this morning? Let me tell you, it's the anti-tech tech drug.

As you may have guessed, many new technologies are prone to abuse e.g. using generative AI to produce deep fakes, or what I learned earlier this week, QR code scams (link). The tech consensus is that we don't see any problems because one can use technology to combat such risks. I call this the anti-tech tech drug.

In the academic community, the digitization of knowledge has had profound impacts. One such impact is the creation of "metrics" - a whole field called bibliometrics - that purportedly measure the influence of specific researchers and research papers. What did I say about metrics? See this post from long time ago called "The inevitable perversion of metrics", which later got expanded into Chapters 1-2 of Numbersense (link).

Someone decided it would be a good idea to measure research contribution just like we measure the value of online advertising. Both count things like citations, clicks, downloads, views, etc. Citations are like hyperlinks. Google, whose initial algorithm was entirely based on counting hyperlinks, quickly discovered that hyperlinking is easily gamed. From very early on, humans invented multiple schemes to defeat it as a measure of importance, e.g. link exchanges, barters, buying links, selling links.

 

Not surprisingly, smart researchers have used myriad schemes to game bibliometrics. According to this recent article in Significance magazine, some less-than-honest researchers are flooding the zone with "fake" papers, written by bots. Just imagine having generative AI generate large numbers of "new" papers by feeding it already published articles.

Some anti-tech tech companies say don't worry, we have technology to combat such abuse. What is this tech? It's a plagiarism detection tool that can find copied contents.

What happens? The tricksters adapted.

***

Plagiarism detectors look for segments of text that have been seen before. What kind of text will evade detection? Unusual phrases, what we might call statistically unlikely phrases. Well, someone already wrote software that injects unusual phrases to replace common ones. The tricksters use such software to continue gaming the metrics.

Thus, these "rephrasing tools" have invented "P esteem" instead of "p value", "bosom peril" in lieu of "breast cancer", "flag to clamor" which means "signal to noise".

***

Even so, the authors of the Significance article were too polite to say out loud what's really broken about the peer review process, which has failed to reject many such papers. They reported that more than a thousand papers retrieved from PubMed contain the words "statically significant" instead of "statistically significant" in its title or abstract.

The authors respond to this enormous red flag by musing that "somehow, these goobledegook sentences also pass through peer review."

I can think of two likely scenarios:

1. some peer reviewers did not even open the papers they were supposed to opine upon
2. some editors colluded with authors in a publishing scam

 

On the evidence raised by SAT distractors

After reading my prior post, long-time reader Mark (link) alerted me to a series of posts he wrote 10 years ago on the anti-SAT movement. This is a topic that bubbles up every few years, and it appears that the current generation of college administrators may go where their predecessors didn't. Some colleges have started to drop the SAT requirement for admissions.

Post 1, Post 2, Post 3, Post 4, Post 5

Let me summarize Mark's various posts and add some color commentary.

***

In his first post (link), Mark explores the relationship between SAT scores and college GPAs. This exercise in analyzing correlation is meaningful to both sides of the debate. The distractors detractors argue that "there already is something that accurately mirrors the grades a student gets in school. Namely: the grades a student gets in school" while proponents say SAT scores are the best predictors of college success.

For example, when MIT reinstated the SAT requirement after the Covid-19 pandemic (link), the dean of admissions stated:

Our research has shown that, in most cases, we cannot reliably predict students will do well at MIT unless we consider standardized test results [especially the math component] alongside grades, coursework, and other factors.

Mark points out that the right measure of any predictor is incremental insight – what additional nuance it adds to the collection of predictors. This explains why the SAT looks more like a general intelligence test than a high-school exam. Otherwise, the SAT score would be an unnecessary duplicate of high-school GPA. (Why the previous sentence is stated forwards but not backwards is another sign of weak analytics in this space.)

Mark's point is self-evident: assume that high-school GPAs predict college performance just as well as the SAT, as the distractors detractors suggest, then SAT scores should mirror high-school GPAs, and if they did, the two metrics would result in identical rankings of the students; if so, what's the cause for complaint?

***

The second post (link) examines various reasons SAT critics have given for dumping the test. Mark is unimpressed by the quality of the evidence, and in many cases, its absence.

The weirdest argument is that according to Google's description of its own hiring practices, Google does not consider standardized test scores or college ranking. Mark remarked that "Google doesn't talk much about SATs at least in part because they've largely maxed out the metric."

Let's say we tally up the background data of Google hires; we are likely to find a good number of Ivy League graduates with high SAT scores. Can Google legitimately claim their hiring does not consider test scores or college ranking?

Sure, someone is going to say "Those people weren't hired because they graduated from an Ivy League school, or that they had high SAT scores. They just happen to have those characteristics. Correlation is not causation!"

Well, admitted students to top colleges not only have high SAT scores, but they also have high GPAs, and other great things on their applications so did they get accepted because of the SAT scores, or the high GPAs?

***

The third post presents a series of math questions from actual SAT exams because Mark realized that many critics of the SAT attacked an imaginary scarecrow version of the SAT test, rather than what the test really is. Seeing these real questions helps ground the debate.

He also touched on one virtue of the SAT test, which is its transparency. Past exams with answers are readily available. By contrast, transparency is definitely missing in high-school GPAs. Nevertheless, in real life, transparency frequently backfires. Where are these SAT critics when it comes to high-school GPAs? Let's compile GPAs and split them by race - I'd be surprised if they didn't find a race gap. The reason that the race gap in GPAs isn't in the news is that schools don't release such data.

***

The fourth post is a must-read. Here is Andrew's recap of what Mark wrote.

Mark trashed those critics who grumbled about the point deduction for wrong answers in the SAT. Mark explained why the point deduction scheme in fact simplifies life for test-takers, because the penalty makes it counterproductive to do "random guessing" (what Mark describes as "guessing without first reading the question"). Tests that don't deduct points for wrong answers are more annoying because smart test-takers should guess on certain questions based on their level of confidence in knowing the answer.

The unspoken point behind Mark's post is that the SAT is rigorously designed, and continuously improved, by lots of statisticians, and therefore it is the best instrument we have. In Numbers Rule Your World (link), half of Chapter 3 is devoted to the statistical work behind a niche issue in SAT tests, work that reduces specific biases due to race and other characteristics.

The biggest irony in the anti-SAT movement is that on the major complaints about the SAT, alternative instruments such as high school GPAs and reference letters are far worse.

***

In the last post of this series (link), Mark turned our intuition upside down. He argued that the students most likely to benefit from the SAT admission requirement are those good students who go to public schools – or if we extend this line of reasoning, good students from underprivileged backgrounds. That is to say, the SAT requirement benefits precisely those people whom its distractors detractors say they care the most about, with the caveat that the benefit accrues to the cream of that crop.

Ten years later, the MIT dean of admissions echoed Mark's point when he said that MIT uses the SAT score to identify disadvantaged students who are most likely to succeed at MIT.

unlike some other inequalities — like access to fancy internships or expensive extracurriculars — our empirical research shows the SAT/ACT actually do help us figure out if someone will do well at MIT.

***

This last discussion leads us to some general topics in the area of statistical measures. No metric is perfect, most metrics are correlated with each other, all metrics promote perverse behavior, alternatives often appear better only because of lack of data or analyses. See Chapter 2 of Numbersense (link) for more on measurement, where I dig into the anti-BMI movement. BMI is the widely accepted metric for assessing obesity; like SAT scores, the metric has attracted a small loud group of distractors detractors.

 

Changing metrics when you dislike the results

(Source: xkcd)

 

Chapter 2 of Numbersense (link) is titled "Can a new statistic make us less fat?" In the context of recent remarks by the acting New York State Governor, perhaps I should have called it "Can a new statistic save some people from dying?"

Trick question:

An executive tracks performance using a key metric displayed on a dashboard. The executive requests a review of the definition of the metric, then decides the definition must be revised, and after the revision, the new metric looks better than before. Which of the following is true?

a) Using the prior definition, the key metric was outperforming expectation

b) Using the prior definition, the key metric was underperforming expectation

***

The New York TImes reported that "Ms. Hochul said that some hospital executives have told her between 20 and 50 percent of their Covid patients are not suffering from severe symptoms, but that they are testing positive in the hospital incidentally, after being admitted for other reasons such as car accidents...As a result, beginning Tuesday, the state will begin to ask hospitals to break down how many patients are being admitted for acute Covid-19 symptoms, in an effort to further decipher this wave’s severity."

This is statistical alchemy that is frequently requested by people who don't like what the numbers are showing. It is a trick because they did not commit to restating the entire data series back to the start of data collection. This aforementioned issue is a "problem" only if they had recently changed the counting rules to include admissions for non Covid-19 reasons, therefore causing recent data to be inflated relative to older data, but that's not what they're saying.

Further, notice that they are commingling two separate re-definitions: one is Covid-19 vs not Covid-19; the other is mild vs "acute" Covid-19. I have never heard that "cases" only refer to "acute" Covid-19 cases, until now. The new definition relies on a subjective definition of "acute" and so they can report whatever number they want going forward.

To have any credibility, they need to (a) publish all the definitions that have been deployed in the last 2+ years, including the latest one, and (b) apply the latest definition to restate all historical data. Otherwise, no one can tell whether the trend is up or down since we are not comparing apples to apples!

To have maximum credibility, they should do (c) a comprehensive review of the definition, and correct for problems that undercount as well as overcount. For example, are they missing lots of cases found by people doing at-home testing? How are they dealing with people with multiple test results? (This is no small matter: if an infected person will take further tests till s/he tests negative, then each positive test is likely to be balanced by one or more negative tests in the near future. This dynamic affects how we can interpret the time series of positive tests divided by number of tests.)

***

In Numbersense (link), I trace how every few years, someone writes an article saying we need a new definition of obesity. The typical reason is that BMI classifies fit athletes as obese. I have never seen someone provide an analysis of how BMI misclassifies the average person. It's always the outliers. Notice in the above quote, the Governor did not give an aggregate proportion of what she considers misclassified. She said "some" hospitals gave her a concerning statistic.

As a data analyst, you have to think about the likelihood that the sample of hospitals she heard from is representative of all hospitals. Imagine you're working at a hospital for which most of the patients testing positive for Covid-19 are being treated for Covid-19. How likely are you to pick up the phone and call the Governor to tell her that you do not have a problem?

Thus, the sample she is working with is biased towards those hospitals experiencing this "problem". The real proportion of cases being "misclassified" is surely lower than what she announced.

Netflix, the data pioneer, gets flagged for data offence

The headline says it all. The Hollywood Reporter reported that "'The Witcher' Breaks Netflix Series Viewership Records -- with One Big Catch."

This should come as no surprise to my readers. It proves yet again you can use data to prove anything you want. Apparently, Netflix switched how it counts "viewers" in 2020.

Those not knee-deep in data work might not appreicate how measuring viewership can be so controversial. How hard is it to count how many people viewed a show?

In this case, Netflix announced 76 million households watched the series during the first four weeks of release. This seemingly simple statement includes a load of unspoken assumptions:

• Does each houshold have to watch all four weeks or one out of four weeks? (Nothing prevents one from requiring each household to watch two or three out of four weeks, either!)
  
  
• How much of each show must be watched in order to count as a "view"? (Hint: if the show is 60 minutes, absolutely no one will count 60 minutes. If you want to guess, select a number closer to 0 than to 60. Correct answer is disclosed below.)
  
  
• How many episodes must be be "viewed" to consider the series "viewed"? (The Witcher Season 1 has eight episodes, all released at the same time. I don't think you need a hint for this one.)
  
  
• Should each device and each sub-account be aggregated or not?

The above list isn't even comprehensive; there are even more subtle issues:

• What is being measured is not "viewing" but "streaming". The video player sends back regular signals to Netflix saying the player is still streaming a particular show. To measure actual viewing, the TV would have to track the viewers' eyes. (This is technically feasible although I think these companies aren't doing this.)
  
  
• Do repeat minutes count? If the rule is 10 minutes of at least two episodes, how do we deal with someone who watched 10 minutes of the same episode, twice?
  
  
• How to deal with fast forward, skipping, etc.?

***

As you look at those questions, think about whether there is a principled way to make these decisions. How does one argue that one out of four weeks is not a view but two out of four is? How does one argue that 10 minutes constitute viewing but 5 minutes don't?

It's pretty clear why you can use data to prove anything you want.

Still, what Netflix did is jaw-dropping. The previous definition of a view is "watching at least 70 percent of one episode of a series or 70 percent of a feature film". The new definition of a view - watching at least 2 minutes of one episode. You are reading this right; the 76 million households have "viewed" one Watcher episode for at least 2 minutes.

The first definition is already quite generous. The rule of the game is to get enough people to turn on the first episode, and it doesn't matter that they never return. Since Netflix has the famous recommendation engine, it's got the tool to drive traffic to the first episode of pretty much anything. The audience have to stick around for 70 percent of the running time, though.

How does Netflix justify dropping the bar from 70 percent of one episode to 2 minutes?

From the Hollywood Reporter:

Our new methodology is similar to the BBC iPlayer in their rankings based on 'requests' for the title, 'most popular' articles on The New York Times, which include those who opened the articles, and YouTube view counts," the company notes in its earnings report. "This way, short and long titles are treated equally, leveling the playing field for all types of our content including interactive content, which has no fixed length."

Netflix is right about Youtube. Youtube counts 30 seconds of run time as "viewing". Also, Youtube is autoplay by default. Once you start playing one video, it performs a version of "doom scrolling" so it keeps playing the next clip. I'm surprised people actually suggest that such "autoplay" views are not counted as views. Doing so would disqualify a ton of legitimate viewing as not views - and Youtube makes money from generating views!

The most objectionable part of the quote is "short and long titles are treated equally". According to Netflix, treating someone who watched 2 minutes of a 10-minute clip the same as someone who watched 2 minutes of a 240-minute movie demonstrates fairness.

You know it's a problem when the agreeable press not only noticed the data offense but also decided to stick it in the headline rather than burying it two paragraphs from the end of the article.

***

The issue of definitions is everywhere in data analysis. Covid19 tests and deaths. Flight delays. Mean tweets. Twitter users. Facebook video view time. Wind chill.

In Chapter 2 of Numbersense (link), I featured some doctors who believed that changing the definition of obesity will solve the obesity problem in the U.S.

P.S. [10/20/2020:

Mark Palko alerted me to his take on this news also. I think what he's saying is Netflix knows (of course) how much time each device "viewed" each minute of each episode, it just chooses not to publicize that data. Great point!

Here is another post by Mark that covers how Netflix uses "deceptive messaging and suspicious statistics". 

]

Numbers indeed rule our world, 10 years out

Statistics is the subject of generalizing data. In statistics, we study how to use imperfect data to draw general conclusions. We often have to wait for the future to reveal the evidence of success.

Ten years ago, McGraw-Hill published my first book, a best-seller, Numbers Rule Your World (link). I’ve hoped to explain five of the most profound concepts from my field in a way that anyone can understand and apply to their daily lives. I attempted the balancing act of featuring specific stories while bringing out the general principles. In the last couple of months, the Covid-19 pandemic has generated immense attention to data and statistics, and provided me confirmation, ten years on, that the contents of Numbers Rule Your World (link) have general applicability. It affirms that statistics is a great subject!

If you’re approaching my book for the first time, or re-reading it, here’s how you can work your way through it, making connections at various points to the Covid-19 crisis.

***

Chapter 2: Bad Spinach / Bad Scores

This is your perfect starting point. Half of the chapter details a disease outbreak investigation. You learn about the information networks the CDC puts in place to monitor unusual patterns of disease around the country. You discover data analysts whose job is to set off the alarm. How do they decide if the first handful of cases foretell an epidemic?

You encounter “shoe leather,” the painstaking collection of data to pinpoint the origin of an outbreak. It’s much harder than you might think. The U.S. politicians are making much noise about the early days of Wuhan. An epidemiologist may learn that 10 out of 15 first patients all visited the Wuhan seafood market before they got sick. But the Wuhan market might be a hugely popular destination, like Central Park in New York City. Finding 10 out of 15 people went to Central Park over a weekend could be more common than you think. If all 10 touched a particular rare species of wildlife, that's another story.

Another controversy of the moment is the huge cost of shutting down the economy, against the benefit of saving lives. I invite readers to think about this cost-benefit tradeoff. Because the disease outbreak was traced to a spinach processing plant in California, the FDA ordered a recall of all spinach, which depressed sales of salad greens for six months or so. The cost was much lower than for this pandemic, but the bacteria is well-known and thus predictable, and the lives saved were most likely zero, and even harder to pin down than in the current crisis.

People are already complaining that social distancing and other measures have not improved our numbers, by saying deaths were “unexpectedly low”. You discover how hard it is to prove the effectiveness of a policy decision because you need the counterfactual – what would have happened if the FDA did not recall the spinach.

Chapter 4: Timid Testers / Magic Lassoes

Everything you should know about the statistics of diagnostic testing and predictive models in one place. The motivating examples are (a) the fight against performance enhancing drugs in sports via anti-doping tests and (b) the false promise of terrorism prediction algorithms.

I wrote several posts (1, 2, 3) about the flawed Stanford antibody study that reported 50 times more infections in Santa Clara county in California than reported cases. Despite this headline, the raw data is also consistent with a scenario of zero proportion having antibodies. The statistical reason is the low underlying prevalence of antibodies, which caps the number of true positives in any sample. The number of false positives will be large relative to true positives, because the false positive rate applies to those without antibodies, which is the bulk of the population.

An extreme case of this phenomenon is terrorist prediction using a dragnet surveillance dataset. The set of terrorists is a drop in the ocean of people under mass surveillance. For a predictive model to capture most of these terrorists (low false negative rate), it must flag a lot of people – even a tiny false positive rate when applied to a huge population results in a lot of false accusations.

You see the distinction between a “chemical” false negative (positive) and a “practical” false negative (positive). To evade steroids testing, dopers use all kinds of tactics, such as handing in someone else’s urine. The diagnostic test finds a true negative but in practical terms, it’s a false negative. For swab testing, one may test negative if the swab doesn’t reach a virus-rich area, or if the patient is tested at the “wrong” time.

All tests have errors. What are the consequences of errors? Also, how likely will the error be discovered? For anti-doping, do dopers inform the test lab they made a mistake?

Chapter 1: Fast Passes / Slow Merges

The first chapter gives important insights on managing capacity, which is a pivotal issue for hospitals during the coronavirus crisis. I cover two case studies: (a) Disney’s management of theme-park capacity, and (b) state’s management of highway congestion.

The strategy of building more capacity is unavailable during a crisis, and wasteful in general. In this extraordinary time, some localities were able to bring up temporary hospitals. Transportation experts discovered that policies of regulating congestion are more effective. Highway ramps slow down the influx of new vehicles onto the highway, and are turned on before congestion arrives. The key is to delay as much as possible the onset of congestion. Sounds familiar? It’s the analogue to “flattening the curve”. Social distancing is supposed to reduce contacts, slowing down community spread. This tactic delays the peak demand for hospital resources. The chapter explains the scientific support for why such tactics work.

Chapter 5: Jet Crashes / Jackpots

Statistical significance is the topic of this chapter, one of the most common and also most commonly mis-used concepts in statistics.

In the Stanford antibody study referenced above, statisticians argued that the observed level of positive results is consistent with zero prevalence. This is a way of saying the result is not statistically significant. What we mean is that the study’s result (50 positives out of over 3,000 tests) would be considered normal even if nobody in the population actually had antibodies. This conclusion does not assert that the true prevalence is zero. The topic of statistical significance is made inaccessible to non-mathematicians by textbook authors piling on equations, but it need not be so. I even made a video on explaining "not statisitical significant" (link).

Before you apply a test of statistical significance, you must make sure you’re comparing apples to apples. In the context of air safety, you shouldn’t compare U.S. and foreign airlines on all routes; you should only compare them on routes they share, in which case, you’ll find no statistical difference. One of the bad stats that is circulating during this pandemic is the purported harm of ventilators. This is usually stated as the vast majority of patients put on ventilators eventually die (presumably, relative to those patients not put on ventilators). But only the most serious patients are placed on ventilators. The proper analysis should focus on patients who qualify for ventilators, and compare those who were put on ventilators to those who got other treatments.

We have already seen a host of invalid comparisons. Comparing complete data from years ago with incomplete data this year. Comparing regions that are in different phases of the epidemiological timeline. Comparing countries that have taken different approaches to managing the crisis.

Chapter 3: Item Bank / Risk Pool

This chapter addresses a deeper issue in using statistical averages. An illustrative example is educational testing like the SAT. Critics pointed to the gap in the average scores across racial groups as evidence of bias. But the racial grouping masks a third factor, which is differential ability between students. Due to various socio-economic factors, white students have higher average ability than black students. So the real question is: Among those students with comparable abilities, does the test still show bias toward one racial group? (The answer is not straightforward.)

During the pandemic, the “denier” faction has argued that the mortality rate of Covid-19 is “on a par with” that of seasonal influenza. Are those two rates comparable? The simple answer is no. Comparing mortality rates masks a third factor, which is the level of immunity in the population. Even at the most optimistic, the current level of immunity to the novel coronavirus is less than 20 percent (This statement generously presumes that (a) having antibodies confer immunity without question and (b) the New York City test result is not a scam, and the test is 100% accurate). By contrast, because of vaccination and recurrence, a good chunk of people have immunity to influenza. The mortality rate of Covid-19 should be compared to that of influenza among a population that has no immunity to either, or a population with similar levels of immunity to each.

The other example in this chapter explains how the insurance industry is an outgrowth of the law of large numbers. In any given year, health or auto insurers cannot predict with high accuracy which covered individuals will file claims, but the magic of statistics allows them to predict with great accuracy the aggregate claims. Disaster insurers run into trouble when rare events happen, such as when Class 5 hurricanes hit Florida. Suddenly, potentially all covered individuals tap into the insurance pool all at once. Insurers try to pool together risks of different types and different areas. The global pandemic is a huge threat to the insurance industry. Perhaps there are lawyers reading this, and they might tell me there are natural disaster exclusions. If those don’t apply, we will see a gigantic number of covered entities filing claims all at once.

Florida’s hurricane insurance market was dysfunctional also because the risk is predictable along some parameters. Those living inland realized they were subsidizing the coastal dwellers and wanted out. The coronavirus also creates different risk subgroups. The younger generations have much lower risks than the elderly. When social-distancing and other practices are applied to the whole community, the subgroup with lower risk peels away. A key difference is community transmission – the risk of hurricane damages does not spread by contact.

***

The above covers less than half of the book. Almost everything in those pages has direct relevance to the Covid-19 world. It’s quite heartening to find out ten years later numbers indeed rule our world.

Here’s a link to get your copy of Numbers Rule Your World. There are also translated copies in Chinese (simplified and traditional), Japanese, Korean, and Portuguese, as far as I know.

Excess deaths: why we want the cake and why we can't eat it yet

In my early March preview of statistical issues relating to coronavirus, I mentioned "people are counted as infected only if they are tested", and I also asked "how are deaths confirmed?"

For a while, the media ignored how meaningless the case counts were. Eventually, they got it, and that was great. But some experts convinced them that the savior is the death count. And now, in the last week, the reporters finally learned that death counts are subject to similar problems of definition.

The Economist just launched a tracker to look at excess all-cause mortality, using data from EuroMOMO. If during this pandemic, the total number of deaths from any cause is higher than normal, then it provides evidence (though not irrefutable) that Covid-19 is not a new name for influenza. Of course, we are finding excess deaths everywhere.

The next puzzle is that the number of deaths attributed to Covid-19 on death certificates is frequently lower than the excess deaths from all causes. This raises the possibility that Covid-19 deaths are under-reported. By itself, this is not necessarily true, because it is also possible that deaths from other causes are being under-counted.

However, it has also emerged that in many countries and regions, the published counts of fatalities so far have only included deaths occurring at hospitals for whom the deceased had returned a positive test for the virus. A good number of deaths occurred in care homes, hospices, and even private residences, and these have been ignored. In fact, many hospitals refused to admit patients who do not report severe symptoms, and told them to isolate at home.

The excess all-cause mortality data take a bit more time to collect, and perhaps not by accident, as these statistics come to light, many countries announced that they are revising how they count Covid-19 deaths, essentially admitting they were under-reporting.

***

The lesson related to data integrity is being taken up very slowly. Even as the Economist published the valuable tracker of all-cause mortality, they wrote:

... people have become grimly familiar with the death tolls... these numbers give a better indicator of a country's trajectory than do counts of confirmed cases, which largely measure how many people have been tested. Nonetheless, official covid-19 death tolls still under-count...

This is better than most but still repeats several fallacies:

a) Given the unknown fatality rate of Covid-19, the death count is a proportion of the case count. If data integrity is not at issue, the death count is a worse indicator of a trajectory than the case count because it just gives the same information after a time delay. If death tolls are better than confirmed cases, it's because we believe the data on deaths are less bad than the data on cases.

b) The emerging evidence shows that the unadjusted death counts are wildly inaccurate, so I'm not sure you can say the data on deaths are "the lesser evil".  In Lombardy, the Economist estimated the "true toll was about 120% higher" than the confirmed deaths. In Spain, the true number might be 60% higher (off a much larger base).

c) Confirmed cases is not a simple function of how many people got tested. More importantly, it is measuring who got tested. I know I have been hounding this point to death. In statistics, we care not only about the size of the sample but the composition of the sample! We could test lot of people with no symptoms and still have few confirmed cases.

d) The under-counting of deaths is still partially due to lack of testing. It is true that Covid-19 testing is not required in those all-cause mortality statistics but without testing, the count of deaths is clearly less reliable, just like the count of cases.

e) Deaths can also be over-counted, especially without testing to confirm coronavirus. How well can a doctor tell if the patient who died of pnuemonia had Covid-19 or some other illness?

f) If it's broke, you can't fix it. Countries change the definition of deaths mid-way but won't (and can't) change the past. This renders the death count series useless, unless you're willing to adjust the data. This was a key point I made about new metrics of obesity in Chapter 2 of Numbersense (link).

***

So, if you've been reading this blog, you know interpretation of statistics is no simple business.

While compiling these excess mortality numbers is necessary and important, here is something you should know, and I'm not going to sugarcoat it.

Imagine wearing glasses or sunglasses in the summer. You walk into an air-conditioned shop. You must wait for the fog to clear.

These excess mortality numbers are shrouded in a fog right now. It is actually somewhat meaningless to discuss them. This is because of a substitution effect. Some of the excess deaths we're seeing are accelerated deaths. (Of course, all excess deaths are accelerated deaths.) What proportion of the excess deaths are relevant will be up for debate once we wait long enough to see clearly.

A store usually runs an annual sale on New Year's Day only. This year, the store starts the discounts on the day before. After New Year's Eve, the store reports 50% jump in sales versus last year ("excess"). Was the promotion particularly effective this year? You don't know until after New Year's Day. When you compare the total sales of both days against the two days last year, you'll find that the jump is less than 50%. Some customers just showed up one day earlier.

 

 

 

 

The waiting game facing our decision-makers

This week saw a spike in COVID-19 cases in the U.S., and with it come the finger-pointing and blame game. First, fingers are pointed at other countries who made the wrong calls and let their cases spike, and possibly exporting the virus here. Then, blame falls on which state governor or mayor took the information seriously or not.

The reality is that each governer or mayor is faced with a very real challenge: the choice of action or waiting when there is very limited, and unreliable data in front of her/him.

I'm reminded of the following diagram, from Chapter 2 of my book Numbers Rule Your World (link). Half of this chapter looks at epidemiologists who monitor disease outbreaks around the country. They are the ones who sound the alarm when there is e-coli in our greens, which frequently lead to wholesale recalls of produce. Every week there are a few cases of disease but most of the time, they don't lead to large-scale outbreaks.

The left column shows what the analyst was looking at up to August or September in different years. In 2005 and 2006, she found a spike in cases. Each week, she had to decide whether to sound the alarm or wait another week. In 2006 (top row), if she didn't sound the alarm by late August, she would have been late to the outbreak, and the media would blame the CDC for underestimating the outbreak.

In 2005 (middle row), if she had sounded the alarm with the spike in September, she would have made a false alarm. The media would criticize the CDC for destroying the produce industry for six months or longer.

The point is, it is a tough call when you have limited data. It's pointless to slam an earlier decision after more data came in. Too much delay always looks bad if things get worse. Overly drastic action always looks bad if the worst did not materialize.

The most scary part of the situation - if you're the decision-maker - is the daily dread. Every day, new data arrive to judge your previous actions. As long as cases continue unabated, you'll be branded a failure. It's a situation that rewards careful interpretation of the data.

***

That's the lesson of Chapter 2 of Numbers Rule Your World (link), and it's here globally in an unprecedented scale.

 

Eight unanswered questions about the coronavirus

Coronaphobia has arrived in the U.S. This past weekend, people started raiding stores for masks and tissue paper.

Ultimately, the one important question we're trying to answer is:

Is COVID-19

(a) just another flu virus

(b) a particularly lethal flu virus that will have a short-term impact (similar to SARS, which  killed about 800 people but did not last beyond one flu season)

(c) a deadly virus that would fuel a much-feared flu pandemic that could kill millions

This is the crucial question because its answer should condition our response to the virus.

(a) typical caution

(b) short-term vigilance

(c) large-scale mobilization and preparation.

 

Small Data Emergency

Because this coronavirus is new, and the disease is spreading while humans are playing catch-up, what we face is a classic "small data" emergency. We have very limited amounts of data (the known and suspected cases), incomplete data (under-reporting, possible hiding or manipulation), data of dubious quality (self-reporting, hastily assembled tests, unpreparedness), assumptions that later turn out to be false, the potential black-swan fallacy (failure to imagine what hasn't happened before), and non-stationary data (old data might not be wrong but become outdated as the virus evolves).

This emergency is typical of any disease outbreaks. This one is larger in scale, and international in scope. To understand the entire process of outbreak investigations, and the pressure investigators work under, I refer you to Chapter 2 of Numbers Rule Your World (link).

Before we complain about incompetence, confusion or muddled messages, we should have some sympathy for the epidemiologists, medical researchers, data analysts, policy analysts, public health advocates, and others who are doing their very best at a very challenging moment.

 

The Overdue Pandemic Backdrop

"Politicians can ignore it, and may even get away with it. But when a pandemic eventually strikes, and history suggests that one is long overdue, the consequences could be devastating." This is a quote from an article published by The Telegraph. In 2008, more than a decade ago.

Much of the medical community have been predicting a pandemic for years and years. They thought SARS was the big one but it wasn't - it did not pass from human to human quickly enough. According to the same article, WHO warned that the H5N1 avian flu could "seed a human pandemic that ... hospitalise 30 million, of whom some 20 per cent would die." That projection was way off the mark.

For a pandemic to occur, we need a mix of infection rate and fatality rate. Coronavirus has a much higher infection rate than SARS, and transmits between humans but it is less lethal. However, at this early stage, our estimates of the infections and fatalities are far from accurate.

I've put together a list of unanswered questions. Whether COVID-19 is another false alarm or not depends on what answers we eventually receive.

 

1) Test accuracy affects the infection count

All tests currently in use are rush jobs which obtained "fast-track" approval. China accepted seven tests within the first two weeks of the outbreaks. These tests use the "nucleic acid" method, which amplifies genetic materials collected from throat swabs, and according to SCMP (the paper of record in Hong Kong), an official later said they were only "30 to 50 percent accurate".

The low accuracy rate should not surprise. Fast does not produce accuracy; in fact, fast probably means less reliable. Our knowledge of the new virus is very limited in these early days.

It's not even easy to tell when a test is inaccurate. Many anecdotes circulate of patients who initially tested negative. The media immediately screamed "false negative"! But it's not that simple. The patient could have caught the virus between tests, especially if s/he is quarantined or living in proximity of someone with the virus. Or the virus may be mutating.

Scientists are negotiating a vicious cycle of inaccurate tests, and inaccurate measurements. A BBC report describes these challenges.

To learn more about diagnostic testing, see Chapter 4 of Numbers Rule Your World (link), in which I look at how the sports world tests for performance-enhancing drugs. The trade-off between false positives and false negatives is a key issue. I also make the distinction between a "chemical" negative and a "real life" negative. An athlete might test negative in a doping test but could still be a doper because the test does not detect the substance s/he is using, or is tricked by a masking agent, or is ambushed by an evasive action. So, it's possible to have a test that is accurate for what it's designed to do but still misses what we'd like to know in real life.

 

2) The elephant in the room: people are counted as infected only if they are tested

The big catch: someone can be counted as infected only if the person is tested for coronavirus. No test, no reported infection. This ABC News report states that South Korea has tested tens of thousands of suspected cases while the U.S. has conducted fewer than 500 tests, as of February 27, 2020. This means the number of infections in South Korea is probably over-stated (see point #1), and the number of infections in the U.S. is under-stated.

If we believe we face scenario (a), then not testing makes sense. It's simply not useful to know how many have the common flu viruses at any given time. But if we are in scenario (c), then stopping the spread of the coronavirus is paramount, which requires knowing who has it.

 

3) Noise coming from common flu viruses

Remember we are in flu season. According to the CDC, influenza and pneumonia causes 56,000 fatalities in the U.S. each year, the eighth leading cause of death. The fatality rate is 17 per 100,000 or 0.02%. This implies an annual infection count of 327 million cases of the common flu (despite the widespread use of the flu vaccine). On average, that's almost 2 million infections a day if we assume the flu season lasts 6 months (the bulk of infections actually occurs between December and March).

Against this backdrop, it is easy to attribute infections or deaths wrongly to COVID-19. The use of indirect tests increases the chance of mis-attribution.

For example, CAT scans find evidence of lung damage but using such tests can make infection counts inaccurate in two ways: lung damage can be caused by other viruses especially in flu season, and not all people with COVID-19 suffer lung damage.

Partly as a reaction to the low test accuracy and/or desire for speed, the Chinese authorities switched the definition of an infection, now counting people with symptoms without waiting for a test confirmation, according to this BBC report. This change in definition led to a spike in "infections" but the new count probably contains more false positives while having fewer false negatives.

Inflating the infection count can be justified by the desire to minimize false negatives, which reduces the chance of people unknowingly spreading the virus around. Overcounting infections, however, leads to an under-estimate of the fatality rate.

Similarly, how are deaths confirmed? Are some of these deaths caused by other flu strains?

 

4) The connection between infections and deaths

The following news item from Twitter is typical of the media's set narrative on the coronavirus:

The two statistics they focus on are the number of infections (cases) and the number of deaths.

Those two numbers are linked. If the number of infections are growing fast, and deaths are not increasing proportionately, the fatality rate is coming down, so that should be a reassuring sign. The fact that both infections and deaths are growing doesn't mean anything. It's the growth of one number with respect to the other.

 

5) The survival rate

One thing that the media refuse to cover is the survival rate. COVID-19 does not cause certain death, far from it. People have recovered. Because it may take a couple of weeks or longer to be cleared, the count of survivors lags. Our estimate of the survival rate improves over time as more people exit the treatment period.

 

6) Intermediate metrics

Another key metric to track is the severity of the symptoms. Among the group of patients who are still surviving, some are mildly sick while others are seriously ill. The media also fail to report on any intermediate metrics.

 

7) Rates not counts

In all of the above, I talk about rates not counts.

Counts are misleading. If we take the example of the Twitter news item and apply its logic to the common flu viruses, then we'd have to report that during each day of the flu season, there are two million new infections, and 300 deaths! Sounds scary but not really.

[See comment below. Counts are relevant for logistical use. The number of patients requiring hospital stays certainly can overwhelm available resources.]

 

8) Community transmission

SARS has a high fatality rate, close to 10 percent of those infected died. However, the virus never quite mastered human-to-human transmission and because the infection rate was low, it eventually failed to become a pandemic.

Current estimates of the fatality rate of COVID-19 is under 2 percent on the low side and maybe 4 percent on the high side. This fatality rate is much lower than SARS but the infection rate is clearly higher, and it's clear that the virus can spread among humans.

 

***

It all comes down to infection rate and fatality rate. The trouble is we have "small data" so our estimates of those rates are limited. Knowing the sources of inaccuracy is helpful to understanding which direction the rates are moving. Also, secondary metrics like severity and survival rates provide some complementary information while we wait for more data to arrive.

Don't panic, but don't take unnecessary risks.

 

P.S. [3/9/2020] Some additional comments given new developments since this post was written

South Korea and Italy are both spoken of as hot zones for COVID19. This is true in terms of known infections but remember point #2, the number of reported infections is highly correlated with the number of tests performed. As of yesterday, New York City said it did less than 100 tests. In South Korea and Italy, they did tens of thousands of tests.

It also appears that the hospitalization and death rates in different countries are different. So there are even more variables, due to factors such as availability of medical facilities, quality of care, age distribution of patients, and so on.

In times of paranoia, seek out data and stick to facts

All around the world, people are getting very afraid of the coronavirus. What's happening now is a live version of a disease outbreak investigation, which is discussed in Chapter 2 of Numbers Rule Your World (link).

To understand what's going on, you should see this as the exact opposite of "big data". We have a new flu virus. There is almost no direct data on this virus. The data are dribbling in as cases are being reported. We have a couple hundred deaths.... in a country with almost 1.5 billion people. Public health officials must make key decisions now before more data become available. But we have already passed the most dangerous phase, because they already found what causes the disease.

I saw a graphic last night, shown during an interview with an expert who investigated the SARS epidemic. Supposedly the mortality rate for the coronavirus is about 3 percent. This number has a large margin of error at this stage because of the shortage of data. This shortage is due to the newness of the phenonemon.

The closest relative to the coronavirus is the SARS virus which had a mortality rate of about 10 percent, which is three times higher than what we currently know about the coronavirus.

We also know that the coronavirus spreads faster than SARS. Most of the news reports I read mention the number of infected but rarely the number of deaths. This is a major oversight.

What's fearful is a deadly disease. A disease that infects lots of people but acts like the common cold is not a public-health crisis. So as data roll in, we should keep an eye on the mortality rate.

The mortality rate is the number of deaths divided by the number of infected. Both counts are growing each day. If the growth of infected is faster than the growth of deaths, the mortality rate will go down (not up!). The mortality rate will go up only if the growth rate of deaths exceeds the growth rate of infected.

In fact, by the time I finish this post, the latest report said the coronavirus death rate is 2 percent, so it has already been adjusted downwards because the flu is spreading fast. (The death rate of regular flu is about 1 percent, according to the graphic that was shown last night. This level acts as a control for interpreting the mortality rate.)

That report contains a typically alarmist quote by a medical professional: "There are likely to be many times more cases in Wuhan than officially confirmed". Okay, but if that is true, the real mortality rate is many times smaller than what is official! In terms of missing data, it's the unreported deaths that are more significant than unreported infections.

Also note the network effect. It's very unlikely that the growth rate of deaths can be higher than the growth rate of infected because infection grows exponentially due to transmissions. Deaths don't transmit.

It would also be useful to know what the cure rate is - because some of the infected will be in the treatment stage. During the report last night, the journalist wrongly asserted that 97 out of 100 infected will survive. This interpretation is correct only if we exclude those infected that are currently in treatment. (The technical term is right-censored data. We don't yet know how many in this subgroup will survive or not.)

While one should be cautious because this coronavirus is new, in times of paranoia, one should seek out the data. Ignore the unfounded rumors, innuendo and folklore. Stick to the facts. So far, the data aren't that alarming.

***

I n Chapter 2 of Numbers Rule Your World (link), I describe two modes of statistical modeling. Disease outbreak investigation illustrates one mode, in which the chief concern is establishing cause-effect relationships. This is always hard if you have observational data - without controls. But it's even more challenging in these crisis situations, with small drips of data. If you understand what statistical tools are available, and what the limitations are, you have a better grasp of what's going on right now.

 

Image credit: By https://www.scientificanimations.com - https://www.scientificanimations.com/wiki-images/, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=86358105

 

P.S. [2/3/20] See also my post on Junk Charts about a graphic used by the New York Times in a report on the virus.

Too much information as bad as no information

I haven't read Kartik's book but it looks like something I'd enjoy.

In this interview with Verge, he cited the following experiment:

One problem with grading in college courses is that different TAs are more or less lenient. So they used an algorithm to normalize or modify the grade so that the level of leniency was consistent. Then, one group received minimal information about how the algorithm worked, a second group got some high-level data, and the third got all of the information on how the algorithm worked and the raw data and all the changes made. The result was that the level of trust in the third group was back down to the same level as the group that didn’t receive any information. So it goes to show that if you reveal that much information, it’s as if you reveal nothing.

This discussion is about transparency when it comes to algorithms. His recommendation is to provide "just enough" information but not "too much". The test group that received details  about the algorithm trusted it less than the group that received high-level information, and about the same as the group that was kept in the dark.

Kartik cites this experiment in the context of explaining to end-users how algorithms work.

The above commentary is equally valid when explaining algorithms to decision-makers, such as senior managers at an organization. Most people experienced at this quickly realize the futility of explaining minute details of an algorithm. Decision-makers are satisified if they gain an intuitive understanding of how the machine functions, and if they observe that the outputs "make sense."

***

One important issue that isn't addressed in the interview is gaming. More transparency is more fodder for those who want to game the system. I discuss this in the chapter about credit scoring in Numbers Rule Your World (link): we can now correct our credit history data, but this will clearly introduce a bias because people will remove incorrect, negative data but not incorrect, positive data.

 

[PS. After I wrote this, the current controversy around Boeing 737 MAX provides a counter-point. The current theory about what caused the crashes (at least the Lion Air one) is that an automation software misdiagnosed the problem. Apparently, Boeing decided not to inundate pilots with too much information and did not disclose the software prior to the Lion Air crash.]