Big Data, Plainly Spoken (aka Numbers Rule Your World)

The unexpected gift of a well-conducted study

We're drowning in a quicksand of bad-quality data during this pandemic, and frustratingly, the situation is getting worse, as we enter the second year. It would have been a challenge to measure the real-world impact of vaccines because of the mass vaccination initiatives. We then made our lives much harder by (a) destroying the placebo groups in the original randomized clinical trials, fossilizing them at 4 months of follow-up (see this post); and (b) applying different rules of masking, testing, hospitalizing, attributing causes of death, etc. to vaccinated and unvaccinated subgroups, literally injecting massive biases into observational data.

It comes as an unexpected gift to find a study that takes on these challenges seriously. Today, I review the preprint of the REACT-1 Study from the U.K. There is a large team of people involved, many of whom from Imperial College.

***

In this study, the researchers collect swabs from randomly selected people from all over England. The study is ongoing, with monthly cohorts. The report I'm reading covers Round 13, which was conducted from mid June to mid July. This cohort is monumental because Round 13 spanned a critical point in the case curve in the UK.

In the period leading to Round 13, the U.K. government was congratulating itself on the hugely successful vaccination campaign, about to declare mission accomplished. But by July, the virus has returned with a new rigor. In Round 13, the researchers analyzed about 100K swabs, with a positivity rate of 0.6%. That is more than four times higher than the positivity rate in Round 12.

The positivity rate found in the REACT-1 Study - especially after adjustments and weighting - can be generalized to the population of England. That's because the study utilizes a random sample. The swab tests find not just symptomatic but also asymptomatic Covid-19 cases. This study gives much better information than most real-world studies - which usually tap into a database of positive test results. A severe problem with those studies is selection bias. Vaccinated people, people with mild symptoms or disease, and people who live in lower-exposure areas are less likely to get tested, so their cases are under-counted. (That said, the "good" study still has to deal with biases, which I'll get to later in this post.)

***

There are many things we can learn from the REACT-1 study. The first thing is that the chance of anyone catching the virus - regardless of vaccination status - is low. Only 6 out of 1,000 people in England were actively sick with Covid-19 during July. The chance of getting severe Covid-19, getting admitted into hospitals, or dying from Covid-19 is a fraction of that number. The danger comes from the spread of the virus, so that a large base of the infected produces a large number of deaths. (Even at its peak in England - Round 8, the prevalence was 1.6%.)

What about the 994 people out of 1,000 who did not get sick during Round 13? We are frequently misled into thinking that the fully vaccinated subset were protected by the vaccine. The fallacy is made clear when you ask about those who are unvaccinated. One can't get infected if one isn't exposed. A lot of these people - vaccinated or not - did not get exposed to the virus during the study period. Most of the calculations you see out there implicitly assume that everyone is exposed every month, month after month.

I am not saying none of those 994 people got exposed. Some proportion of those were not infected because they did not get exposed. This is a missing data problem that we haven't grappled with yet. If they do get exposed, it's likely that the vaccinated ones have better outcomes; better may mean 30-50% reduction in chance of getting sick; we now know it's not anywhere close to 90%.

***

In "explaining" the surge in cases in England in July, the study's authors pointed the finger at unvaccinated 13-17 year olds. They drew this conclusion by comparing the percentage increase in infection rate between Rounds 12 and 13 by age group. See this chart (with my edits):

The study estimates that the prevalence of Covid-19 in the 13-17 age group jumped from 0.2% to 1.6%, which is a nine-fold surge, compared to an aggregate surge of about 4 times.

Are the teenagers responsible for the entire surge? To answer this question, I looked up the age distribution of England's population (shown on the chart in the green boxes).

From this analysis, I found that 13-17 year olds accounted for 15% share of cases in this period of time. The bulk (60%) of cases affected those between 18 and 64.

The above analysis contains one hopeful insight. The 65+ age group, which enjoys the highest vaccination rates, is getting some measure of protection, as indicated by the lower share of infection. This goes a long way to explaining why deaths in England have not increased as much as infections - younger people are getting infected, and they tend to recover more readily.

(Statements about deaths must be interpreted with great caution. This study is typical: the cutoff date of Round 13 was in the middle of the surge of cases, which means the analysis covered a period of time when the deaths arising from the surge of cases have not occurred yet. The researchers would never return to update the analysis and so we are always dealing with over-optimistic, premature estimates of effectiveness against deaths.)

***

Next, I made a similar analysis breaking the study population down by vaccination status: fully vaccinated, partly vaccinated and unvaccinated.

This leads to a surprising finding that the surge in infections was even stronger among those who were vaccinated (at least one dose) than those who were unvaccinated. This problem illustrates the difficulty of observational datasets. There are so many possible explanations for this. My own favorite is that many vaccinated people have stopped taking precautions.

***

Another nice feature of the REACT-1 preprint is the careful description of how they interpreted test results. It's not a straightforward science - something I explained in Chapter 4 of Numbers Rule Your World (link) using testing for steriods and HGH as a case study. Someone has to draw a line on some continuous scale to decide who's positive and who's negative.

These UK researchers described their positivity criteria as "both E and N gene targets were detected, or if N gene was detected with CT value < 37". CT is cycle threshold: the lower the CT value, the higher the viral load.

I highly recommend reading this section (pp. 10-11) of the paper. You will learn that to figure out what positivity threshold to use, the scientists need to procure positive and negative controls (samples known with 100% certainty to be positive or negative), these controls may be synthetically created, experimenters may be blinded or unblinded, testing analysis may be done externally or internally, multiple experiments by multiple labs may be required, these results may not agree, etc. In other words, scientists don't wave their hands and write down CT < 37; there is method to the madness.

Unfortunately, there does not seem to be a global standard for what test to use and what positivity threshold to apply. This is yet another aspect of the data quality problem I mentioned up top.

***

While the selected participants represent the whole of England, the study cannot escape selection bias in the form of non-response. The researchers reported that response rates (of sending back the swabs) have declined steadily and was at 12% in Round 3. That's a very respectable number.

What's concerning is that the responders were clearly different from the non-responders. Response rates for younger people and people living in deprived neighborhoods were only in the 5% range. Those, of course, are also the least vaccinated subgroups.

As if to prove the point that biases pervade every corner of real-world data, the researchers also reported finding selection bias among those who gave them permission to link their NHS records. Ex ante, it is not obvious why there should be a difference in infection rates between those who gave consent and those who didn't.

It turns out vaccinated people who gave consent are more likely to get infected than vaccinated people who didn't give consent. The reverse is true for unvaccinated people.

(Linking is used to conduct off-book accounting as described in this post. This process disappears vaccinated, and infected people because of some arbitrary case-counting window - which cannot be applied to the unvaccinated. In the case of England, the 2D+14 window means that no cases are counted on the vaccinated side until at least 14 weeks after the first shot because of a long delay between doses, based on embarrassingly wrong analyses by advisors to the UK government. This gimmick puts our most creative corporate accountants to shame.)

***

We are drowning in bad-quality data. Don't trust any study that gets published. Be discriminating, and value those researchers who are making an honest effort at doing good analyses. It takes a lot of work to properly handle real-world data, so let's take a moment to appreciate those who tackle this task seriously.

 

 

 

 

Sizing up the pandemic

From the beginning, the U.S. has never gotten religion about accurately measuring the pandemic. In March 2020, I wrote a piece for Wired about the need for broad-based testing because selectively testing only severe cases is bound to give us a biased picture of the crisis. It was known even then that a significant portion of infections is asymptomatic - which constitutes a silent path by which the virus passes from person to person. The situation has not improved much since. Recent counting rule changes imposed selectively on vaccinated people have made it far worse but that's the subject of a different post.

I'm delighted to report on a valiant effort to correct this oversight - in Slovenia, a country of about 2 million people. The relevant papers have just been published here and here.

***

Because we want to minimize bias, randomization is our friend. I have previously explained how randomly assignment treatment in a clinical trial creates covariate balance (here) which allows researchers to assume "all else equal" except for the treatment.

In this case, we want to test a random sample of people selected from the population to avoid the self-selection bias that permeates the usual counts of "cases". The demographic and medical profiles of the sample should mimic that of the entire population. In the Slovenian study, they apply standard statistical sampling methodology, as follows: the country is divided into 300 census districts; from each district, they select 10 persons at random; a total of 3,000 Slovenians were sent communications inviting them to participate in this study; just under half agreed, leading to a sample size of 1,368.

Because of this random (i.e. broad-based) testing, the researchers uncovered not only self-reported symptomatic cases but also asymptomatic cases. All participants were given PCR tests at the start of the study in April, which was towards the end of the first wave in Slovenia. The estimated prevalence was 0.15%, or 150 per 100,000.

It's important to understand prevalence as a measure of infections at a point in time (late April-early May 2020 in this case). The PCR test reveals whether someone is currently or has recently been infected with the novel coronavirus. This metric does not answer the important epidemiological question: what proportion of the population has been infected since the start of the pandemic?

***

The research team's second paper deals with this question - the canonical approach is through serological testing i.e. testing for antibodies. Most people who have been sick with Covid-19 have developed antibodies. So we expect the proportion of population that have had Covid-19 to be larger than the proportion who are currently/recently infected. Technically, we say the seroprevalence is greater than the prevalence.

I have discussed serological testing before - at the start of the pandemic. Two research teams - one at Oxford and one at Stanford - jumped the gun when they published serological studies during the first wave of Covid-19. The Oxford team (in)famously declared that up to 40% of the U.K. population have already been infected with SAR-CoV-2 by April 2020 - a result that has proven extremely embarrassing. To be precise, they used a mathematical model to come up with the alarming headline, and called for serological studies. The Stanford team argued that total infections in California were 50 to 85 times the reported number of cases - back in April 2020. This group actually performed tests to derive their estimate. Their analysis was roundly criticized. Recent handwaving analyses tend to multiply cases by 4 times, which gives you an idea of the magnitude of the error in the Stanford study.

Meanwhile, I'm still expecting that one day, the New York State Department of Health will release the data behind the almost daily pronouncements by Governor Cuomo - again in spring and summer of 2020 - that at least 15% of New Yorkers have been infected. Those numbers were supposed to have come from antibody testing throughout the state of New York but no data or science has seen the light of day. While not as far-fetched as the two other studies, that estimate of seroprevalence is also clearly repudiated by subsequent waves of Covid-19.

In the Slovenian study, the unadjusted estimate of seroprevalence around mid-November 2020 (in the middle of a devastating second wave) was only 4 percent. At the time, the cumulative reported cases was around 27,000, or 1.3% of the population. This estimate is much more solid than the previous studies because these researchers used a random sample. The design of research studies matters a lot - when devices like randomization are utilized, fewer adjustments and assumptions are necessary to interpret the data. (See my three-part talk to learn more about research methods.)

***

Much of the second paper deals with the complications of the analysis even though the team utilized a random sample. The study interviewed participants at 3-week intervals. There were two rounds of blood samples (April 2020, and October 2020).

In April 2020, the reported cases (about 700) was 0.03% of the Slovenian population. With serological testing, they estimated seroprevalence to be roughly 0.8-0.9%, so roughly 18,000 people have been infected. More precisely, they estimated that 0.8-0.9% of their random sample have evidence of SARS-Cov-2 antibodies (that are true positives), and generalized that result to the entire population.

The raw data actually showed 3 percent, which was revised down to 0.8-0.9%. This is because of two factors that artificially inflated the unadjusted number.

The first is test inaccuracy. All tests are inaccurate to some degree - which is a key message of Chapter 4 of Numbers Rule Your World (link). In this situation, because the vast majority of Slovenians have not been infected, false-positive errors can be devastating to any analysis. Even a small false-postive error rate (e.g. 2 or 3 percent) gets amplified by the large base population into an abundance of incorrect positive test results. Correcting for test inaccuracy therefore pulls down the number of positive cases, and thus seroprevalence.

We have encountered this issue before - when pointing out the methodological weaknesses of the 2020 Stanford study. Their initial analysis did not correct for false-positive results. But it's actually a really hard problem. To know that a positive result is definitely a false positive implies that we know the true result - if we did, we would not need to run a test. In practice, the researchers must obtain an estimate of the probability of false positives from other studies. 

For example, they gather blood samples collected for other reasons - preferably from before first half of 2019 when they are sure the blood samples did not have any SARS-Cov-2 antibodies (or RNA, depending on what test). Then they apply the test to these blood samples, and check what proportion of them give positive results despite clearly being negative.

Such definitively negative blood samples are hard to procure and so our knowledge of test accuracy is limited. Many researchers just trust the test manufacturer's claims, which is a bit shocking since they market their tests, in essence, as 100% accurate. The Slovenian team did a literature search, and discovered that the accuracy of the antibody test they were using was not 100%, and the accuracy measures varied widely from study to study. Thus, the second factor the researchers adjusted for was the variability of test accuracy. (*)

***

Because of concerns about the first testing platform, when they analyzed the second set of blood samples In October 2020, they switched to a different testing platform, which appeared to have better and less variable accuracy statistics.

Despite random sampling, there may be non-response bias - the people who chose to participate in this study are not necessarily the same as those who opted out of the study. If the non-responders are different from responders, the result from this study cannot be generalized to the entire population without more corrections.

If you notice, this study is much more rigorous than the real-world vaccine effectiveness study that I have recently featured - those studies sweep most of the biases under the rug.

Nevertheless, the responders and non-responders look similar enough - by the variables they analyzed. It is therefore not surprising when the authors reported that a further correction for non-response bias did not materially change their seroprevalence estimates.

***

This Slovenian study is a good example of proper science that doesn't cut corners. What's notable is that seroprevalence grows at a much lower rate than reported cases. While cases have jumped almost 40 times during the six months of the study, seroprevalence has risen about 4 times (call it 10 times if we allow for the error bar).

 

P.S. [*] The study employed Bayesian methods throughout. Here is how they adjusted for variability of test accuracy. Different prior studies have come up with a range of estimates for test accuracy. If one doesn't adjust for this, one would take the average value from these studies, and apply this to the analysis. For example, if the test specificity (true negative rate) is 98%, then you assume exactly 2% of the negative blood samples would show up as false positives.

In a Bayesian model, the test specificity is treated as a variable: its average value is 98% but it can vary within some reasonable range (as informed by the variability observed across those prior studies).

 

 

 

 

 

 

 

A Look Back at 2020

There is no bigger news in 2020 than the pandemic. This once-in-a-lifetime (hopefully) event has shadowed much of my blog writing last year. I'm glad you've stuck around, and if you've been reading from the start, you'd have been ahead of the curve on all the key issues. Data have been central to navigating this new world, however messy, sparse, and incomplete they are. I'm confident you've picked up more about statistics on these pages than in an entire statistics course.

In this post, I've selected some of the posts that have aged particularly well. If you missed these before, be sure to read them now!

***

In mid March, as New York was poised to enter a lockdown, I predicted that this lockdown would have a different ending than those in China or South Korea.

The second type of lockdown - soon to be exemplified by the U.S. - is based on severely limited data. It is based on the assumption that everyone has been infected. The necessity of this kitchen-sink assumption is driven by the lack of knowledge. It is a position of weakness. The duration of the lockdown will be longer, the spread of infections will be wider, and the long-term costs will be higher. It is not the case that everyone has the coronavirus but we have to assume it because we are not testing nearly enough people.

Nine months later, the U.S. has not tamed the virus while the economy has suffered.
(1) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/two-types-of-lockdowns-why-us-must-rethink-its-testing-policy.html

In March and April, when news outlets were one-upping each other with variations of log-scaled charts showing the "exponential" rise in Covid-19 cases, I pointed out that the growth curves were sub-exponential, which also means the log scale is inappropriate. Today, log-scaled charts have disappeared because they have been proven useless.

(2) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/note-about-fitting-and-visualizing-exponential-models.html
(3) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/new-video-validating-data-science-models-a-case-study-with-covid-19-data.html
(4) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/another-note-on-fitting-exponential-models.html
(5) https://junkcharts.typepad.com/junk_charts/2020/06/presented-without-comment.html

In May, I was already drawing the connection between advocating antibody testing and pushing herd immunity:

In places that have fallen down on rolling out diagnostic testing (the U.S. and the U.K., for example), the officials have started talking up “antibody testing”... Antibody testing is a rear view mirror. If someone is found to have antibodies, it’s because the person has recovered from the disease. It follows that the person was infected, and so the person could have died. Antibody testing without diagnostic testing means death over life.

In the same post, I made the following alarming calculation:

With a 0.1% fatality rate, conservatively set at that of influenza, the U.S. would suffer over 200,000 deaths to get there. The path to herd immunity is paved with body bags.

The current U.S. death toll is over 300,000 and counting.
(6) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/the-shift-to-antibody-testing-is-a-choice-of-death-over-life.html

Nevertheless, several countries (U.S., U.K., and Sweden, being the most prominent) continued to deny advancing herd-immunity policies, while most observers recognized them as such. In June, I pointed out that one cannot both believe the coronavirus is the common flu and support herd immunity through natural infections because influenza is seasonal, signifying that natural infection does not confer immunity.
(7) https://junkcharts.typepad.com/numbersruleyourworld/2020/06/coronavirus-has-not-gone-away-neither-have-these-covid-fallacies.html

In October, I objected to herd immunity as a public health objective as it rewards the wrong behavior. I warned against bystander behavior because "you have tied your health to the actions of people who hold opposite views as you." Herd immunity is mathematical immunity, not biological immunity.
(8) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/i-dont-like-the-term-herd-immunity.html

At the end of October, I pronounced the death of the Swedish experiment. I have seen enough. If your soccer (football) team goes down 0-10 in the first 10 minutes of a match, what choice words do you have for the coach who is preaching patience, waiting for the comeback win? By mid-December, Sweden's King admitted that "we have failed."
(9) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/the-swedish-mirage-the-verdict-is-already-written.html

At the time, I assumed 75 percent of the population would need to be vaccinated to attain herd immunity. Many experts placed the number much lower, about 40-60 percent (link). By end of the year, Dr. Fauci is estimating a range of 70-85%.

In one of my very first posts about the coronavirus in early March, I predicted the thorny data quality issues that would bedevil analysts to this day:

We have very limited amounts of data (the known and suspected cases), incomplete data (under-reporting, possible hiding or manipulation), data of dubious quality (self-reporting, hastily assembled tests, unpreparedness), assumptions that later turn out to be false, the potential black-swan fallacy (failure to imagine what hasn't happened before), and non-stationary data (old data might not be wrong but become outdated as the virus evolves).

(10) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/eight-unanswered-questions-about-the-coronavirus.html

In March, I already warned about "the elephant in the room: people are counted as infected only if they are tested", and asked how deaths are confirmed. It would take weeks of regurgitating official counts before the mainstream media acknowledged these shortcomings.
(11) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/eight-unanswered-questions-about-the-coronavirus.html

In April, I explained the benefit of "excess deaths" while also noting why death statistics are subject to their own set of issues. It is here that I said all excess deaths are accelerated deaths. (I will have another new post about this point in early 2021.)
(12) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/excess-deaths-why-we-want-the-cake-and-why-we-cant-eat-it-yet.html

In May and then in December, while studying the data on excess deaths in Florida, I created the following graphic showing the sale.

(13) https://junkcharts.typepad.com/junk_charts/2020/05/how-covid-19-deaths-sneaked-into-floridas-statistics.html
(14) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/its-time-to-visit-florida-again-graphically.html

In May, when Dr. Birx the politician complained that CDC data was terrible, I responded thus:

The members of the Task Force can immediately help make the data better if they cared about the quality of the data. When they aren't pushing diagnostic testing, while trashing the data, their real perceived enemy is not bad data but data.

(15) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/their-real-enemy-is-not-bad-data-but-data-period.html

By June, the media have gone through the five stages of grief:

In the short history of the Covid pandemic, people started with case statistics. Then, they claimed that death statistics are less manipulated than case statistics, when they learned about testing. Then, they claimed that testing statistics are less manipulated, until they realized governments determined who got tested. Some governments counted tests shipped, not test results. Then, they claimed hospitalization statistics are less manipulated, until they learned that hospitals sent sick patients to nursing homes.

(16) https://junkcharts.typepad.com/numbersruleyourworld/2020/06/coronavirus-has-not-gone-away-neither-have-these-covid-fallacies.html

The antibody testing lobby has a particular disregard for data quality. New York's Governor Cuomo, whom I heard won an Emmy for his press conferences, claimed for weeks that 20 percent of New Yorkers had already been infected by April, citing "randomized" antibody testing at supermarkets (during lockdown). To this day, the only "datum" provided by the state's health department which conducted these tests is this FAQ. (Click with no expectation!)

Around the same time, an Oxford research team made headlines, claiming 40 percent of the U.K. population were already infected based on antibody testing. This finding was laughable at the time, and embarrassing given the latest surge in the U.K. In April, I wrote a series of six posts about the Oxford study that explains how scientists build models based on sparse data to make such predictions.

(17) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-1.html
(18) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-2.html
(19) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-3.html
(20) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-4.html
(21) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-5.html
(22) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-6.html

The lead investigator of the Oxford study turns out to be a co-author of the Great Barrington Declaration, which pushes the amoral and dubious theory of herd immunity through natural infections.

Another author of that Declaration is a Stanford professor who is infamous for his own role in the debunked study of antibody testing in Santa Clara, California.
(23) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/the-percent-with-antibodies-could-be-3-percent-or-it-could-be-zero-thats-what-the-stanford-study-rea.html

The first draft of the Stanford study neglected the enormous false-positive problem associated with testing rare events, a topic which I covered in depth in Chapter 4 of Numbers Rule Your World (link).
(24) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/to-justify-zero-false-positive-assumption-stanford-study-needs-a-reference-sample-10-times-bigger.html

The corrected manuscript reveals statistical issues with estimating test accuracy when developing new tests, and failed to quiet the skeptics.
(25) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/the-fda-looks-past-these-statistical-issues-but-maybe-you-shoudnt.html

Most importantly, the antibody testing lobby lost credibility in the face of real-world evidence. They predicted a high level of immunity from natural infections, which would prevent a second wave. Everywhere they made that prediction - Sweden, U.K., California, reality bit back, hard.

Meanwhile, mainstream epidemiology has triumphed. Certain colleges re-opened in the fall, implementing strict test, trace and isolate protocols that were swatted aside by the U.S. government. I wrote about the successes at Cornell and Georgia Tech.
(26) https://junkcharts.typepad.com/numbersruleyourworld/2020/09/good-news-from-cornell.html
(27) https://junkcharts.typepad.com/numbersruleyourworld/2020/08/nice-explainer-from-georgia-tech-about-their-re-opening-model.html

Epidemiologists also have predicted the rise of deaths following rise of cases. This lag is frequently ignored by journalists who are professionally trained to think only in the present, and not one or two weeks ahead, and by politicians who say black is white. In a series of posts, I showed how to see the time delay between cases and deaths, in Lombardia (Italy), Texas, Illinois and Iowa.
(28) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/how-to-act-like-a-data-scientist-9-quantifying-our-intuition.html
(29) https://junkcharts.typepad.com/numbersruleyourworld/2020/06/how-to-act-like-a-data-scientist-12-say-no-to-the-daily-grind.html
(30) https://junkcharts.typepad.com/numbersruleyourworld/2020/07/its-high-time-to-call-out-those-who-got-it-right.html
(31) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-importance-of-matching-data.html

As the year ends, we received good news from several vaccine trials. In July, I complained that Moderna provided almost no useful information in its submission to ClinicalTrials.gov, the supposed center of open science.
(32) https://junkcharts.typepad.com/numbersruleyourworld/2020/07/were-told-almost-nothing-about-the-moderna-vaccine-trial.html

By September, Moderna was pressured to release detailed protocols. So followed other vaccine developers. Once these protocols were released, I computed that the earliest approval date for vaccines would be mid January 2021. At the time, the media reiterated unrealistic timelines suggesting results by Election Day (early November).
(33) https://junkcharts.typepad.com/numbersruleyourworld/2020/09/notes-on-the-moderna-vaccine-trial-protocol-1.html
(34) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/political-headline-decrying-politics.html

If the pharmas adhered to those timelines - the Pfizer CEO was pretending they could, it implied that the trial participants would have been observed for only one month, which would at best prove vaccine efficacy for one month. Such a possibility was squashed when the FDA "strengthened" guidelines around the observation window.
(35) https://junkcharts.typepad.com/numbersruleyourworld/2020/09/behind-the-proposed-strengthening-of-fda-vaccine-approval-rules.html

This guideline was not as strong as it could be. I proposed that all participants - instead of the median participant - be observed for at least two months. A week later, a group of prominent statisticians sent a letter to the FDA, making this same point.

When reviewing the trial results, I noted that neither Moderna nor Pfizer met that FDA guideline for median observation window. This is one of numerous tradeoffs made to rush the approval process.
(36) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-worst-kept-secret-will-be-revealed-today.html
(37) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-second-worst-kept-secret-of-the-pandemic-year-and-some-decimal-mischief.html

The metric of vaccine efficacy is frequently misinterpreted. I wrote about VP Mike Pence's tweet, claiming that 90% VE means the vaccine prevents infections in 90% of volunteers. Not even close. Most of the trial participants have not yet been exposed to the virus so they would not get infected whether they got the vaccine shot or the placebo.
(38) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/cutting-the-infection-rate-by-half-is-not-the-same-as-protecting-half-the-people-from-infection.html
(39) https://junkcharts.typepad.com/numbersruleyourworld/2020/11/watching-statistical-gravity-in-action-elections-and-vaccines.html

These vaccine trial results demonstrate a key tenet of designing experiments: decisions have consequences.

Neither Pfizer nor Moderna has proven vaccine efficacy for two months. Their results are based on less than half the participants reaching the two-month mark. If the FDA rule had been all participants reaching two months of observation, then yes.
(40) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/unexpected-sightings-and-unknowns-from-the-pfizer-result.html

We cannot conclude that one dose gives partial protection for two months because no one tested single doses. Anyone suggesting otherwise makes the assumption that the second dose has no value, creating a tautology, saying the same thing twice.
(41) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/one-dose-pfizer-is-not-happening-and-heres-why.html

Any analysis of subgroups (race, age, severe cases, etc.) should be ignored. The trials were designed to minimize total sample size so as to achieve warp speed; the casualty of this design is subgroup analysis.
(42) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-worst-kept-secret-will-be-revealed-today.html

Throughout this pandemic, the U.S. government, apparently supported by CDC experts, ignores the threat of asymptomatic spread. The policy of triage testing discourages people without symptoms from getting tested. The design of the vaccine trials - with case definitions based on self-reported symptoms - produces no knowledge about asymptomatic transmission. In March, I wrote a column in Wired to call for broad-based testing.
(43) https://www.wired.com/story/the-problem-with-trumps-triage-testing/
(44) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/when-will-we-get-more-and-broader-testing.html

The clinical trials provide higher quality data compared to research studies that use observational data. I outlined the many problems of observational studies that were thrown together in haste, and published as preprints. These include a Yale study of sewage, a Harvard study of parking lot images and search engine traffic, and a Kings College (UK) study of mobile tracking app data.
(45) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/yale-meds-please-meet-yale-stats.html
https://junkcharts.typepad.com/numbersruleyourworld/2020/06/a-glimpse-into-surveillance-data-via-the-harvard-study-of-parking-lots.html
(46) https://www.wired.com/story/beware-the-lofty-promises-of-covid-19-tracker-apps/
(47) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/it-begins-with-the-data.html
(48) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/pre-processing-data-to-avoid-putting-your-foot-in-that-puddle.html

All these examples show that a model that explains the past well does not necessarily predict the future.
(49) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/prediction-versus-explanation-simple-to-state-yet-hard-to-grasp.html

Regarding the Covid-19 symptom tracking app, I explained how the analytical methodology for estimating population prevalence can justify any level of prevalence. That was back in April. We have barely heard a pip about Covid apps since then.
(50) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/the-first-major-study-using-covid-tracking-app-data-stumbles-out-of-the-gate.html

Between blogging, I was a guest on the CSAI podcast with John Reid and Rusen Aktas:
(51) https://anchor.fm/csai-podcast/episodes/CSAI-2--Covid-19-and-Data-Science-ege3si

More recently, on Ryan Ray's podcast, I talked about how I wrote my books:
(52) https://junkcharts.typepad.com/junk_charts/2020/11/podcast-highlights.html

On the sister blog (Junk Charts), I featured several nice data visualizations. SCMP published this beautiful picture of contact tracing efforts in Hong Kong:
(53) https://junkcharts.typepad.com/junk_charts/2020/09/unlocking-the-secrets-of-a-marvellous-data-visualization.html

The New York Times visualized excess deaths in the U.S.:
(54) https://junkcharts.typepad.com/junk_charts/2020/08/deaths-as-percent-neither-of-cases-nor-of-population-deaths-as-percent-of-normal.html

The New York Times traced the exodus of the rich from NYC:
(55) https://junkcharts.typepad.com/junk_charts/2020/06/designs-of-two-variables-map-dot-plot-line-chart-table.html

The Visual Capitalist depicted how we changed our consumption habits:
(56) https://junkcharts.typepad.com/junk_charts/2020/05/consumption-patterns-during-the-pandemic.html

***

I wrote 212 posts across the two blogs in 2020. Since I started, people have read posts more than 4 million times. Thank you for reading, come back often, dig into the archives, and recommend me to your friends and colleagues. See you in 2021!

Kaiser

 

[1-1-2020: Added an index to the links, and corrected a link error.]

Numbers indeed rule our world, 10 years out

Statistics is the subject of generalizing data. In statistics, we study how to use imperfect data to draw general conclusions. We often have to wait for the future to reveal the evidence of success.

Ten years ago, McGraw-Hill published my first book, a best-seller, Numbers Rule Your World (link). I’ve hoped to explain five of the most profound concepts from my field in a way that anyone can understand and apply to their daily lives. I attempted the balancing act of featuring specific stories while bringing out the general principles. In the last couple of months, the Covid-19 pandemic has generated immense attention to data and statistics, and provided me confirmation, ten years on, that the contents of Numbers Rule Your World (link) have general applicability. It affirms that statistics is a great subject!

If you’re approaching my book for the first time, or re-reading it, here’s how you can work your way through it, making connections at various points to the Covid-19 crisis.

***

Chapter 2: Bad Spinach / Bad Scores

This is your perfect starting point. Half of the chapter details a disease outbreak investigation. You learn about the information networks the CDC puts in place to monitor unusual patterns of disease around the country. You discover data analysts whose job is to set off the alarm. How do they decide if the first handful of cases foretell an epidemic?

You encounter “shoe leather,” the painstaking collection of data to pinpoint the origin of an outbreak. It’s much harder than you might think. The U.S. politicians are making much noise about the early days of Wuhan. An epidemiologist may learn that 10 out of 15 first patients all visited the Wuhan seafood market before they got sick. But the Wuhan market might be a hugely popular destination, like Central Park in New York City. Finding 10 out of 15 people went to Central Park over a weekend could be more common than you think. If all 10 touched a particular rare species of wildlife, that's another story.

Another controversy of the moment is the huge cost of shutting down the economy, against the benefit of saving lives. I invite readers to think about this cost-benefit tradeoff. Because the disease outbreak was traced to a spinach processing plant in California, the FDA ordered a recall of all spinach, which depressed sales of salad greens for six months or so. The cost was much lower than for this pandemic, but the bacteria is well-known and thus predictable, and the lives saved were most likely zero, and even harder to pin down than in the current crisis.

People are already complaining that social distancing and other measures have not improved our numbers, by saying deaths were “unexpectedly low”. You discover how hard it is to prove the effectiveness of a policy decision because you need the counterfactual – what would have happened if the FDA did not recall the spinach.

Chapter 4: Timid Testers / Magic Lassoes

Everything you should know about the statistics of diagnostic testing and predictive models in one place. The motivating examples are (a) the fight against performance enhancing drugs in sports via anti-doping tests and (b) the false promise of terrorism prediction algorithms.

I wrote several posts (1, 2, 3) about the flawed Stanford antibody study that reported 50 times more infections in Santa Clara county in California than reported cases. Despite this headline, the raw data is also consistent with a scenario of zero proportion having antibodies. The statistical reason is the low underlying prevalence of antibodies, which caps the number of true positives in any sample. The number of false positives will be large relative to true positives, because the false positive rate applies to those without antibodies, which is the bulk of the population.

An extreme case of this phenomenon is terrorist prediction using a dragnet surveillance dataset. The set of terrorists is a drop in the ocean of people under mass surveillance. For a predictive model to capture most of these terrorists (low false negative rate), it must flag a lot of people – even a tiny false positive rate when applied to a huge population results in a lot of false accusations.

You see the distinction between a “chemical” false negative (positive) and a “practical” false negative (positive). To evade steroids testing, dopers use all kinds of tactics, such as handing in someone else’s urine. The diagnostic test finds a true negative but in practical terms, it’s a false negative. For swab testing, one may test negative if the swab doesn’t reach a virus-rich area, or if the patient is tested at the “wrong” time.

All tests have errors. What are the consequences of errors? Also, how likely will the error be discovered? For anti-doping, do dopers inform the test lab they made a mistake?

Chapter 1: Fast Passes / Slow Merges

The first chapter gives important insights on managing capacity, which is a pivotal issue for hospitals during the coronavirus crisis. I cover two case studies: (a) Disney’s management of theme-park capacity, and (b) state’s management of highway congestion.

The strategy of building more capacity is unavailable during a crisis, and wasteful in general. In this extraordinary time, some localities were able to bring up temporary hospitals. Transportation experts discovered that policies of regulating congestion are more effective. Highway ramps slow down the influx of new vehicles onto the highway, and are turned on before congestion arrives. The key is to delay as much as possible the onset of congestion. Sounds familiar? It’s the analogue to “flattening the curve”. Social distancing is supposed to reduce contacts, slowing down community spread. This tactic delays the peak demand for hospital resources. The chapter explains the scientific support for why such tactics work.

Chapter 5: Jet Crashes / Jackpots

Statistical significance is the topic of this chapter, one of the most common and also most commonly mis-used concepts in statistics.

In the Stanford antibody study referenced above, statisticians argued that the observed level of positive results is consistent with zero prevalence. This is a way of saying the result is not statistically significant. What we mean is that the study’s result (50 positives out of over 3,000 tests) would be considered normal even if nobody in the population actually had antibodies. This conclusion does not assert that the true prevalence is zero. The topic of statistical significance is made inaccessible to non-mathematicians by textbook authors piling on equations, but it need not be so. I even made a video on explaining "not statisitical significant" (link).

Before you apply a test of statistical significance, you must make sure you’re comparing apples to apples. In the context of air safety, you shouldn’t compare U.S. and foreign airlines on all routes; you should only compare them on routes they share, in which case, you’ll find no statistical difference. One of the bad stats that is circulating during this pandemic is the purported harm of ventilators. This is usually stated as the vast majority of patients put on ventilators eventually die (presumably, relative to those patients not put on ventilators). But only the most serious patients are placed on ventilators. The proper analysis should focus on patients who qualify for ventilators, and compare those who were put on ventilators to those who got other treatments.

We have already seen a host of invalid comparisons. Comparing complete data from years ago with incomplete data this year. Comparing regions that are in different phases of the epidemiological timeline. Comparing countries that have taken different approaches to managing the crisis.

Chapter 3: Item Bank / Risk Pool

This chapter addresses a deeper issue in using statistical averages. An illustrative example is educational testing like the SAT. Critics pointed to the gap in the average scores across racial groups as evidence of bias. But the racial grouping masks a third factor, which is differential ability between students. Due to various socio-economic factors, white students have higher average ability than black students. So the real question is: Among those students with comparable abilities, does the test still show bias toward one racial group? (The answer is not straightforward.)

During the pandemic, the “denier” faction has argued that the mortality rate of Covid-19 is “on a par with” that of seasonal influenza. Are those two rates comparable? The simple answer is no. Comparing mortality rates masks a third factor, which is the level of immunity in the population. Even at the most optimistic, the current level of immunity to the novel coronavirus is less than 20 percent (This statement generously presumes that (a) having antibodies confer immunity without question and (b) the New York City test result is not a scam, and the test is 100% accurate). By contrast, because of vaccination and recurrence, a good chunk of people have immunity to influenza. The mortality rate of Covid-19 should be compared to that of influenza among a population that has no immunity to either, or a population with similar levels of immunity to each.

The other example in this chapter explains how the insurance industry is an outgrowth of the law of large numbers. In any given year, health or auto insurers cannot predict with high accuracy which covered individuals will file claims, but the magic of statistics allows them to predict with great accuracy the aggregate claims. Disaster insurers run into trouble when rare events happen, such as when Class 5 hurricanes hit Florida. Suddenly, potentially all covered individuals tap into the insurance pool all at once. Insurers try to pool together risks of different types and different areas. The global pandemic is a huge threat to the insurance industry. Perhaps there are lawyers reading this, and they might tell me there are natural disaster exclusions. If those don’t apply, we will see a gigantic number of covered entities filing claims all at once.

Florida’s hurricane insurance market was dysfunctional also because the risk is predictable along some parameters. Those living inland realized they were subsidizing the coastal dwellers and wanted out. The coronavirus also creates different risk subgroups. The younger generations have much lower risks than the elderly. When social-distancing and other practices are applied to the whole community, the subgroup with lower risk peels away. A key difference is community transmission – the risk of hurricane damages does not spread by contact.

***

The above covers less than half of the book. Almost everything in those pages has direct relevance to the Covid-19 world. It’s quite heartening to find out ten years later numbers indeed rule our world.

Here’s a link to get your copy of Numbers Rule Your World. There are also translated copies in Chinese (simplified and traditional), Japanese, Korean, and Portuguese, as far as I know.

Why tests are not perfect

From Andrew's blog, I learned about Tom Lumley's post that provides insight into the accuracy of the two types of Covid-19 tests. Tom is a biostatistician.

As you probably know by now, there is a diagnostic test, the most prominent of which is called the (RT-)PCR test, that involves using a swab to get sputum from the nose or throat. This material is taken to a lab which looks for the presence of SARS-CoV-2, the name for the virus that causes Covid-19 the disease. This test identifies who currently are infected.

Then, there is an antibody test, which looks for antibodies that are found in the blood of infected people, as the body fights against the virus. This test does not detect the presence of the virus; it looks for a sign that it passed through.

These two tests are not interchangeable. Some politicians on both sides of the Atlantic try to cover up the slow rollout of diagnostic testing by talking up antibody testing. In fact, early reports from antibody testing program such as the Stanford study found less than 5 percent have antibodies which means the virus is not as prevalent as claimed. (Here's why many statisticians are dismissing the Stanford study.)

It might seem like a simple issue: is the virus/antibody there or not? In both cases, if someone were not infected, their sputum or blood shouldn't have that specific virus or antibody in it. Why can't tests be perfect?

***

Let's first consider the antibody test. Antibodies are generated by our bodies naturally to fight an antigen. According to Tom, the antibody to SARS-CoV-2 is not unique - it varies from person to person slightly. This means that any test needs to look for a "range" of antibodies. To complicate matters, antibodies to similar viruses are also similar. Thus, the test will not find the exact target antibody but something similar. Then, the test has to decide whether the difference is due to different individuals or different (but similar) antigens.

One should hope that the person-to-person variability is smaller than the antigen-to-antigen variability. A test would then set up a "range" of outcomes that qualify a positive finding. If the measurement is near the edge of this range, there is a potential for error.

If the acceptance range is too "tight", the test might mistakenly decide it found antibodies for a different antigen when it's just a different individual. This produces false negative results. If acceptance is too "lax", it might mistakenly conclude that an antibody for a similar infection is an antibody for SAR-CoV-2 for a different individual. This produces false positive results.

There is a trade-off between these two errors, controlled by how the test developer sets the acceptance "range". For the antibody test, doctors don't like false positives. These are people who don't have antibodies but are told they do. As I discussed the other day, because the vast majority of people don't have antibodies, even a tiny false-positive rate will result in a lot of people being told they have antibodies when they don't.

The ability to correctly identify people without antibodies is called specificity. Some researchers are assuming antibody tests could be 100% specific. I think that is overly optimistic.

***

The PCR diagnostic test illustrates another aspect of testing accuracy. Sometimes, the error arises externally. In Chapter 4 of Numbers Rule Your World (link), I demonstrate this using steroids tests. A false-negative result arises when the lab mistakenly concludes the amount of chemicals in the doper's urine is not high enough to warrant a red flag. That's a "chemical" false negative.

But a false-negative result is really just any doper passing an anti-doping test. Someone might submit a fake urine sample, or hide from inspectors. In one case, the fake sample will not test positive, and in the other, there will not be a test. Both should be considered false negative cases even though the testing lab did not make a mistake.

Tom says "it is almost impossible to get a positive [PCR test] result without any virus". But low false-positives usually mean higher false-negative results. One reason for false negative is testing at the wrong time, or failing to swab properly. Bad timing is particularly important because "if you get tested too early, there might not be enough virus, and if you get tested too late the infection might have relocated to your chest."

Timing is also important for antibody testing because "a positive reaction [from one's body to the virus] takes time — at least a few days, maybe a week — and it stays around at least a short time after you recover."

Timing is an external factor. The consequent false negative error has nothing to do with the testing lab. Even if the test itself is perfect, there will still be people walking around with the false security of a negative test result.

So errors from testing occur because the test itself cannot be perfect (e.g. antibodies against similar infections may look like antibodies against the same virus coming from different individuals), and even when the test is close to perfect, other factors outside the lab, such as bad timing, cause errors.

***

One final comment on the politics. At various times, some officials argue that the rollout of diagnostic testing has been slow because tests are not accurate enough. Don't believe a word of that.

Let's say 5 percent of the population are infected. Borrowing some numbers from Tom, I assume that the diagnostic test has 35% false negative rate and 2% false positive rate. For every 1,000 (random) people given the test, there are 50 infected and 950 not infected. Of the 50 infected, 65% or 33 will get a positive finding and 17 will falsely believe they don't have the virus. Of the 950 not infected, 930 are told they don't have the virus correctly, with 20 told they have it when they don't. In total, 930+33 = 963 out of 1,000 will get the right answer.

So what the politicians are really saying when they reject this test is that having 17 out of 1,000 people falsely believing they don't have the virus thus spreading it is reason enough to deny 963 out of 1,000 knowing the correct answer. For every false positive, the test identifies two true positives, who can be isolated. It doesn't stem the spread completely but it does slow it down. Given that these politicians are all in for "flattening the curve", they should be clamouring for more testing.

 

 

 

 

 

 

The percent with antibodies could be 3 percent or it could be zero. That's what the Stanford study really said.

As predicted, test accuracy is getting major play in the media this week. One of the big headlines involved the claim that about 3 percent of the U.S. population may already have been infected and recovered, according to a study coming out of Stanford. The good thing about this study is its attempt to escape from the straitjacket of "triage" or targeted testing. It recruited about 3,330 people without requiring that they have symptoms. They were tested for antibodies, which is an indication that the person has recovered from a previous SARS-CoV-19 infection. Showing that a large proportion of the population have been infected and recovered is key to those pursuing the "herd immunity" theory.

The bad news is that the methodology of the study has already been debunked by Andrew Gelman. Andrew's post is here in its entirety. In this post, I give you an executive summary, especially for those not trained in the statistics of testing.

***

Before I get to the numbers, the idea that this study supports the "herd immunity" theory is far fetched. Herd immunity models typically show the vast majority of the population would be infected, much higher than five percent. We would need about 20 times the number in Santa Clara to get to herd immunity. Also, the presence of antibodies is not identical to having immunity. It is still not clear whether humans have immunity after recovery from Covid-19, and if so, for how long. The headline drawing attention to the framing as 50 to 85 times higher than reported is burying the fact that they found antibodies in only 3 percent of the samples.

If the result of this study is accepted, it exposes the sham of diagnostic testing in Santa Clara.

***

There are two major statistical issues with the Stanford study. Both are related to the construction of the sample - its composition and its size.

1. The way volunteers are recruited introduces bias into the statistics.

The 3 percent headline result is statistically adjusted. The 3,330 tests actually returned only 50 positives (1.5 percent). The adjustments were necessary because the researchers discovered biases in their sample by comparing basic demographic statistics to the county averages.

Bias means certain subgroups are under-represented in the sample. If a certain subgroup (say, men) are over-represented, then the composite subgroup (women) is under-represented so it's really two sides of the same coin.

While the sample size of 3,330 for one county is pretty large, the selection of volunteers was decidedly not random. People were recruited through Facebook ads, and they must drive themselves to the testing centers. 

A non-random selection has the chance of introducing unknown biases. In this case, some possible sources of biases are:

• people who have had difficulty getting tested (e.g. health workers) may show up in larger numbers
• some people who strictly adhere to the shelter-in-place order may not show up
• some people may like getting tested for free
• some people may believe Covid-19 is a hoax, or testing is pointless
• some people may find it more convenient to get to one of those test centers
• some people do not have Facebook
• some people do not respond to Facebook ads
• people who already have been tested, especially those who tested positive and recovered

These biases draw more or fewer infected people to the testing sites, and the strength of one versus another also matters. There is one bias that is irrefutable. Anyone who has severe symptoms or are hospitalized will not be represented at all.

It seems hard if impossible to measure and then correct for these biases, which is why statisticians usually recommend random selection. An ounce of prevention is worth a pound of cure. (A proverb that is most appropriate for this pandemic!)

It is sometimes mistakenly believed that larger samples cure sample bias. It doesn't. No matter how big the sample is, they will not find hospitalized patients in their sample. But those are almost surely infected.

***

2. The proportion of antibody-positive reported in the study is so small it may be all noise.

It turns out that even with 3,330 tests, a pretty large sample, the key result that 3 percent have antibodies could all be noise.

What is statistical noise? Imagine you repeat this experiment many times. For each set of 3,330 volunteers, you record the number of positive results. You aren't going to get 3 percent each time. The proportion with antibodies fluctuate with each sample but these variations are meaningless, an artifact caused by sampling.

The following analysis follows the plan used in Chapter 4 of Numbers Rule Your World (link) to analyze steroids testing and terrorist prediction systems. So if you have the book, you can compare the current situation with those stories.

In the raw data, the researchers found 50 positives out of 3,330 tests. (As mentioned in the first section, this 1.5 percent was revised up to 3 percent due to statistical adjustments which I won't describe here. If interested, Andrew has comments on those.)

Every test has errors. Some of the 50 positives are false positives. The false positive rate of a test describes the chance that the test will say someone has antibodies when they don't. The FP rate is one measure - but not the only one - of test accuracy.

To illustrate the issue, we consider the implausible, extreme scenario where none of the 3,330 volunteers have antibodies. So in a perfect test, the number of positives should be zero. If we get 50 positives, all 50 are false positives since no one has antibodies. That implies a false positive rate of 50/3330 = 1.5 percent.

Next, we assume that the researchers are right, that indeed 3 percent of the population has antibodies. (We further assume that the sampling is not biased.) That means the sample would be split 3% and 97%, that is, 100 people with antibodies and 3,230 without.

Let's proceed. We think that the test has a false positive rate of 1.5 percent. Applied to the 3,230 test-takers without antibodies, this antibody test should report 48 (false) positives. Note that in the actual experiment, the researchers counted 50 positives, so almost all of these are false positives!

Here is where things get confusing. Even though the test is rated as very accurate, 1.5 percent false positive rate, 98.5 percent true negative rate (also known as specificity), almost all the positives are false positives! The reason is the rarity of antibodies. The 1.5 percent false positive rate is applied on almost the entire test population because very few has antibodies. A small percentage of a large base is still a large number. (Another tie-in to the pandemic: a low fatality rate applied to a large infected base through fast community spread leads to a flood of deaths.)

***

When developing a new test, the false positive rate is not known because if we knew who has antibodies and who doesn't, we don't need to test people. We test because we don't know. In the above calculation, I used a thought experiment, the extreme case, to guess at the false positive rate.

The Stanford study had a better way of estimating it. They applied the antibody test to some blood samples collected before the Covid-19 pandemic when they're sure these should test negative if the test is accurate. Using that data, Andrew pegged the false positive rate at 0 to 2 percent.

At one end, the false positive rate may be 2 percent (instead of the 1.5 percent I used before). In that case, the test-takers without antibodies generate 65 false positives. This number is higher than the total number of positives (50) found in the experiment. So, test inaccuracy might account for all positive results!

At the other end, if the test was so accurate as to have zero false positives, then by definition, all 50 positives would be true positives, and we could run with their result. Anyone who knows a little about testing recognizes this scenario as implausible. It also implicates a high false negative rate!

Remember if the population were indeed split 3% with antibodies, 97% without as per the study's conclusion, there are 100 people with antibodies in the test sample. The test only returned 50 positives so the other 50 are false negatives.

This is the crux of understanding statistics of testing. You can't focus on just one accuracy metric. When one number goes up, another goes down. Here's the bottom line: if we think there are 100 people with antibodies in the test population of 3,330, then any reasonable test must return at least 100 positive results if it had any hope of catching all those with antibodies. That implies the test must have a positive rate (true or false positives included) of at least 100/3330 = 3 percent. To account for false positives, the test must return even more positive results. If it doesn't, then there must be false negatives.

False negatives are considered less damaging than false positives in the context of antibody testing because these people with antibodies would be taking unnecessary caution. It's not completely harmless since they might suffer from lost income. False positives are feared as they might have a false sense of security and take unknowing risks, and get infected.

Let me now relate this back to sample design. The above analysis shows that the observed test result is consistent with 0 percent of the population having antibodies. This is different from saying we believe the true proportion is 0 percent. It could be 3 percent as the researchers asserted. But the evidence is not strong enough to rule out 0 percent. In other words, even though 3,330 is a lot for one county, the sample is still not large enough to measure such a small signal.

***

I hope you're excited to learn more about testing.

See my previous post, or read Chapter 4 of Numbers Rule Your World (link).

 

 

 

 

 

 

 

 

Enter at your own peril: the statistics of diagnostic testing

 

In the days ahead, you're going to start hearing a lot of commentators try to say something smart about testing accuracy - and fail. This is a very confusing area, with multiple redundant sets of terminology used by practitioners from different backgrounds and no one willing to standardize to the same jargon. It is also an area where the drive for simplicity has doomed many explainers.

Take this explainer published by the New York Times on coronavirus diagnostic testing, written by a Yale doctor. (Tip from Antonio R.) The author made the following statement:

The good news is that the tests appear to be highly specific: If your test comes back positive, it is almost certain you have the infection.

The bad news is "specificity" and PPV are different concepts.

I am working on a bunch of posts at the moment so I will point you to my 2010 post that covers this subject for now. The two key numbers you should look for are PPV and NPV.

Another key takeaway is that the accuracy of a test cannot be reduced to a single number. To understand this material, you must be capable of holding two or three numbers in your head at the same time. I hope that is not too much to ask!

It's actually easy to mix up the terminology, even when you know your stuff. So I wouldn't read too much into the published mistake.

If you read Numbers Rule Your World (link: Chapter 4 covers steroids testing and terrorist prediction) or Numbersense (link: the Marketing chapters cover the accuarcy of targeting models), this should just be a review.

***

The insufficient accuracy of testing has become a popular excuse by governments that have continued to pursue a test-less policy. There is no perfect test. All tests face a trade-off between two types of errors. Reducing one increases the other. A serious critique of testing accuracy needs to involve actual numbers and target levels. What level of accuracy do these officials consider as adequate? And what are those numbers for the tests that they have rejected?

That's all I'll say on this issue for now, and I'll return to this subject soon. For now, here is the post that will give you everything you need to know.

 

Eight unanswered questions about the coronavirus

Coronaphobia has arrived in the U.S. This past weekend, people started raiding stores for masks and tissue paper.

Ultimately, the one important question we're trying to answer is:

Is COVID-19

(a) just another flu virus

(b) a particularly lethal flu virus that will have a short-term impact (similar to SARS, which  killed about 800 people but did not last beyond one flu season)

(c) a deadly virus that would fuel a much-feared flu pandemic that could kill millions

This is the crucial question because its answer should condition our response to the virus.

(a) typical caution

(b) short-term vigilance

(c) large-scale mobilization and preparation.

 

Small Data Emergency

Because this coronavirus is new, and the disease is spreading while humans are playing catch-up, what we face is a classic "small data" emergency. We have very limited amounts of data (the known and suspected cases), incomplete data (under-reporting, possible hiding or manipulation), data of dubious quality (self-reporting, hastily assembled tests, unpreparedness), assumptions that later turn out to be false, the potential black-swan fallacy (failure to imagine what hasn't happened before), and non-stationary data (old data might not be wrong but become outdated as the virus evolves).

This emergency is typical of any disease outbreaks. This one is larger in scale, and international in scope. To understand the entire process of outbreak investigations, and the pressure investigators work under, I refer you to Chapter 2 of Numbers Rule Your World (link).

Before we complain about incompetence, confusion or muddled messages, we should have some sympathy for the epidemiologists, medical researchers, data analysts, policy analysts, public health advocates, and others who are doing their very best at a very challenging moment.

 

The Overdue Pandemic Backdrop

"Politicians can ignore it, and may even get away with it. But when a pandemic eventually strikes, and history suggests that one is long overdue, the consequences could be devastating." This is a quote from an article published by The Telegraph. In 2008, more than a decade ago.

Much of the medical community have been predicting a pandemic for years and years. They thought SARS was the big one but it wasn't - it did not pass from human to human quickly enough. According to the same article, WHO warned that the H5N1 avian flu could "seed a human pandemic that ... hospitalise 30 million, of whom some 20 per cent would die." That projection was way off the mark.

For a pandemic to occur, we need a mix of infection rate and fatality rate. Coronavirus has a much higher infection rate than SARS, and transmits between humans but it is less lethal. However, at this early stage, our estimates of the infections and fatalities are far from accurate.

I've put together a list of unanswered questions. Whether COVID-19 is another false alarm or not depends on what answers we eventually receive.

 

1) Test accuracy affects the infection count

All tests currently in use are rush jobs which obtained "fast-track" approval. China accepted seven tests within the first two weeks of the outbreaks. These tests use the "nucleic acid" method, which amplifies genetic materials collected from throat swabs, and according to SCMP (the paper of record in Hong Kong), an official later said they were only "30 to 50 percent accurate".

The low accuracy rate should not surprise. Fast does not produce accuracy; in fact, fast probably means less reliable. Our knowledge of the new virus is very limited in these early days.

It's not even easy to tell when a test is inaccurate. Many anecdotes circulate of patients who initially tested negative. The media immediately screamed "false negative"! But it's not that simple. The patient could have caught the virus between tests, especially if s/he is quarantined or living in proximity of someone with the virus. Or the virus may be mutating.

Scientists are negotiating a vicious cycle of inaccurate tests, and inaccurate measurements. A BBC report describes these challenges.

To learn more about diagnostic testing, see Chapter 4 of Numbers Rule Your World (link), in which I look at how the sports world tests for performance-enhancing drugs. The trade-off between false positives and false negatives is a key issue. I also make the distinction between a "chemical" negative and a "real life" negative. An athlete might test negative in a doping test but could still be a doper because the test does not detect the substance s/he is using, or is tricked by a masking agent, or is ambushed by an evasive action. So, it's possible to have a test that is accurate for what it's designed to do but still misses what we'd like to know in real life.

 

2) The elephant in the room: people are counted as infected only if they are tested

The big catch: someone can be counted as infected only if the person is tested for coronavirus. No test, no reported infection. This ABC News report states that South Korea has tested tens of thousands of suspected cases while the U.S. has conducted fewer than 500 tests, as of February 27, 2020. This means the number of infections in South Korea is probably over-stated (see point #1), and the number of infections in the U.S. is under-stated.

If we believe we face scenario (a), then not testing makes sense. It's simply not useful to know how many have the common flu viruses at any given time. But if we are in scenario (c), then stopping the spread of the coronavirus is paramount, which requires knowing who has it.

 

3) Noise coming from common flu viruses

Remember we are in flu season. According to the CDC, influenza and pneumonia causes 56,000 fatalities in the U.S. each year, the eighth leading cause of death. The fatality rate is 17 per 100,000 or 0.02%. This implies an annual infection count of 327 million cases of the common flu (despite the widespread use of the flu vaccine). On average, that's almost 2 million infections a day if we assume the flu season lasts 6 months (the bulk of infections actually occurs between December and March).

Against this backdrop, it is easy to attribute infections or deaths wrongly to COVID-19. The use of indirect tests increases the chance of mis-attribution.

For example, CAT scans find evidence of lung damage but using such tests can make infection counts inaccurate in two ways: lung damage can be caused by other viruses especially in flu season, and not all people with COVID-19 suffer lung damage.

Partly as a reaction to the low test accuracy and/or desire for speed, the Chinese authorities switched the definition of an infection, now counting people with symptoms without waiting for a test confirmation, according to this BBC report. This change in definition led to a spike in "infections" but the new count probably contains more false positives while having fewer false negatives.

Inflating the infection count can be justified by the desire to minimize false negatives, which reduces the chance of people unknowingly spreading the virus around. Overcounting infections, however, leads to an under-estimate of the fatality rate.

Similarly, how are deaths confirmed? Are some of these deaths caused by other flu strains?

 

4) The connection between infections and deaths

The following news item from Twitter is typical of the media's set narrative on the coronavirus:

The two statistics they focus on are the number of infections (cases) and the number of deaths.

Those two numbers are linked. If the number of infections are growing fast, and deaths are not increasing proportionately, the fatality rate is coming down, so that should be a reassuring sign. The fact that both infections and deaths are growing doesn't mean anything. It's the growth of one number with respect to the other.

 

5) The survival rate

One thing that the media refuse to cover is the survival rate. COVID-19 does not cause certain death, far from it. People have recovered. Because it may take a couple of weeks or longer to be cleared, the count of survivors lags. Our estimate of the survival rate improves over time as more people exit the treatment period.

 

6) Intermediate metrics

Another key metric to track is the severity of the symptoms. Among the group of patients who are still surviving, some are mildly sick while others are seriously ill. The media also fail to report on any intermediate metrics.

 

7) Rates not counts

In all of the above, I talk about rates not counts.

Counts are misleading. If we take the example of the Twitter news item and apply its logic to the common flu viruses, then we'd have to report that during each day of the flu season, there are two million new infections, and 300 deaths! Sounds scary but not really.

[See comment below. Counts are relevant for logistical use. The number of patients requiring hospital stays certainly can overwhelm available resources.]

 

8) Community transmission

SARS has a high fatality rate, close to 10 percent of those infected died. However, the virus never quite mastered human-to-human transmission and because the infection rate was low, it eventually failed to become a pandemic.

Current estimates of the fatality rate of COVID-19 is under 2 percent on the low side and maybe 4 percent on the high side. This fatality rate is much lower than SARS but the infection rate is clearly higher, and it's clear that the virus can spread among humans.

 

***

It all comes down to infection rate and fatality rate. The trouble is we have "small data" so our estimates of those rates are limited. Knowing the sources of inaccuracy is helpful to understanding which direction the rates are moving. Also, secondary metrics like severity and survival rates provide some complementary information while we wait for more data to arrive.

Don't panic, but don't take unnecessary risks.

 

P.S. [3/9/2020] Some additional comments given new developments since this post was written

South Korea and Italy are both spoken of as hot zones for COVID19. This is true in terms of known infections but remember point #2, the number of reported infections is highly correlated with the number of tests performed. As of yesterday, New York City said it did less than 100 tests. In South Korea and Italy, they did tens of thousands of tests.

It also appears that the hospitalization and death rates in different countries are different. So there are even more variables, due to factors such as availability of medical facilities, quality of care, age distribution of patients, and so on.

Financial and statistical incentives to over-diagnose and over-treat

Nice article in the New York Times about the "overdiagnosis" problem in cancer screening. The particular case is thyroid cancer in South Korea.

There are a number of things about any form of screening tests that one should always bear in mind:

• Death rate is measured as the number of deaths divided by the number of people with the disease. The latter number increases with better diagnosis techniques.
• Better diagnosis techniques for cancers inevitably identify tiny tumors, almost all of which will never develop into cancer during anyone's lifetime. Peggy Orenstein had a fabulous article a year ago about the same issue in breast cancer. In that case, those tiny tumors weren't even called cancer until the diagnosis movement sprang alive.
• Once a tumor is labeled cancerous, patients will opt to fix it. Because these tumors would never have killed the patients, they inflate the number of diagnosed cases without increasing the number of deaths. So just by virtue of increased diagnoses, the death rate is brought down.
• The immediate outcome of a nationwide screening program is to dramatically increase cases. The article is a little unclear about whether it is the number or the rate of death that did not fall. It doesn't really matter; either case leads us to conclude that the screening has failed to improve health.
• One must not forget that screening tests, subsequent confirmatory tests, treatments, etc. all cost money, so there is a financial incentive to over-diagnose and over-treat.
• In addition to the financial incentive, there is the issue that I raised in Chapter 4 of Numbers Rule Your World (link). A false negative is a very public error on the part of the medical establishment while a false positive (followed by say removal of the thyroid) is an unobservable error. So there is a statistical incentive to over-diagnose and over-treat.

 

Round-up of coverage of the Big Miss of Big Data

There is now some serious soul-searching in the mainstream media about their (previously) breath-taking coverage of the Big Data revolution. I am collecting some useful links here for those interested in learning more.

Here's my Harvard Business Review article in which I discussed the Science paper disclosing that Google Flu Trends, that key exhibit of the Big Data lobby, has systematically over-estimated flu activity for 100 out of the last 108 weeks. I also wrote about the OCCAM framework, which I find useful to think about the "Big Data" datasets we analyze today versus more traditional datasets from the past.

The HBR article caught the attention of various outlets, including New York Times and ABC News.

Slate was probably the earliest to react, and noticed a post on this blog that was the precursor to the HBR article.

Readers who are specifically interested in GFT should read the source materials themselves, which are quite accessible. Start with the Science paper. After that, you can read the original research article by the Google team, hosted at google.org (click on the PDF link in the blue box at the bottom of the page). There are some bold claims in this paper, as well as caveats. They seemed to be concerned about "false alerts" at the time, such as news events rather than illness that drive certain searches. (For those statistically inclined, the underlying model involves only 1,152 points of data--128 weekly aggregates in nine regions--but a search through 450 million simple logistic models to not only define which search terms are important but also determine how many search terms to include in the final regression.)

Then read the article by Cook, et. al., which covers an update to the model made after the 2009 season when GFT totally missed the pH1N1 swine flu epidemic. Notice that this Big Miss is the opposite error to the "false alert" problem. (See Chapter 4 of Numbers Rule Your World for a thorough discussion of different types of prediction errors, and how to think about them.) From the charts in the Cook article, you can see that in the runup to the Big Miss, GFT systematically under-estimated flu activities for as many weeks as you can count.

The overhaul was drastic. The search term topics that accounted for 70% of the original model were reduced in importance to 6% while two other topics that counted for 8% originally were inflated to 69% in the updated model. This dramatically improved the "fit" statistic (RMSE) for the "first phase" of the Big Miss from 0.008 to 0.001.

Next, there is Butler's article for Nature, (Feb 2013) which precedes the Science article, but first pointed out the over-estimation problem for the 2012 flu season. One possibility is that the model update described above over-compensated for the Big Miss, making it more susceptible to the False Alert.

Other media coverage of Google Flu Trends include Guardian (which focuses on the need to understand causality), CRN, and the Economist (which talks mostly about Twitter data which is much more problematic than Google search data).

***

Tim Harford has probably the most educational of the pieces revisiting Big Data for the Financial Times. (When I wrote this line, the FT link wasn't working. The title of the article is "Big Data: Are we making a big mistake?" if you need to find a different link.) His is the longest and covers a lot of ground, and has great examples, including one of my own. Highly recommended.

***

One of the slogans of the Big Data industry (of which I'm a part) is the push toward "evidence-based" decision-making in place of "gut feelings" or "instincts". Until now, I'm afraid there has been plenty little "evidence" presented to support the assertions of universal, revolutionary goodness of Big Data (try searching for quantitative assessment of Big Data projects). I hope we are witnessing the birth of evidence-based decision-making inside the industry of dishing out evidence-based advice.

The reason I put out the OCCAM framework is to steer our community toward a more constructive approach to tackling "Big Data" problems. It requires a fundamental shift in how we define the problem. I have a moderately more technical take on some of the statistical challenges in an article published in Significance, earlier this year. This article discusses six technical challenges where we need substantial progress.

Statisticians sometimes dismiss these as "old news," claiming that the same problems exist in smaller datasets and the problems are well known. A recent example is Jon Schwabish's tweet saying that this discussion induces a "yawn". This reaction feels a bit like Fermat writing in the margin claiming he has a proof. The rest of the world don't want to wait 358 years to figure out what the goods are which these statisticians are hiding.

In my view, there has been some interesting work but nothing to settle debates. If we have great solutions, we won't be discussing these same problems today.

***

Back to flu prediction. It's really something that is well worth pursuing!

It's a nice-defined, self-contained problem that has social benefits and whose results can be easily measured. We should be grateful that the Googlers spent time working on it. It's a problem I'd love to work on if I have time and resources on my hands.

The researchers also have pioneered this type of research using search term data. This is highly significant, and the data represent a perfect example of what I call OCCAM data: the data is purely Observational (related to what Harford calls "found data" or what Dan Eckles calls "data exhaust"); it has no Controls; it is seemingly Complete; it wasn't collected for the purpose of predicting flu trends, that is, it is Adapted from other uses; and the search data was Merged with the CDC data (the matching of states and regions, and of weeks were not exact as you can tell from the original research article).

The several published versions of the predictive models are clearly failures but anyone who is in this business knows model building is an iteractive process. One can learn from these mistakes. I happen to think they need to wipe the slate clean and use an entirely different approach. It's a small price to pay if there is reward down the road.

I sincerely hope that this coverage will lead to improved modeling and analytical techniques rather than a retrenchment.

If you find other related links, please place them in the comments.