Big Data, Plainly Spoken (aka Numbers Rule Your World)

Reading Gelman's take on election polling

The spector of Covid-19 has pushed U.S. elections off the headlines. Nonetheless, political scientist Andrew Gelman has just offered his take on the state of election polling (link to PDF). This article is perfect for those of us who have a keen interest in but no deep knowledge about polling as Andrew covers how such polls are conducted. Andrew, by the way, was the brain behind the election forecasts by the Economist in the 2020 U.S. presidential elections so he's not merely lecturing from the pulpit.

If you've followed recent U.S. elections, polling inaccuracy has been one of the popular themes. Funny enough, before Trump's stunning victory over Hillary Clinton, the media was trumpeting the exact opposite, how election polls were so damn accurate. Those were the glory days of Nate Silver. Recently, the narrative has flip-flopped: how pollsters could have given Clinton 90-99.9% chance of winning, how polls over-estimated Biden's victory margin, etc.

The following are notes from reading Andrew's article.

***

p. 1

Pollsters are aware that "recent pre-election polling errors have been an underrepresentation of choices favored by lower-education white voters" but "adjusting for ethnicity and education... did not solve the problem... this past November".

This gets at something important: making "adjustments" is necessary but may not be sufficient to make the problem go away. This is a very common issues I see all over the place. One has an observational dataset on people. One then shoves a pile of demographic data (age, sex, income groups, etc.) into the statisticial model, now known as the adjusted model, and voila, heaven opens up and all our sorrows have gone away! If someone asks about biases in the observational dataset, one points to the adjustment factors, and moves on. An analogous situation is peer review in a published journal. Is the research finding credible? In some corners, if it is published in a peer-reviewed journal, there should be no further questions. Reality is much more complicated. Are the right variables in the adjustment? Are the variables structured in the right way? Are there confounding variables that haven't been measured?

p. 2

the moment of truth when poll-based forecasts are compared to election outcomes.

The election prediction problem is unusual in that there is an inescapable, quick, definitive evaluation of the pollster's performance. Election forecasting is the most accountable type of data science. Most data science problems do not have this property. Sometimes, only certain types of errors are visible. In Chapter 4 of Numbers Rule Your World (link), I discussed the anti-doping labs: their false negative errors are unlikely to be discovered, while false positive errors elicit immediate outcry from the harmed athletes.

The key challenges [of polling] are (a) attaining a representative sample of potential voters, and (b) predicting turnout.

The progression is from eligible voters to potential voters to actual voters. There isn't much uncertainty in eligible voters, nor actual voters once the election is over. But turnout is uncertain, which is the link between potential voters and actual ones. Eligible voters are not usually a representative sample from the underlying population.

We cannot expect to eliminate non-sampling errors, because conditions for new elections are always changing. Unlike sampling errors, non-sampling error cannot be reduced simply by conducting more and larger surveys.

The last part debunks a Big Data myth. Collecting more data can't solve all problems. It reduces sampling error but does not eliminate non-sampling errors. Another way to think of non-sampling errors is that the people in the sample aren't representative of those in the population. It is helpful to extend our progression: eligible voters -> potential voters -> surveyed voters -> survey respondents -> actual voters. There are two sources of inaccuracy here: some actual voters may not be surveyed, or if they received the survey, may not have responded.

p. 5

If we are estimating opinion in 4 age categories, 2 categories of sex, 4 ethnicity categories, 5 education levels, and 50 states, that’s 8000 cells. Even if you are analyzing a set of polls with a total of 20,000 respondents, this will still leave with many cells with 0, 1, or 2 respondents, hardly enough to get any sort of estimate.

Each "cell" is a demographic subgroup. Polls rarely have 20,000 respondents so this problem is clearly pervasive. Andrew's statistical approach pools data from "similar" cells so there are enough respondents to come up with an estimate. Such a model is, however, vulnerable to being wrong about "similarity". Nevertheless, any method will be vulnerable to one issue or another because fundamentally, the available data are incomplete. You can, for example, produce just 8 cells based on age and sex alone, and then in each cell, you have enough respondents; but this model misses the effect of all the other variables, and therefore is inaccurate in a different way.

The adjustments only adjust for the variables included in the model.

The beauty of these models is that they are not black boxes. We know exactly what factors are being adjusted for, and the functional form. But if the form is incorrect, the adjusted model will be inaccurate. This points back to the point I made above: be skeptical when someone claims to have solved all biases by throwing a bunch of standard demographic variables into a model - be doubly skeptical if they only include one factor at a time, or only "main effects" - be triply skeptical if the outcome is affected by non-demographic factors, such as behavior.

 

p. 8 He goes into the crux of the matter:

Why were the polls as imprecise as they were? Some possible explanations are differential nonresponse, differential turnout, changes in opinion, and insincere survey responses. These last two explanations, which can also be phrased as “last-minute swings” and “shy Trump voters,” are natural explanations, but I doubt they are a major part of the story in 2020.

By "shy Trump voters", Andrew specifically meant Trump voters who lied to pollsters and claimed they would vote Democratic. His view that this was a non-factor appears to be commonly held among the political science community. I wonder if there is another kind of shy voter who appears under the heading of "differential nonresponse". What if Trump voters were systematically less likely to respond to polls?

p. 9

Some polls also adjust for partisanship, using recorded party registration, stated party identification, or stated vote in the previous election. But even the surveys that made all these adjustments were off by about 2 percentage points, on average.

There is a philosophical question in there. The residual inaccuracy can be either evidence of inadequate adjustment or of partisan bias being unimportant.

p. 10

An error of 2 or 3 percentage points is a problem for predicting very close elections but otherwise is not so consequential.

This is an important observation. The closeness of recent presidential elections makes the error bar a problem.

p. 11

He turns this around and praises the unreasonable accuracy of polls. Think about it. Surveying only some thousands of people among a population of 300 milllion gets the estimates down to a margin of error of only a few percentage points.

p. 13

In retrospect I wish that we’d expressed our 2020 Economist forecast conditionally: instead of simply stating a forecast interval and probability of each candidate winning, we could’ve graphed this interval as a function of the national polling error, thus indicating the confidence we had in our forecast at different levels of survey accuracy and making this dependence clear.

Good idea. Not sure if the public will appreciate it.

well-funded campaigns and advocacy groups... can do more effective survey adjustment using the voter file, which has information including past turnout history on nearly 200 million Americans.

I wonder how much this costs. Sounds like it could be a fortune.

it would be good to see less focus on political campaigns and more on surveys of attitudes

I really like this suggestion. I don't understand the value of election forecasting (except for the campaigns themselves).

 

Why FDA says stick to the test

It pays to know some basic statistics when "experts" are putting out bad information these days. Business Insider publicized a recent study that claims that rapid Covid-19 tests may fail to catch Omicron before the infection becomes infectious unless a throat swab is used in place of / in addition to a nasal swab. The test was designed to evaluate a nasal swab, and therefore, the FDA has come out to recommend against "taking matters into your own hands".

The recommendation to use a throat swab sounds reasonable and logical but is actually flawed. It only makes sense if one ignores the statistics of testing - and with that, one is ignoring the inexactitude of testing science.

I covered a lot of this material in Chapter 4 of Numbers Rule Your World (link) using steroids testing in sports as the motivating example.

***

Imagine making a soup, and there is a bay leaf somewhere in the pot, which you want to fish out before serving. That's the wrong analogy for a steroids test or a Covid-19 test. Testing is not about fishing out a whole organism from a soup.

Realize that typical tests are not looking for the whole virus but some marker - or proxy, which may be some chemical. It's a guessing game. If the sample has an amount of that marker above some threshold, it will be marked positive.

There is tremendous variability in humans, so that the amount of that marker varies from one to another, while the test developer has to decide on one threshold that works for everyone. This just means that no test can be 100% accurate. A false positive is a sample meeting the threshold but the person is actually not infected while a false negative is a sample that comes under the threshold but the person is actually infected. Wherever the developer draws the line, some proportion of the samples will be false positives or false negatives. The stricter the threshold, the fewer samples are classified as positives, and it follows that the test produces fewer false positives, and more false negatives.

How does the test developer calibrate this threshold? It typically must find a set of definitely positive samples and another set of definitely negative samples. The former is expensive to obtain while the latter relies on swabs that were taken pre-Covid-19 and somehow preserved. Now, the lab can measure the rate of false positives and false negatives, with different thresholds. The selected threshold depends on the tolerance of false positives and false negatives.

Notice that the popular Binax rapid home test indicates that they favor having more false negatives to having more false positives. That's why they tell you a positive test is highly credible while a negative test has a chance of error.

Now that you know how tests are calibrated, you should see why the FDA says don't switch to throat swabs using the same test kits. While the conceptual principle of the test works whether you're taking throat or nasal samples, the statistics may not work! With throat samples, or mixed throat-nasal samples, we have no idea if the previously measured false positive and false negative rates apply. The test threshold was selected using the amount of markers in the nasal samples. It is likely that a different threshold would be required if the input materials are altered.

The test developers can recalibrate the tests. So let's wait.

***

The study that Business Insider cites uses a small sample (30 people) that doesn't sound like it was selected randomly, and therefore it's unclear they are representative. The report tells us, interestingly, that all 30 people have had 3 vaccine shots, and 28 out of 30 were infected with Omicron.

Does biometrics authentication work?

As your smartphone scans your face and then unlocks the device, have you ever asked how well biometrics authentication work? Does it work just as well as fingerprints? How does one go about measuring its accuracy? 

***

Biometrics started with fingerprints, expanded to face recognition, and now encompasses other types of measurements such as voiceprints.

The basic steps of biometrics authentication are: capturing the signal (image, voice, video, etc.), turning the signal into data, converting data into scores, which measure the likelihood that the biometrics data came from the device owner, and determining whether to grant access based on  exceeding a certain threshold.

As with any AI/predictive model, the software must be pre-trained using labeled datasets, e.g. images known to be those of the device owners.

The authentication software makes a binary decision (Allow/Block). Block might involve iterative Retries but I'll ignore this complication. Such a prediction system makes two types of errors: blocking someone who is the device owner (false rejection error FRR) or accepting someone who isn't the owner (false acceptance error FAR). While we've all heard anecdotes of people who've been erroneously shut out of their phones, I haven't actually seen a numeric error rate ... until now.

Reading Significance magazine recently, I came across one study that quantifies the error rates of biometrics authentication. A more detailed article by the same team is found in Communications of ACM.

We tend to think fingerprints are unique. It turns out that authenticating users with other biometrics data are much less accurate - multiple orders of magnitude less so! According to these authors, the field summarizes the two error rates with one number known as "Equal Error Rate" (EER), which is the setting under which FRR and FAR have the same values. If you've read Chapter 4 of Numbers Rule Your World (link), you'll hopefully question why FRR should equal FAR. The costs of the two types of errors are clearly different, and reflect an individual treadoff between convenience and security. (Given the lack of scrutiny of these systems, one infers that most people sacrifice security at the altar of convenience.) Note also that falsely rejecting the owner is an annoyance each and every time it happens, directly felt by the true owner, while falsely accepting an imposter may not be discovered until the owner realizes s/he has been harmed.

The researchers said that their face recognition software has EER of 4 percent, which means that 4 of 100 times the owner's face is read, the software erroneously deny entry, and 4 out of 100 times someone else requests access, their intrusion goes undetected.

Voice recognition software (voice-printing) is shown to be more than 8 times worse than face recognition: the error rates are 35% each. About a third of the time the owner requests access, the software would decide to block. (These authors are pitching a system that combines multiple sources of biometrics data.)

***

When reading reports about predictive models, we should examine how the error rates are measured.

Take the false rejection rate. One would have to present images of true owners to the phone, and then compute what proportion of these images the phone erroneously decides to be imposters. Whether the FRR is credible depends on how the investigators select the set of test images. For example, if the test images are the same images used to train the model, the chance of error is smaller.

Each face recognition system has its strengths and weaknesses. Some, for example, are fooled by glasses. People find that they have to take off their glasses to unlock their phones. If the set of test images does not include images of the true owners wearing glasses, then the FRR is under-estimated. If detecting glasses is a strength, not a weakness, of the system being evaluated, the FRR is now over-estimated.

The false acceptance rate is even harder to measure accurately because the investigators must compile a set of images of imposters. There are infinitely many ways to evade the software. The researchers of this study adopted a popular method: "We performed the testing through a randomly selected face-and-voice sample from a subject we selected randomly from among the 54 subjects in the database, leaving out the training samples." This method relies on "randomization".

Almost always, such a method of selecting test samples over-estimates the accuracy rates. That's because in real life, the bad guys are not randomly selected from the entire population - but from the subset of bad guys. The bad guys who are attempting to unlock your phones are likely to exploit weaknesses of the technology. The authors included an interesting example of one such attack strategy. Certain software assesses the quality of the images submitted for verification, and assigns higher importance to higher-quality images. This makes sense but the system is open to a form of attack: the bad guys deliberately submit poor-quality images, knowing that the software can't keep locking out true owners who provide low-quality images.

It's challenging to compile test images for measuring false acceptance rate, as it requires specifying how imposters are likely to attack the system.

The key takeaway is that error rates coming from research studies is heavily affected by the investigators' choice of test images. As modelers, we can fool ourselves by presenting easy but unrealistic images for validation. Ideally, we should employ a neutral third party to evaluate these systems.

***

This last section dives into some technical details, which you can skip if not interested.

Below is a table and a figure included in the Significance article, which provides data on the error rates:

The researchers compared three authentication schemes: face recognition, voice recognition and one that fuses both face images and voice prints (described as "feature-level multimodal fusion"). Table 1 says that the fusion scheme has the lowest EER, followed by face recognition; voice recognition has a terrible EER of 35%.

Figure 2 presents the components of the EER in the form of an ROC curve. However, the results shown in this Figure does not match what is shown in Table 1.

The ROC curve plots the "true positive rate" against the "false positive rate". According to the authors, the false positive rate is the FAR, the chance of letting an imposter through. (This definition implies that a positive result is positively identifying the true owner.) The inverse of a "true positive" is a false negative, that is to say, to mistakenly block the true owner of the device. Thus the FRR (false rejection rate) is the false negative rate, which is 1 - the true positive rate.

With ROC curves, the top left corner represents the perfect system that makes zero false positiive mistakes and 100% true positives (i.e. zero false negative mistakes). The ranking of the three authentication schemes should therefore be Blue > Red > Green, i.e. Feature/Fusion > Voice > Face. But this chart contradicts the data shown in Table 1 where the ranking is Feature/Fusion > Face >>> Voice.

The EER is "the value at which FAR and FRR are equal". The table shows a summary statistic computed from the FAR and FRR while the chart plots the two metrics separately. If the underlying values of FAR and FRR are the same, the ranking should not differ.

Let's locate the points at which FAR equals FRR on the ROC curve. Recall that the vertical axis is 1-FRR while the horizontal axis is FAR. Thus, when FAR equals FRR, y = 1-x.

As shown above, all three schemes have EERs around 20%, with the best one closer to 10%. None of these have EER under 10% and voice recognition isn't 5 times worse than face recognition. So I'm very confused by these figures. I can't find any more details about this research beyond those two articles.

A 20% error rate is a far cry from what we have come to expect from fingerprinting.

 

The unexpected gift of a well-conducted study

We're drowning in a quicksand of bad-quality data during this pandemic, and frustratingly, the situation is getting worse, as we enter the second year. It would have been a challenge to measure the real-world impact of vaccines because of the mass vaccination initiatives. We then made our lives much harder by (a) destroying the placebo groups in the original randomized clinical trials, fossilizing them at 4 months of follow-up (see this post); and (b) applying different rules of masking, testing, hospitalizing, attributing causes of death, etc. to vaccinated and unvaccinated subgroups, literally injecting massive biases into observational data.

It comes as an unexpected gift to find a study that takes on these challenges seriously. Today, I review the preprint of the REACT-1 Study from the U.K. There is a large team of people involved, many of whom from Imperial College.

***

In this study, the researchers collect swabs from randomly selected people from all over England. The study is ongoing, with monthly cohorts. The report I'm reading covers Round 13, which was conducted from mid June to mid July. This cohort is monumental because Round 13 spanned a critical point in the case curve in the UK.

In the period leading to Round 13, the U.K. government was congratulating itself on the hugely successful vaccination campaign, about to declare mission accomplished. But by July, the virus has returned with a new rigor. In Round 13, the researchers analyzed about 100K swabs, with a positivity rate of 0.6%. That is more than four times higher than the positivity rate in Round 12.

The positivity rate found in the REACT-1 Study - especially after adjustments and weighting - can be generalized to the population of England. That's because the study utilizes a random sample. The swab tests find not just symptomatic but also asymptomatic Covid-19 cases. This study gives much better information than most real-world studies - which usually tap into a database of positive test results. A severe problem with those studies is selection bias. Vaccinated people, people with mild symptoms or disease, and people who live in lower-exposure areas are less likely to get tested, so their cases are under-counted. (That said, the "good" study still has to deal with biases, which I'll get to later in this post.)

***

There are many things we can learn from the REACT-1 study. The first thing is that the chance of anyone catching the virus - regardless of vaccination status - is low. Only 6 out of 1,000 people in England were actively sick with Covid-19 during July. The chance of getting severe Covid-19, getting admitted into hospitals, or dying from Covid-19 is a fraction of that number. The danger comes from the spread of the virus, so that a large base of the infected produces a large number of deaths. (Even at its peak in England - Round 8, the prevalence was 1.6%.)

What about the 994 people out of 1,000 who did not get sick during Round 13? We are frequently misled into thinking that the fully vaccinated subset were protected by the vaccine. The fallacy is made clear when you ask about those who are unvaccinated. One can't get infected if one isn't exposed. A lot of these people - vaccinated or not - did not get exposed to the virus during the study period. Most of the calculations you see out there implicitly assume that everyone is exposed every month, month after month.

I am not saying none of those 994 people got exposed. Some proportion of those were not infected because they did not get exposed. This is a missing data problem that we haven't grappled with yet. If they do get exposed, it's likely that the vaccinated ones have better outcomes; better may mean 30-50% reduction in chance of getting sick; we now know it's not anywhere close to 90%.

***

In "explaining" the surge in cases in England in July, the study's authors pointed the finger at unvaccinated 13-17 year olds. They drew this conclusion by comparing the percentage increase in infection rate between Rounds 12 and 13 by age group. See this chart (with my edits):

The study estimates that the prevalence of Covid-19 in the 13-17 age group jumped from 0.2% to 1.6%, which is a nine-fold surge, compared to an aggregate surge of about 4 times.

Are the teenagers responsible for the entire surge? To answer this question, I looked up the age distribution of England's population (shown on the chart in the green boxes).

From this analysis, I found that 13-17 year olds accounted for 15% share of cases in this period of time. The bulk (60%) of cases affected those between 18 and 64.

The above analysis contains one hopeful insight. The 65+ age group, which enjoys the highest vaccination rates, is getting some measure of protection, as indicated by the lower share of infection. This goes a long way to explaining why deaths in England have not increased as much as infections - younger people are getting infected, and they tend to recover more readily.

(Statements about deaths must be interpreted with great caution. This study is typical: the cutoff date of Round 13 was in the middle of the surge of cases, which means the analysis covered a period of time when the deaths arising from the surge of cases have not occurred yet. The researchers would never return to update the analysis and so we are always dealing with over-optimistic, premature estimates of effectiveness against deaths.)

***

Next, I made a similar analysis breaking the study population down by vaccination status: fully vaccinated, partly vaccinated and unvaccinated.

This leads to a surprising finding that the surge in infections was even stronger among those who were vaccinated (at least one dose) than those who were unvaccinated. This problem illustrates the difficulty of observational datasets. There are so many possible explanations for this. My own favorite is that many vaccinated people have stopped taking precautions.

***

Another nice feature of the REACT-1 preprint is the careful description of how they interpreted test results. It's not a straightforward science - something I explained in Chapter 4 of Numbers Rule Your World (link) using testing for steriods and HGH as a case study. Someone has to draw a line on some continuous scale to decide who's positive and who's negative.

These UK researchers described their positivity criteria as "both E and N gene targets were detected, or if N gene was detected with CT value < 37". CT is cycle threshold: the lower the CT value, the higher the viral load.

I highly recommend reading this section (pp. 10-11) of the paper. You will learn that to figure out what positivity threshold to use, the scientists need to procure positive and negative controls (samples known with 100% certainty to be positive or negative), these controls may be synthetically created, experimenters may be blinded or unblinded, testing analysis may be done externally or internally, multiple experiments by multiple labs may be required, these results may not agree, etc. In other words, scientists don't wave their hands and write down CT < 37; there is method to the madness.

Unfortunately, there does not seem to be a global standard for what test to use and what positivity threshold to apply. This is yet another aspect of the data quality problem I mentioned up top.

***

While the selected participants represent the whole of England, the study cannot escape selection bias in the form of non-response. The researchers reported that response rates (of sending back the swabs) have declined steadily and was at 12% in Round 3. That's a very respectable number.

What's concerning is that the responders were clearly different from the non-responders. Response rates for younger people and people living in deprived neighborhoods were only in the 5% range. Those, of course, are also the least vaccinated subgroups.

As if to prove the point that biases pervade every corner of real-world data, the researchers also reported finding selection bias among those who gave them permission to link their NHS records. Ex ante, it is not obvious why there should be a difference in infection rates between those who gave consent and those who didn't.

It turns out vaccinated people who gave consent are more likely to get infected than vaccinated people who didn't give consent. The reverse is true for unvaccinated people.

(Linking is used to conduct off-book accounting as described in this post. This process disappears vaccinated, and infected people because of some arbitrary case-counting window - which cannot be applied to the unvaccinated. In the case of England, the 2D+14 window means that no cases are counted on the vaccinated side until at least 14 weeks after the first shot because of a long delay between doses, based on embarrassingly wrong analyses by advisors to the UK government. This gimmick puts our most creative corporate accountants to shame.)

***

We are drowning in bad-quality data. Don't trust any study that gets published. Be discriminating, and value those researchers who are making an honest effort at doing good analyses. It takes a lot of work to properly handle real-world data, so let's take a moment to appreciate those who tackle this task seriously.

 

 

 

 

Sizing up the pandemic

From the beginning, the U.S. has never gotten religion about accurately measuring the pandemic. In March 2020, I wrote a piece for Wired about the need for broad-based testing because selectively testing only severe cases is bound to give us a biased picture of the crisis. It was known even then that a significant portion of infections is asymptomatic - which constitutes a silent path by which the virus passes from person to person. The situation has not improved much since. Recent counting rule changes imposed selectively on vaccinated people have made it far worse but that's the subject of a different post.

I'm delighted to report on a valiant effort to correct this oversight - in Slovenia, a country of about 2 million people. The relevant papers have just been published here and here.

***

Because we want to minimize bias, randomization is our friend. I have previously explained how randomly assignment treatment in a clinical trial creates covariate balance (here) which allows researchers to assume "all else equal" except for the treatment.

In this case, we want to test a random sample of people selected from the population to avoid the self-selection bias that permeates the usual counts of "cases". The demographic and medical profiles of the sample should mimic that of the entire population. In the Slovenian study, they apply standard statistical sampling methodology, as follows: the country is divided into 300 census districts; from each district, they select 10 persons at random; a total of 3,000 Slovenians were sent communications inviting them to participate in this study; just under half agreed, leading to a sample size of 1,368.

Because of this random (i.e. broad-based) testing, the researchers uncovered not only self-reported symptomatic cases but also asymptomatic cases. All participants were given PCR tests at the start of the study in April, which was towards the end of the first wave in Slovenia. The estimated prevalence was 0.15%, or 150 per 100,000.

It's important to understand prevalence as a measure of infections at a point in time (late April-early May 2020 in this case). The PCR test reveals whether someone is currently or has recently been infected with the novel coronavirus. This metric does not answer the important epidemiological question: what proportion of the population has been infected since the start of the pandemic?

***

The research team's second paper deals with this question - the canonical approach is through serological testing i.e. testing for antibodies. Most people who have been sick with Covid-19 have developed antibodies. So we expect the proportion of population that have had Covid-19 to be larger than the proportion who are currently/recently infected. Technically, we say the seroprevalence is greater than the prevalence.

I have discussed serological testing before - at the start of the pandemic. Two research teams - one at Oxford and one at Stanford - jumped the gun when they published serological studies during the first wave of Covid-19. The Oxford team (in)famously declared that up to 40% of the U.K. population have already been infected with SAR-CoV-2 by April 2020 - a result that has proven extremely embarrassing. To be precise, they used a mathematical model to come up with the alarming headline, and called for serological studies. The Stanford team argued that total infections in California were 50 to 85 times the reported number of cases - back in April 2020. This group actually performed tests to derive their estimate. Their analysis was roundly criticized. Recent handwaving analyses tend to multiply cases by 4 times, which gives you an idea of the magnitude of the error in the Stanford study.

Meanwhile, I'm still expecting that one day, the New York State Department of Health will release the data behind the almost daily pronouncements by Governor Cuomo - again in spring and summer of 2020 - that at least 15% of New Yorkers have been infected. Those numbers were supposed to have come from antibody testing throughout the state of New York but no data or science has seen the light of day. While not as far-fetched as the two other studies, that estimate of seroprevalence is also clearly repudiated by subsequent waves of Covid-19.

In the Slovenian study, the unadjusted estimate of seroprevalence around mid-November 2020 (in the middle of a devastating second wave) was only 4 percent. At the time, the cumulative reported cases was around 27,000, or 1.3% of the population. This estimate is much more solid than the previous studies because these researchers used a random sample. The design of research studies matters a lot - when devices like randomization are utilized, fewer adjustments and assumptions are necessary to interpret the data. (See my three-part talk to learn more about research methods.)

***

Much of the second paper deals with the complications of the analysis even though the team utilized a random sample. The study interviewed participants at 3-week intervals. There were two rounds of blood samples (April 2020, and October 2020).

In April 2020, the reported cases (about 700) was 0.03% of the Slovenian population. With serological testing, they estimated seroprevalence to be roughly 0.8-0.9%, so roughly 18,000 people have been infected. More precisely, they estimated that 0.8-0.9% of their random sample have evidence of SARS-Cov-2 antibodies (that are true positives), and generalized that result to the entire population.

The raw data actually showed 3 percent, which was revised down to 0.8-0.9%. This is because of two factors that artificially inflated the unadjusted number.

The first is test inaccuracy. All tests are inaccurate to some degree - which is a key message of Chapter 4 of Numbers Rule Your World (link). In this situation, because the vast majority of Slovenians have not been infected, false-positive errors can be devastating to any analysis. Even a small false-postive error rate (e.g. 2 or 3 percent) gets amplified by the large base population into an abundance of incorrect positive test results. Correcting for test inaccuracy therefore pulls down the number of positive cases, and thus seroprevalence.

We have encountered this issue before - when pointing out the methodological weaknesses of the 2020 Stanford study. Their initial analysis did not correct for false-positive results. But it's actually a really hard problem. To know that a positive result is definitely a false positive implies that we know the true result - if we did, we would not need to run a test. In practice, the researchers must obtain an estimate of the probability of false positives from other studies. 

For example, they gather blood samples collected for other reasons - preferably from before first half of 2019 when they are sure the blood samples did not have any SARS-Cov-2 antibodies (or RNA, depending on what test). Then they apply the test to these blood samples, and check what proportion of them give positive results despite clearly being negative.

Such definitively negative blood samples are hard to procure and so our knowledge of test accuracy is limited. Many researchers just trust the test manufacturer's claims, which is a bit shocking since they market their tests, in essence, as 100% accurate. The Slovenian team did a literature search, and discovered that the accuracy of the antibody test they were using was not 100%, and the accuracy measures varied widely from study to study. Thus, the second factor the researchers adjusted for was the variability of test accuracy. (*)

***

Because of concerns about the first testing platform, when they analyzed the second set of blood samples In October 2020, they switched to a different testing platform, which appeared to have better and less variable accuracy statistics.

Despite random sampling, there may be non-response bias - the people who chose to participate in this study are not necessarily the same as those who opted out of the study. If the non-responders are different from responders, the result from this study cannot be generalized to the entire population without more corrections.

If you notice, this study is much more rigorous than the real-world vaccine effectiveness study that I have recently featured - those studies sweep most of the biases under the rug.

Nevertheless, the responders and non-responders look similar enough - by the variables they analyzed. It is therefore not surprising when the authors reported that a further correction for non-response bias did not materially change their seroprevalence estimates.

***

This Slovenian study is a good example of proper science that doesn't cut corners. What's notable is that seroprevalence grows at a much lower rate than reported cases. While cases have jumped almost 40 times during the six months of the study, seroprevalence has risen about 4 times (call it 10 times if we allow for the error bar).

 

P.S. [*] The study employed Bayesian methods throughout. Here is how they adjusted for variability of test accuracy. Different prior studies have come up with a range of estimates for test accuracy. If one doesn't adjust for this, one would take the average value from these studies, and apply this to the analysis. For example, if the test specificity (true negative rate) is 98%, then you assume exactly 2% of the negative blood samples would show up as false positives.

In a Bayesian model, the test specificity is treated as a variable: its average value is 98% but it can vary within some reasonable range (as informed by the variability observed across those prior studies).

 

 

 

 

 

 

 

A Look Back at 2020

There is no bigger news in 2020 than the pandemic. This once-in-a-lifetime (hopefully) event has shadowed much of my blog writing last year. I'm glad you've stuck around, and if you've been reading from the start, you'd have been ahead of the curve on all the key issues. Data have been central to navigating this new world, however messy, sparse, and incomplete they are. I'm confident you've picked up more about statistics on these pages than in an entire statistics course.

In this post, I've selected some of the posts that have aged particularly well. If you missed these before, be sure to read them now!

***

In mid March, as New York was poised to enter a lockdown, I predicted that this lockdown would have a different ending than those in China or South Korea.

The second type of lockdown - soon to be exemplified by the U.S. - is based on severely limited data. It is based on the assumption that everyone has been infected. The necessity of this kitchen-sink assumption is driven by the lack of knowledge. It is a position of weakness. The duration of the lockdown will be longer, the spread of infections will be wider, and the long-term costs will be higher. It is not the case that everyone has the coronavirus but we have to assume it because we are not testing nearly enough people.

Nine months later, the U.S. has not tamed the virus while the economy has suffered.
(1) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/two-types-of-lockdowns-why-us-must-rethink-its-testing-policy.html

In March and April, when news outlets were one-upping each other with variations of log-scaled charts showing the "exponential" rise in Covid-19 cases, I pointed out that the growth curves were sub-exponential, which also means the log scale is inappropriate. Today, log-scaled charts have disappeared because they have been proven useless.

(2) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/note-about-fitting-and-visualizing-exponential-models.html
(3) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/new-video-validating-data-science-models-a-case-study-with-covid-19-data.html
(4) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/another-note-on-fitting-exponential-models.html
(5) https://junkcharts.typepad.com/junk_charts/2020/06/presented-without-comment.html

In May, I was already drawing the connection between advocating antibody testing and pushing herd immunity:

In places that have fallen down on rolling out diagnostic testing (the U.S. and the U.K., for example), the officials have started talking up “antibody testing”... Antibody testing is a rear view mirror. If someone is found to have antibodies, it’s because the person has recovered from the disease. It follows that the person was infected, and so the person could have died. Antibody testing without diagnostic testing means death over life.

In the same post, I made the following alarming calculation:

With a 0.1% fatality rate, conservatively set at that of influenza, the U.S. would suffer over 200,000 deaths to get there. The path to herd immunity is paved with body bags.

The current U.S. death toll is over 300,000 and counting.
(6) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/the-shift-to-antibody-testing-is-a-choice-of-death-over-life.html

Nevertheless, several countries (U.S., U.K., and Sweden, being the most prominent) continued to deny advancing herd-immunity policies, while most observers recognized them as such. In June, I pointed out that one cannot both believe the coronavirus is the common flu and support herd immunity through natural infections because influenza is seasonal, signifying that natural infection does not confer immunity.
(7) https://junkcharts.typepad.com/numbersruleyourworld/2020/06/coronavirus-has-not-gone-away-neither-have-these-covid-fallacies.html

In October, I objected to herd immunity as a public health objective as it rewards the wrong behavior. I warned against bystander behavior because "you have tied your health to the actions of people who hold opposite views as you." Herd immunity is mathematical immunity, not biological immunity.
(8) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/i-dont-like-the-term-herd-immunity.html

At the end of October, I pronounced the death of the Swedish experiment. I have seen enough. If your soccer (football) team goes down 0-10 in the first 10 minutes of a match, what choice words do you have for the coach who is preaching patience, waiting for the comeback win? By mid-December, Sweden's King admitted that "we have failed."
(9) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/the-swedish-mirage-the-verdict-is-already-written.html

At the time, I assumed 75 percent of the population would need to be vaccinated to attain herd immunity. Many experts placed the number much lower, about 40-60 percent (link). By end of the year, Dr. Fauci is estimating a range of 70-85%.

In one of my very first posts about the coronavirus in early March, I predicted the thorny data quality issues that would bedevil analysts to this day:

We have very limited amounts of data (the known and suspected cases), incomplete data (under-reporting, possible hiding or manipulation), data of dubious quality (self-reporting, hastily assembled tests, unpreparedness), assumptions that later turn out to be false, the potential black-swan fallacy (failure to imagine what hasn't happened before), and non-stationary data (old data might not be wrong but become outdated as the virus evolves).

(10) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/eight-unanswered-questions-about-the-coronavirus.html

In March, I already warned about "the elephant in the room: people are counted as infected only if they are tested", and asked how deaths are confirmed. It would take weeks of regurgitating official counts before the mainstream media acknowledged these shortcomings.
(11) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/eight-unanswered-questions-about-the-coronavirus.html

In April, I explained the benefit of "excess deaths" while also noting why death statistics are subject to their own set of issues. It is here that I said all excess deaths are accelerated deaths. (I will have another new post about this point in early 2021.)
(12) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/excess-deaths-why-we-want-the-cake-and-why-we-cant-eat-it-yet.html

In May and then in December, while studying the data on excess deaths in Florida, I created the following graphic showing the sale.

(13) https://junkcharts.typepad.com/junk_charts/2020/05/how-covid-19-deaths-sneaked-into-floridas-statistics.html
(14) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/its-time-to-visit-florida-again-graphically.html

In May, when Dr. Birx the politician complained that CDC data was terrible, I responded thus:

The members of the Task Force can immediately help make the data better if they cared about the quality of the data. When they aren't pushing diagnostic testing, while trashing the data, their real perceived enemy is not bad data but data.

(15) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/their-real-enemy-is-not-bad-data-but-data-period.html

By June, the media have gone through the five stages of grief:

In the short history of the Covid pandemic, people started with case statistics. Then, they claimed that death statistics are less manipulated than case statistics, when they learned about testing. Then, they claimed that testing statistics are less manipulated, until they realized governments determined who got tested. Some governments counted tests shipped, not test results. Then, they claimed hospitalization statistics are less manipulated, until they learned that hospitals sent sick patients to nursing homes.

(16) https://junkcharts.typepad.com/numbersruleyourworld/2020/06/coronavirus-has-not-gone-away-neither-have-these-covid-fallacies.html

The antibody testing lobby has a particular disregard for data quality. New York's Governor Cuomo, whom I heard won an Emmy for his press conferences, claimed for weeks that 20 percent of New Yorkers had already been infected by April, citing "randomized" antibody testing at supermarkets (during lockdown). To this day, the only "datum" provided by the state's health department which conducted these tests is this FAQ. (Click with no expectation!)

Around the same time, an Oxford research team made headlines, claiming 40 percent of the U.K. population were already infected based on antibody testing. This finding was laughable at the time, and embarrassing given the latest surge in the U.K. In April, I wrote a series of six posts about the Oxford study that explains how scientists build models based on sparse data to make such predictions.

(17) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-1.html
(18) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-2.html
(19) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-3.html
(20) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-4.html
(21) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-5.html
(22) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-6.html

The lead investigator of the Oxford study turns out to be a co-author of the Great Barrington Declaration, which pushes the amoral and dubious theory of herd immunity through natural infections.

Another author of that Declaration is a Stanford professor who is infamous for his own role in the debunked study of antibody testing in Santa Clara, California.
(23) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/the-percent-with-antibodies-could-be-3-percent-or-it-could-be-zero-thats-what-the-stanford-study-rea.html

The first draft of the Stanford study neglected the enormous false-positive problem associated with testing rare events, a topic which I covered in depth in Chapter 4 of Numbers Rule Your World (link).
(24) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/to-justify-zero-false-positive-assumption-stanford-study-needs-a-reference-sample-10-times-bigger.html

The corrected manuscript reveals statistical issues with estimating test accuracy when developing new tests, and failed to quiet the skeptics.
(25) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/the-fda-looks-past-these-statistical-issues-but-maybe-you-shoudnt.html

Most importantly, the antibody testing lobby lost credibility in the face of real-world evidence. They predicted a high level of immunity from natural infections, which would prevent a second wave. Everywhere they made that prediction - Sweden, U.K., California, reality bit back, hard.

Meanwhile, mainstream epidemiology has triumphed. Certain colleges re-opened in the fall, implementing strict test, trace and isolate protocols that were swatted aside by the U.S. government. I wrote about the successes at Cornell and Georgia Tech.
(26) https://junkcharts.typepad.com/numbersruleyourworld/2020/09/good-news-from-cornell.html
(27) https://junkcharts.typepad.com/numbersruleyourworld/2020/08/nice-explainer-from-georgia-tech-about-their-re-opening-model.html

Epidemiologists also have predicted the rise of deaths following rise of cases. This lag is frequently ignored by journalists who are professionally trained to think only in the present, and not one or two weeks ahead, and by politicians who say black is white. In a series of posts, I showed how to see the time delay between cases and deaths, in Lombardia (Italy), Texas, Illinois and Iowa.
(28) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/how-to-act-like-a-data-scientist-9-quantifying-our-intuition.html
(29) https://junkcharts.typepad.com/numbersruleyourworld/2020/06/how-to-act-like-a-data-scientist-12-say-no-to-the-daily-grind.html
(30) https://junkcharts.typepad.com/numbersruleyourworld/2020/07/its-high-time-to-call-out-those-who-got-it-right.html
(31) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-importance-of-matching-data.html

As the year ends, we received good news from several vaccine trials. In July, I complained that Moderna provided almost no useful information in its submission to ClinicalTrials.gov, the supposed center of open science.
(32) https://junkcharts.typepad.com/numbersruleyourworld/2020/07/were-told-almost-nothing-about-the-moderna-vaccine-trial.html

By September, Moderna was pressured to release detailed protocols. So followed other vaccine developers. Once these protocols were released, I computed that the earliest approval date for vaccines would be mid January 2021. At the time, the media reiterated unrealistic timelines suggesting results by Election Day (early November).
(33) https://junkcharts.typepad.com/numbersruleyourworld/2020/09/notes-on-the-moderna-vaccine-trial-protocol-1.html
(34) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/political-headline-decrying-politics.html

If the pharmas adhered to those timelines - the Pfizer CEO was pretending they could, it implied that the trial participants would have been observed for only one month, which would at best prove vaccine efficacy for one month. Such a possibility was squashed when the FDA "strengthened" guidelines around the observation window.
(35) https://junkcharts.typepad.com/numbersruleyourworld/2020/09/behind-the-proposed-strengthening-of-fda-vaccine-approval-rules.html

This guideline was not as strong as it could be. I proposed that all participants - instead of the median participant - be observed for at least two months. A week later, a group of prominent statisticians sent a letter to the FDA, making this same point.

When reviewing the trial results, I noted that neither Moderna nor Pfizer met that FDA guideline for median observation window. This is one of numerous tradeoffs made to rush the approval process.
(36) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-worst-kept-secret-will-be-revealed-today.html
(37) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-second-worst-kept-secret-of-the-pandemic-year-and-some-decimal-mischief.html

The metric of vaccine efficacy is frequently misinterpreted. I wrote about VP Mike Pence's tweet, claiming that 90% VE means the vaccine prevents infections in 90% of volunteers. Not even close. Most of the trial participants have not yet been exposed to the virus so they would not get infected whether they got the vaccine shot or the placebo.
(38) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/cutting-the-infection-rate-by-half-is-not-the-same-as-protecting-half-the-people-from-infection.html
(39) https://junkcharts.typepad.com/numbersruleyourworld/2020/11/watching-statistical-gravity-in-action-elections-and-vaccines.html

These vaccine trial results demonstrate a key tenet of designing experiments: decisions have consequences.

Neither Pfizer nor Moderna has proven vaccine efficacy for two months. Their results are based on less than half the participants reaching the two-month mark. If the FDA rule had been all participants reaching two months of observation, then yes.
(40) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/unexpected-sightings-and-unknowns-from-the-pfizer-result.html

We cannot conclude that one dose gives partial protection for two months because no one tested single doses. Anyone suggesting otherwise makes the assumption that the second dose has no value, creating a tautology, saying the same thing twice.
(41) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/one-dose-pfizer-is-not-happening-and-heres-why.html

Any analysis of subgroups (race, age, severe cases, etc.) should be ignored. The trials were designed to minimize total sample size so as to achieve warp speed; the casualty of this design is subgroup analysis.
(42) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-worst-kept-secret-will-be-revealed-today.html

Throughout this pandemic, the U.S. government, apparently supported by CDC experts, ignores the threat of asymptomatic spread. The policy of triage testing discourages people without symptoms from getting tested. The design of the vaccine trials - with case definitions based on self-reported symptoms - produces no knowledge about asymptomatic transmission. In March, I wrote a column in Wired to call for broad-based testing.
(43) https://www.wired.com/story/the-problem-with-trumps-triage-testing/
(44) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/when-will-we-get-more-and-broader-testing.html

The clinical trials provide higher quality data compared to research studies that use observational data. I outlined the many problems of observational studies that were thrown together in haste, and published as preprints. These include a Yale study of sewage, a Harvard study of parking lot images and search engine traffic, and a Kings College (UK) study of mobile tracking app data.
(45) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/yale-meds-please-meet-yale-stats.html
https://junkcharts.typepad.com/numbersruleyourworld/2020/06/a-glimpse-into-surveillance-data-via-the-harvard-study-of-parking-lots.html
(46) https://www.wired.com/story/beware-the-lofty-promises-of-covid-19-tracker-apps/
(47) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/it-begins-with-the-data.html
(48) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/pre-processing-data-to-avoid-putting-your-foot-in-that-puddle.html

All these examples show that a model that explains the past well does not necessarily predict the future.
(49) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/prediction-versus-explanation-simple-to-state-yet-hard-to-grasp.html

Regarding the Covid-19 symptom tracking app, I explained how the analytical methodology for estimating population prevalence can justify any level of prevalence. That was back in April. We have barely heard a pip about Covid apps since then.
(50) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/the-first-major-study-using-covid-tracking-app-data-stumbles-out-of-the-gate.html

Between blogging, I was a guest on the CSAI podcast with John Reid and Rusen Aktas:
(51) https://anchor.fm/csai-podcast/episodes/CSAI-2--Covid-19-and-Data-Science-ege3si

More recently, on Ryan Ray's podcast, I talked about how I wrote my books:
(52) https://junkcharts.typepad.com/junk_charts/2020/11/podcast-highlights.html

On the sister blog (Junk Charts), I featured several nice data visualizations. SCMP published this beautiful picture of contact tracing efforts in Hong Kong:
(53) https://junkcharts.typepad.com/junk_charts/2020/09/unlocking-the-secrets-of-a-marvellous-data-visualization.html

The New York Times visualized excess deaths in the U.S.:
(54) https://junkcharts.typepad.com/junk_charts/2020/08/deaths-as-percent-neither-of-cases-nor-of-population-deaths-as-percent-of-normal.html

The New York Times traced the exodus of the rich from NYC:
(55) https://junkcharts.typepad.com/junk_charts/2020/06/designs-of-two-variables-map-dot-plot-line-chart-table.html

The Visual Capitalist depicted how we changed our consumption habits:
(56) https://junkcharts.typepad.com/junk_charts/2020/05/consumption-patterns-during-the-pandemic.html

***

I wrote 212 posts across the two blogs in 2020. Since I started, people have read posts more than 4 million times. Thank you for reading, come back often, dig into the archives, and recommend me to your friends and colleagues. See you in 2021!

Kaiser

 

[1-1-2020: Added an index to the links, and corrected a link error.]

Numbers indeed rule our world, 10 years out

Statistics is the subject of generalizing data. In statistics, we study how to use imperfect data to draw general conclusions. We often have to wait for the future to reveal the evidence of success.

Ten years ago, McGraw-Hill published my first book, a best-seller, Numbers Rule Your World (link). I’ve hoped to explain five of the most profound concepts from my field in a way that anyone can understand and apply to their daily lives. I attempted the balancing act of featuring specific stories while bringing out the general principles. In the last couple of months, the Covid-19 pandemic has generated immense attention to data and statistics, and provided me confirmation, ten years on, that the contents of Numbers Rule Your World (link) have general applicability. It affirms that statistics is a great subject!

If you’re approaching my book for the first time, or re-reading it, here’s how you can work your way through it, making connections at various points to the Covid-19 crisis.

***

Chapter 2: Bad Spinach / Bad Scores

This is your perfect starting point. Half of the chapter details a disease outbreak investigation. You learn about the information networks the CDC puts in place to monitor unusual patterns of disease around the country. You discover data analysts whose job is to set off the alarm. How do they decide if the first handful of cases foretell an epidemic?

You encounter “shoe leather,” the painstaking collection of data to pinpoint the origin of an outbreak. It’s much harder than you might think. The U.S. politicians are making much noise about the early days of Wuhan. An epidemiologist may learn that 10 out of 15 first patients all visited the Wuhan seafood market before they got sick. But the Wuhan market might be a hugely popular destination, like Central Park in New York City. Finding 10 out of 15 people went to Central Park over a weekend could be more common than you think. If all 10 touched a particular rare species of wildlife, that's another story.

Another controversy of the moment is the huge cost of shutting down the economy, against the benefit of saving lives. I invite readers to think about this cost-benefit tradeoff. Because the disease outbreak was traced to a spinach processing plant in California, the FDA ordered a recall of all spinach, which depressed sales of salad greens for six months or so. The cost was much lower than for this pandemic, but the bacteria is well-known and thus predictable, and the lives saved were most likely zero, and even harder to pin down than in the current crisis.

People are already complaining that social distancing and other measures have not improved our numbers, by saying deaths were “unexpectedly low”. You discover how hard it is to prove the effectiveness of a policy decision because you need the counterfactual – what would have happened if the FDA did not recall the spinach.

Chapter 4: Timid Testers / Magic Lassoes

Everything you should know about the statistics of diagnostic testing and predictive models in one place. The motivating examples are (a) the fight against performance enhancing drugs in sports via anti-doping tests and (b) the false promise of terrorism prediction algorithms.

I wrote several posts (1, 2, 3) about the flawed Stanford antibody study that reported 50 times more infections in Santa Clara county in California than reported cases. Despite this headline, the raw data is also consistent with a scenario of zero proportion having antibodies. The statistical reason is the low underlying prevalence of antibodies, which caps the number of true positives in any sample. The number of false positives will be large relative to true positives, because the false positive rate applies to those without antibodies, which is the bulk of the population.

An extreme case of this phenomenon is terrorist prediction using a dragnet surveillance dataset. The set of terrorists is a drop in the ocean of people under mass surveillance. For a predictive model to capture most of these terrorists (low false negative rate), it must flag a lot of people – even a tiny false positive rate when applied to a huge population results in a lot of false accusations.

You see the distinction between a “chemical” false negative (positive) and a “practical” false negative (positive). To evade steroids testing, dopers use all kinds of tactics, such as handing in someone else’s urine. The diagnostic test finds a true negative but in practical terms, it’s a false negative. For swab testing, one may test negative if the swab doesn’t reach a virus-rich area, or if the patient is tested at the “wrong” time.

All tests have errors. What are the consequences of errors? Also, how likely will the error be discovered? For anti-doping, do dopers inform the test lab they made a mistake?

Chapter 1: Fast Passes / Slow Merges

The first chapter gives important insights on managing capacity, which is a pivotal issue for hospitals during the coronavirus crisis. I cover two case studies: (a) Disney’s management of theme-park capacity, and (b) state’s management of highway congestion.

The strategy of building more capacity is unavailable during a crisis, and wasteful in general. In this extraordinary time, some localities were able to bring up temporary hospitals. Transportation experts discovered that policies of regulating congestion are more effective. Highway ramps slow down the influx of new vehicles onto the highway, and are turned on before congestion arrives. The key is to delay as much as possible the onset of congestion. Sounds familiar? It’s the analogue to “flattening the curve”. Social distancing is supposed to reduce contacts, slowing down community spread. This tactic delays the peak demand for hospital resources. The chapter explains the scientific support for why such tactics work.

Chapter 5: Jet Crashes / Jackpots

Statistical significance is the topic of this chapter, one of the most common and also most commonly mis-used concepts in statistics.

In the Stanford antibody study referenced above, statisticians argued that the observed level of positive results is consistent with zero prevalence. This is a way of saying the result is not statistically significant. What we mean is that the study’s result (50 positives out of over 3,000 tests) would be considered normal even if nobody in the population actually had antibodies. This conclusion does not assert that the true prevalence is zero. The topic of statistical significance is made inaccessible to non-mathematicians by textbook authors piling on equations, but it need not be so. I even made a video on explaining "not statisitical significant" (link).

Before you apply a test of statistical significance, you must make sure you’re comparing apples to apples. In the context of air safety, you shouldn’t compare U.S. and foreign airlines on all routes; you should only compare them on routes they share, in which case, you’ll find no statistical difference. One of the bad stats that is circulating during this pandemic is the purported harm of ventilators. This is usually stated as the vast majority of patients put on ventilators eventually die (presumably, relative to those patients not put on ventilators). But only the most serious patients are placed on ventilators. The proper analysis should focus on patients who qualify for ventilators, and compare those who were put on ventilators to those who got other treatments.

We have already seen a host of invalid comparisons. Comparing complete data from years ago with incomplete data this year. Comparing regions that are in different phases of the epidemiological timeline. Comparing countries that have taken different approaches to managing the crisis.

Chapter 3: Item Bank / Risk Pool

This chapter addresses a deeper issue in using statistical averages. An illustrative example is educational testing like the SAT. Critics pointed to the gap in the average scores across racial groups as evidence of bias. But the racial grouping masks a third factor, which is differential ability between students. Due to various socio-economic factors, white students have higher average ability than black students. So the real question is: Among those students with comparable abilities, does the test still show bias toward one racial group? (The answer is not straightforward.)

During the pandemic, the “denier” faction has argued that the mortality rate of Covid-19 is “on a par with” that of seasonal influenza. Are those two rates comparable? The simple answer is no. Comparing mortality rates masks a third factor, which is the level of immunity in the population. Even at the most optimistic, the current level of immunity to the novel coronavirus is less than 20 percent (This statement generously presumes that (a) having antibodies confer immunity without question and (b) the New York City test result is not a scam, and the test is 100% accurate). By contrast, because of vaccination and recurrence, a good chunk of people have immunity to influenza. The mortality rate of Covid-19 should be compared to that of influenza among a population that has no immunity to either, or a population with similar levels of immunity to each.

The other example in this chapter explains how the insurance industry is an outgrowth of the law of large numbers. In any given year, health or auto insurers cannot predict with high accuracy which covered individuals will file claims, but the magic of statistics allows them to predict with great accuracy the aggregate claims. Disaster insurers run into trouble when rare events happen, such as when Class 5 hurricanes hit Florida. Suddenly, potentially all covered individuals tap into the insurance pool all at once. Insurers try to pool together risks of different types and different areas. The global pandemic is a huge threat to the insurance industry. Perhaps there are lawyers reading this, and they might tell me there are natural disaster exclusions. If those don’t apply, we will see a gigantic number of covered entities filing claims all at once.

Florida’s hurricane insurance market was dysfunctional also because the risk is predictable along some parameters. Those living inland realized they were subsidizing the coastal dwellers and wanted out. The coronavirus also creates different risk subgroups. The younger generations have much lower risks than the elderly. When social-distancing and other practices are applied to the whole community, the subgroup with lower risk peels away. A key difference is community transmission – the risk of hurricane damages does not spread by contact.

***

The above covers less than half of the book. Almost everything in those pages has direct relevance to the Covid-19 world. It’s quite heartening to find out ten years later numbers indeed rule our world.

Here’s a link to get your copy of Numbers Rule Your World. There are also translated copies in Chinese (simplified and traditional), Japanese, Korean, and Portuguese, as far as I know.

Why tests are not perfect

From Andrew's blog, I learned about Tom Lumley's post that provides insight into the accuracy of the two types of Covid-19 tests. Tom is a biostatistician.

As you probably know by now, there is a diagnostic test, the most prominent of which is called the (RT-)PCR test, that involves using a swab to get sputum from the nose or throat. This material is taken to a lab which looks for the presence of SARS-CoV-2, the name for the virus that causes Covid-19 the disease. This test identifies who currently are infected.

Then, there is an antibody test, which looks for antibodies that are found in the blood of infected people, as the body fights against the virus. This test does not detect the presence of the virus; it looks for a sign that it passed through.

These two tests are not interchangeable. Some politicians on both sides of the Atlantic try to cover up the slow rollout of diagnostic testing by talking up antibody testing. In fact, early reports from antibody testing program such as the Stanford study found less than 5 percent have antibodies which means the virus is not as prevalent as claimed. (Here's why many statisticians are dismissing the Stanford study.)

It might seem like a simple issue: is the virus/antibody there or not? In both cases, if someone were not infected, their sputum or blood shouldn't have that specific virus or antibody in it. Why can't tests be perfect?

***

Let's first consider the antibody test. Antibodies are generated by our bodies naturally to fight an antigen. According to Tom, the antibody to SARS-CoV-2 is not unique - it varies from person to person slightly. This means that any test needs to look for a "range" of antibodies. To complicate matters, antibodies to similar viruses are also similar. Thus, the test will not find the exact target antibody but something similar. Then, the test has to decide whether the difference is due to different individuals or different (but similar) antigens.

One should hope that the person-to-person variability is smaller than the antigen-to-antigen variability. A test would then set up a "range" of outcomes that qualify a positive finding. If the measurement is near the edge of this range, there is a potential for error.

If the acceptance range is too "tight", the test might mistakenly decide it found antibodies for a different antigen when it's just a different individual. This produces false negative results. If acceptance is too "lax", it might mistakenly conclude that an antibody for a similar infection is an antibody for SAR-CoV-2 for a different individual. This produces false positive results.

There is a trade-off between these two errors, controlled by how the test developer sets the acceptance "range". For the antibody test, doctors don't like false positives. These are people who don't have antibodies but are told they do. As I discussed the other day, because the vast majority of people don't have antibodies, even a tiny false-positive rate will result in a lot of people being told they have antibodies when they don't.

The ability to correctly identify people without antibodies is called specificity. Some researchers are assuming antibody tests could be 100% specific. I think that is overly optimistic.

***

The PCR diagnostic test illustrates another aspect of testing accuracy. Sometimes, the error arises externally. In Chapter 4 of Numbers Rule Your World (link), I demonstrate this using steroids tests. A false-negative result arises when the lab mistakenly concludes the amount of chemicals in the doper's urine is not high enough to warrant a red flag. That's a "chemical" false negative.

But a false-negative result is really just any doper passing an anti-doping test. Someone might submit a fake urine sample, or hide from inspectors. In one case, the fake sample will not test positive, and in the other, there will not be a test. Both should be considered false negative cases even though the testing lab did not make a mistake.

Tom says "it is almost impossible to get a positive [PCR test] result without any virus". But low false-positives usually mean higher false-negative results. One reason for false negative is testing at the wrong time, or failing to swab properly. Bad timing is particularly important because "if you get tested too early, there might not be enough virus, and if you get tested too late the infection might have relocated to your chest."

Timing is also important for antibody testing because "a positive reaction [from one's body to the virus] takes time — at least a few days, maybe a week — and it stays around at least a short time after you recover."

Timing is an external factor. The consequent false negative error has nothing to do with the testing lab. Even if the test itself is perfect, there will still be people walking around with the false security of a negative test result.

So errors from testing occur because the test itself cannot be perfect (e.g. antibodies against similar infections may look like antibodies against the same virus coming from different individuals), and even when the test is close to perfect, other factors outside the lab, such as bad timing, cause errors.

***

One final comment on the politics. At various times, some officials argue that the rollout of diagnostic testing has been slow because tests are not accurate enough. Don't believe a word of that.

Let's say 5 percent of the population are infected. Borrowing some numbers from Tom, I assume that the diagnostic test has 35% false negative rate and 2% false positive rate. For every 1,000 (random) people given the test, there are 50 infected and 950 not infected. Of the 50 infected, 65% or 33 will get a positive finding and 17 will falsely believe they don't have the virus. Of the 950 not infected, 930 are told they don't have the virus correctly, with 20 told they have it when they don't. In total, 930+33 = 963 out of 1,000 will get the right answer.

So what the politicians are really saying when they reject this test is that having 17 out of 1,000 people falsely believing they don't have the virus thus spreading it is reason enough to deny 963 out of 1,000 knowing the correct answer. For every false positive, the test identifies two true positives, who can be isolated. It doesn't stem the spread completely but it does slow it down. Given that these politicians are all in for "flattening the curve", they should be clamouring for more testing.

 

 

 

 

 

 

The percent with antibodies could be 3 percent or it could be zero. That's what the Stanford study really said.

As predicted, test accuracy is getting major play in the media this week. One of the big headlines involved the claim that about 3 percent of the U.S. population may already have been infected and recovered, according to a study coming out of Stanford. The good thing about this study is its attempt to escape from the straitjacket of "triage" or targeted testing. It recruited about 3,330 people without requiring that they have symptoms. They were tested for antibodies, which is an indication that the person has recovered from a previous SARS-CoV-19 infection. Showing that a large proportion of the population have been infected and recovered is key to those pursuing the "herd immunity" theory.

The bad news is that the methodology of the study has already been debunked by Andrew Gelman. Andrew's post is here in its entirety. In this post, I give you an executive summary, especially for those not trained in the statistics of testing.

***

Before I get to the numbers, the idea that this study supports the "herd immunity" theory is far fetched. Herd immunity models typically show the vast majority of the population would be infected, much higher than five percent. We would need about 20 times the number in Santa Clara to get to herd immunity. Also, the presence of antibodies is not identical to having immunity. It is still not clear whether humans have immunity after recovery from Covid-19, and if so, for how long. The headline drawing attention to the framing as 50 to 85 times higher than reported is burying the fact that they found antibodies in only 3 percent of the samples.

If the result of this study is accepted, it exposes the sham of diagnostic testing in Santa Clara.

***

There are two major statistical issues with the Stanford study. Both are related to the construction of the sample - its composition and its size.

1. The way volunteers are recruited introduces bias into the statistics.

The 3 percent headline result is statistically adjusted. The 3,330 tests actually returned only 50 positives (1.5 percent). The adjustments were necessary because the researchers discovered biases in their sample by comparing basic demographic statistics to the county averages.

Bias means certain subgroups are under-represented in the sample. If a certain subgroup (say, men) are over-represented, then the composite subgroup (women) is under-represented so it's really two sides of the same coin.

While the sample size of 3,330 for one county is pretty large, the selection of volunteers was decidedly not random. People were recruited through Facebook ads, and they must drive themselves to the testing centers. 

A non-random selection has the chance of introducing unknown biases. In this case, some possible sources of biases are:

• people who have had difficulty getting tested (e.g. health workers) may show up in larger numbers
• some people who strictly adhere to the shelter-in-place order may not show up
• some people may like getting tested for free
• some people may believe Covid-19 is a hoax, or testing is pointless
• some people may find it more convenient to get to one of those test centers
• some people do not have Facebook
• some people do not respond to Facebook ads
• people who already have been tested, especially those who tested positive and recovered

These biases draw more or fewer infected people to the testing sites, and the strength of one versus another also matters. There is one bias that is irrefutable. Anyone who has severe symptoms or are hospitalized will not be represented at all.

It seems hard if impossible to measure and then correct for these biases, which is why statisticians usually recommend random selection. An ounce of prevention is worth a pound of cure. (A proverb that is most appropriate for this pandemic!)

It is sometimes mistakenly believed that larger samples cure sample bias. It doesn't. No matter how big the sample is, they will not find hospitalized patients in their sample. But those are almost surely infected.

***

2. The proportion of antibody-positive reported in the study is so small it may be all noise.

It turns out that even with 3,330 tests, a pretty large sample, the key result that 3 percent have antibodies could all be noise.

What is statistical noise? Imagine you repeat this experiment many times. For each set of 3,330 volunteers, you record the number of positive results. You aren't going to get 3 percent each time. The proportion with antibodies fluctuate with each sample but these variations are meaningless, an artifact caused by sampling.

The following analysis follows the plan used in Chapter 4 of Numbers Rule Your World (link) to analyze steroids testing and terrorist prediction systems. So if you have the book, you can compare the current situation with those stories.

In the raw data, the researchers found 50 positives out of 3,330 tests. (As mentioned in the first section, this 1.5 percent was revised up to 3 percent due to statistical adjustments which I won't describe here. If interested, Andrew has comments on those.)

Every test has errors. Some of the 50 positives are false positives. The false positive rate of a test describes the chance that the test will say someone has antibodies when they don't. The FP rate is one measure - but not the only one - of test accuracy.

To illustrate the issue, we consider the implausible, extreme scenario where none of the 3,330 volunteers have antibodies. So in a perfect test, the number of positives should be zero. If we get 50 positives, all 50 are false positives since no one has antibodies. That implies a false positive rate of 50/3330 = 1.5 percent.

Next, we assume that the researchers are right, that indeed 3 percent of the population has antibodies. (We further assume that the sampling is not biased.) That means the sample would be split 3% and 97%, that is, 100 people with antibodies and 3,230 without.

Let's proceed. We think that the test has a false positive rate of 1.5 percent. Applied to the 3,230 test-takers without antibodies, this antibody test should report 48 (false) positives. Note that in the actual experiment, the researchers counted 50 positives, so almost all of these are false positives!

Here is where things get confusing. Even though the test is rated as very accurate, 1.5 percent false positive rate, 98.5 percent true negative rate (also known as specificity), almost all the positives are false positives! The reason is the rarity of antibodies. The 1.5 percent false positive rate is applied on almost the entire test population because very few has antibodies. A small percentage of a large base is still a large number. (Another tie-in to the pandemic: a low fatality rate applied to a large infected base through fast community spread leads to a flood of deaths.)

***

When developing a new test, the false positive rate is not known because if we knew who has antibodies and who doesn't, we don't need to test people. We test because we don't know. In the above calculation, I used a thought experiment, the extreme case, to guess at the false positive rate.

The Stanford study had a better way of estimating it. They applied the antibody test to some blood samples collected before the Covid-19 pandemic when they're sure these should test negative if the test is accurate. Using that data, Andrew pegged the false positive rate at 0 to 2 percent.

At one end, the false positive rate may be 2 percent (instead of the 1.5 percent I used before). In that case, the test-takers without antibodies generate 65 false positives. This number is higher than the total number of positives (50) found in the experiment. So, test inaccuracy might account for all positive results!

At the other end, if the test was so accurate as to have zero false positives, then by definition, all 50 positives would be true positives, and we could run with their result. Anyone who knows a little about testing recognizes this scenario as implausible. It also implicates a high false negative rate!

Remember if the population were indeed split 3% with antibodies, 97% without as per the study's conclusion, there are 100 people with antibodies in the test sample. The test only returned 50 positives so the other 50 are false negatives.

This is the crux of understanding statistics of testing. You can't focus on just one accuracy metric. When one number goes up, another goes down. Here's the bottom line: if we think there are 100 people with antibodies in the test population of 3,330, then any reasonable test must return at least 100 positive results if it had any hope of catching all those with antibodies. That implies the test must have a positive rate (true or false positives included) of at least 100/3330 = 3 percent. To account for false positives, the test must return even more positive results. If it doesn't, then there must be false negatives.

False negatives are considered less damaging than false positives in the context of antibody testing because these people with antibodies would be taking unnecessary caution. It's not completely harmless since they might suffer from lost income. False positives are feared as they might have a false sense of security and take unknowing risks, and get infected.

Let me now relate this back to sample design. The above analysis shows that the observed test result is consistent with 0 percent of the population having antibodies. This is different from saying we believe the true proportion is 0 percent. It could be 3 percent as the researchers asserted. But the evidence is not strong enough to rule out 0 percent. In other words, even though 3,330 is a lot for one county, the sample is still not large enough to measure such a small signal.

***

I hope you're excited to learn more about testing.

See my previous post, or read Chapter 4 of Numbers Rule Your World (link).

 

 

 

 

 

 

 

 

Enter at your own peril: the statistics of diagnostic testing

 

In the days ahead, you're going to start hearing a lot of commentators try to say something smart about testing accuracy - and fail. This is a very confusing area, with multiple redundant sets of terminology used by practitioners from different backgrounds and no one willing to standardize to the same jargon. It is also an area where the drive for simplicity has doomed many explainers.

Take this explainer published by the New York Times on coronavirus diagnostic testing, written by a Yale doctor. (Tip from Antonio R.) The author made the following statement:

The good news is that the tests appear to be highly specific: If your test comes back positive, it is almost certain you have the infection.

The bad news is "specificity" and PPV are different concepts.

I am working on a bunch of posts at the moment so I will point you to my 2010 post that covers this subject for now. The two key numbers you should look for are PPV and NPV.

Another key takeaway is that the accuracy of a test cannot be reduced to a single number. To understand this material, you must be capable of holding two or three numbers in your head at the same time. I hope that is not too much to ask!

It's actually easy to mix up the terminology, even when you know your stuff. So I wouldn't read too much into the published mistake.

If you read Numbers Rule Your World (link: Chapter 4 covers steroids testing and terrorist prediction) or Numbersense (link: the Marketing chapters cover the accuarcy of targeting models), this should just be a review.

***

The insufficient accuracy of testing has become a popular excuse by governments that have continued to pursue a test-less policy. There is no perfect test. All tests face a trade-off between two types of errors. Reducing one increases the other. A serious critique of testing accuracy needs to involve actual numbers and target levels. What level of accuracy do these officials consider as adequate? And what are those numbers for the tests that they have rejected?

That's all I'll say on this issue for now, and I'll return to this subject soon. For now, here is the post that will give you everything you need to know.