Big Data, Plainly Spoken (aka Numbers Rule Your World)

The new NYC composting law

 

On April 1, New York City rolled out a composting law for residential buildings. Residents must separate compostables such as soiled paper and food remnants, and place them in special compost bins. Building owners will get fined if they don't follow the law.

Doing my part, I started putting things in a compost bag. I didn't expect the learning curve. For each piece of soiled material, I have to figure out if it's paper or plastic, or both; and whether it belongs to the compost bag or not.

Then I wondered how the inspectors decide what constitutes a violation.

Do they consider both the following offenses?

• Putting non-compostable items into the compost bag (false positives)
• Putting compostable items into common trash (false negatives)

***

This decision problem is similar to those covered in Chapter 4 of Numbers Rule Your World (link).

One consideration is the potential harm of each type of error. If having non-compostables ruins the compost process, then the cost of false positives is high. The cost of false negatives is a smaller amount of compost, which feels like a lesser harm, given the prior state of no composting.

Another consideration is prevalence. That in turn depends on what proportion of household waste is compostable.

Textbook analysis then takes these two factors and create an expected value of harm. In my book, I bring up other important considerations.

Yet another consideration is the cost of inspection (thus, the discoverability of the error). Looking for false positives involves digging into compost bags while looking for false negatives requires searching through the common trash. The latter seems far more onerous; I doubt they would spend the time to do it.

If the inspectors focus on uncovering false positives, then those errors become more visible. In my book, I illustrate this using steroids testing of elite athletes. A false negative is when a doper receives a clean test. Will we ever find out about a false negative? Is the doper going to announce that the anti-doping lab screwed up?

Because the testers don't get into trouble for missing dopers but do get a lot of negative publicity for falsely accusing an athlete of doping, one should expect that they should err on the side of minimizing false positives, which leads to more false negatives. In the case of composting, if those inspectors are more concerned about false positives, then we may get away with throwing some compostable items into the common trash.

Given these considerations, I'd err on the side of more false negatives. That is to say, I should put things into the compost bag only if I'm highly certain they are compostable.

I'm not really sure how the inspection works. Anyone who knows, please make a comment.

***

In the meantime, the city has already issued thousands of violations (link), but the penalty is laughably small ($25-100 per ticket).

 

Aging well, book-wise

2024 is a year that has brought me quite a bit of satisfaction. More precisely, world events have elevated the relevance of my two books, which is energizing. When I developed the themes of these books, I consciously sought topics that have enduring value.

The second chapter of Numbers Rule Your World (link) describes one of the grand successes of modern statistical science: the investigation of foodborne disease outbreaks. Toward the end of 2024, the U.S. has seen a spate of food recalls due to outbreaks of E.coli, salmonella, listeria, etc. (cucumbers, frozen waffles, frozen chicken, frozen cookie dough, carrots, deli meat, ...), the most famous of which was the McDonald’s E.coli outbreak of November 2024. Investigators believed that the outbreak was caused by contaminated onions served at McDonald’s stores (link). How epidemiologists make such determination is the subject of (half of) Chapter 2. The spinach outbreak featured in the chapter represents a best-case scenario, in which the investigators were able to confirm their cause-effect hypothesis by finding contaminated spinach at its source that tested positive for the exact strain of E.coli that caused the outbreak. As I wrote on this post (link), the McDonald’s investigation was not as lucky. The cost-benefit of food recalls deserves more scrutiny.

During the hurricane season, I revisited a key question of Chapter 3 of Numbers Rule Your World (link): is hurricane risk insurable? The insurance market is a game of musical chairs, in which players shift the risk around, and once in a while, the music stops, and a few get scorched. Many large insurers have already exited the scene, while small startups keep popping up, take incentives from the government, get run into the ground, and then bailed out by the government. See this blog post for more.

Numbers Rule Your World (link)’s Chapter 4, half of which concerns steroids testing in professional sports, became relevant again this year, as the top tennis player of either gender has become embroiled in controversies due to positive tests (link; link). At the time of writing, my take on steroids testing, that false negative rather than false positive is the Achilles heel of the testing programs, was controversial; later, Lance Armstrong got snitched and subsequently confessed, which laid waste to hundreds of “negative” tests over his long career (hello Holger Rune). Both tennis stars got a slap on the wrist because the governing bodies accepted their alternative explanation of contamination. Accidental contamination, and a myriad of other excuses, have always followed positive tests; it’s one of those applications that reveal the limit of statistical evidence: we can prove only the likelihood of foul play; to really nail a perpetrator, we need a confession, or physical evidence (e.g. caught with blood bags). My post about the tennis controversy is here.

***

Inflation continues to run unimpeded in New York City, and probably also where you live. The place where I used to get croissants (cornetti, to be geographically correct) now charges $7; it was $6 a couple of months ago, $5 a few years ago, and $4 when they opened about 10 years ago. (Those prices don’t include tax and tip, which adds at least 25%.) If you do the math, an increase from $4 to $7 in 10 years implies an average growth rate of 6% per year; and if you look up official statistics (link), the CPI index moved from 237 in Nov 2014 to 316 in Nov 2024, suggesting a growth rate of 3% per year. Chapter 7 of Numbersense (link) explains why the official statistic deviates so dramatically from lived experience. It demonstrates a very important key to success in data analytics: the analyst must know inside-out how numbers have come into being.

Chapter 6 of Numbersense (link) addresses the other big economic statistic, unemployment, in which, again, the official statistic seems to misrepresent reality.

Obesity drugs are all the rage, being pitched as the ultimate solution to the obesity crisis in the U.S. Chapter 2 of Numbersense (link) looked at the measurement of obesity, and treatments ranging from diets to surgery. CDC recently announced that the obesity rate of Americans has apparently plateaued, with the latest number showing a (statistically insignificant) slight decline (link). People were quick to attribute this to the rise of obesity drugs; thus, they are making the case that obesity drugs are causing a fall in average BMI, which is the standard metric used to measure obesity. I wonder how many of these people complained about using BMI to measure obesity during an earlier time when most treatments appear to be ineffective. The jury is still out on whether these drugs lead to long-term weight loss, which is one of the toughest tests of obesity treatments.

***

If you're looking for a gift this holiday season, get Numbers Rule Your World (link), get Numbersense (link), or get both!

The forbidding math of finding terrorists

New Yorkers have been traumatized by sensational stories of subway riders being pushed into the tracks, randomly stabbed, etc. Personal security is a hot topic. So, the MTA, which runs public transportation, has been initiating various efforts to enhance security. Mayor Eric Adams in particular loves technological solutions.

One such effort is hiring an AI company that claims they could detect who's carrying weapons. Evolv's technology reportedly uses scanners.

The city conducted a pilot program in 20 stations for 30 days. They have not released full results, and, after multiple requests by media outlets, only issued a four-sentence statement, so some of the following is necessarily speculative. (link)

It seems like passengers were selected and instructed to walk through the scanners, and a total of 2,749 scans triggered 130 positive signals.

The limited released data are incoherent in many ways.

The rate of scanning is surprisingly low. 2,749 scans in 30 days is about 90 scans per day on average, which means fewer than 5 scans per station per day. That makes no sense to me.

On top of that, who was responsible for picking which people to send through scanners, and how? They were definitely not scanning everyone. Let's say they do five scans a day, and since this is a small number, let's say they do all five during one hour of the day. Which five of the passengers do they stop and scan?

Ideally, they should pick five random passengers but the base rate of carrying guns in the NYC subway is probably low, so a proper test would have to scan a lot of people, much much larger than five per station per day.

The algo flagged 130 out of 2,749 scans, which is just under 5%. If we assumed all positives are correct, then at least 5% of New York subway riders carry guns, one in 20, so on average there are several (concealed) guns in each car of the subway during rush hours.

But, how many of those 130 positive signals were correct? Presumably, the police then searched those suspects, and they found ... ... ... found ... ... ... zero guns. Oops. So, the experiment yielded a positive predictive value of 0% (0/130). In plain English, a positive scan result holds zero value as an indicator of gun carry. Additionally, if any of the 2,749 people who were scanned carried guns, they were not detected. The false negative rate is 100% (all true positives were predicted to be negative.).

Wait, that's not what the city officials said in those four sentences. They disclosed that Evolv found 12 knives, so they claimed a positive predictive value of just under 10% (12/130). I guess that sounds better than 0%! I highly doubt the MTA (or any other establishment) would invest in scanners that can detect only knives but no guns.

Several reports I have seen computes the ratio 118/2,749 = 4.3% and called it the "false positive rate". This is the wrong definition of the FP rate; the denominator should be the number of true negatives, not the total number of scans. In any case, the 4.3% is not a useful metric.

I discussed the kind of math that underlies all security-related statistical detection problems in Numbers Rule Your World (Chapter 4; link); refer to that chapter for more details.

***

For this blog post, let's focus on a simple mental model for this type of problem. We start with a guess of the base rate of gun carry (Bayesians call this the prior). Let's say it's 1%.

If the base rate is 1%, and we picked 2,749 people randomly, then we should expect to find 28 guns. The first thing to realize is that if the predictive algo were perfect, it would flag exactly 28 scans as positive. In this case, the positive rate for the scanning program should be 28/2749 = 1%.

But perfection is illusory. All models make mistakes. To allow for mistakes, a realistic algo ought to flag more than 28 scans as positive, which means it must make false positive mistakes.

The Evolv model tagged 5% of the scans as positive but at most 1% of the 5% can be positive so its accuracy is upper bounded at 1/5 = 20%. If we demand a higher positive predictive value, the algo cannot spit out that many positives.

Moreover, the accuracy can dip below 20% because it might not catch all of the gun carriers (false negatives). The lower bound is 0%, as demonstrated by the MTA experiment. If we require the algo to declare fewer positives, the chance of false negatives increases.

If we were evaluating these algos, one of the most important metrics should be the false negative rate i.e. of those who were carrying guns, what proportion went through the scanners undetected?

***

What if the base rate were much higher, like 10%, or 280 gun carriers?

Because the algo only tags 5% of the scans as positives, it "gave up" on 5% of the gun carriers, so at most it could catch half of them. The other half could also be incorrect.

If the harm of a single gun is intolerable, then the algo has to flag more subway riders, annoying them. But if the goal isn't to banish all guns from the subway, then the inconvenience of getting scanned can be reduced by lowering the frequency of positive signals. With Evolv, they can't lower the frequency since they are not finding any guns.

In this Wired article (link), the reporter found that Evolv's technology when installed at schools or hospitals has also underperformed expectations.

Doping pops up in the news

Thanks to the Olympics, doping has once again popped up in the news.

Some American athletes (and journalists) harshly criticized WADA (link), the international anti-doping authority, accusing it of covering up about 20 positive drug tests of swimmers from China who were subsequently exonerated and continue to compete; the Chinese investigators submitted evidence that these athletes unknowingly ingested the banned substances at a hotel. These accusations were flying around for months prior to the Paris Olympics, and created a situation in which any Chinese medalist in swimming would be viewed suspiciously, regardless of whether the athlete failed a drug test during these Olympics.

To someone who's not familiar with the landscape or history of doping in sports, the facts looked damning. But if you know anything about this subject, or have read Chapter 4 of Numbers Rule Your World (link), you should know better. The WADA process as described by these complainers is exactly how it has always been handled, even when the accused are U.S. athletes.

The vast majority of athletes who have tested positive for performance enhancing drugs have denied doping, and used a variety of excuses to explain away the test findings. There are even official "therapeutic use exceptions" (link) that permit athletes to use banned drugs if they suffer from some conditions. TUEs should be controversial. It has been previously noted that many swimmers have been granted exceptions due to asthma, making them much, much more likely than the general population to suffer from the condition.

Eventually, the Chinese authorities struck back, pointing to the case of an American track athlete who, you guessed it, failed a drug test, and was allowed to continue to compete after the American investigators submitted evidence that he ingested the substance unknowingly from contaminated meat (link).

This post was prompted by long-time reader Antonio R. who sent me an article about Jannik Sinner, the current world No. 1 tennis player from Italy, who has recently failed drug tests, and, you guessed it, was allowed to continue to compete after the Italian investigators accepted evidence that suggested that the banned substance could have entered his body through a spray used by his physiotherapist (link).

***

As outside observers, it's impossible for us to judge who's lying and who's not. In my book, I made some observations about the anti-doping program:

1. It is far more likely that dopers are getting away with false negatives than innocent athletes who are wrongfully flagged with false positives. (Hence, the reference to "timid testers" in the title of Chapter 4.)
   
   
   
2. To come up with your own judgment of the effectiveness of antidoping, you can start by guessing what percentage of Olympics athletes might be doping. Is it 30%, 10%, 1%? Then, all you need to do is to look up what percent of tests have come back positive. When I did this exercise, the proportion of positives is way below the proportion of athletes who could be doping, which tells you that most of the dopers have been tested and tested negative.
   
   
3. The book was published before the fall of famous American cyclist Lance Armstrong, who, prior to admitting guilt, had always insisted that the hundreds of negative test results during his career represented a mountain of evidence that he was not a doper.
   
   
4. We should distinguish between a "chemical positive" and a "real positive". The chemical positive reveals that a banned substance was found to exceed the allowable level in the test. Crucially, the test finding itself cannot prove that the accused athlete doped, even if we want to believe it. Such a conclusion would require the assumption that the only way by which such substance could have entered the athlete's sample is via intentional doping.
   
   The history of athletes submitting alternative explanations for their positive tests is really very long. It ranges from the relatively boring such as eating contaminated meats to the extremely creative such as "the vanishing twin" (link).
   
   To nail down a "real positive," investigators have to locate actual evidence of doping, such as drug supplies or paraphernalia. Needless to say, it's very hard to prove such a thing. Armstrong was ultimately brought down by one of his former lieutenants.
   
   
   
5. It is equally hard for the accused athlete to prove that s/he has eaten a contaminated piece of meat. At best, they could show that meat purchased from some vendor contains the forbidden substance. But that tested meat isn't the meat that was eaten.
   
   It appears that the anti-doping agencies are quite willing to accept these explanations, and whether it's China, US, or Italy, they have been exonerating positive results based on these theories. Note that these athletes have not proven that the chemical positive was a false positive; by their response, they concede that the test was indeed positive, they just argue that it's not a "real positive".

***
This topic fascinates me as it relates to statistical thinking. There are the measures (via tests) used as indicators of hypothesized causes. Then there is the causal gap, because the indicators themselves don't prove a cause.

For further thoughts on this topic, see Chapter 4 of Numbers Rule Your World (link).

Covid vaccines contain PEDs, according to an athlete

Another sport has the doping demons. News arrived that U.S. triathlete Collin Chartier has been banned for three years after testing positive for EPO, a favorite performance-enhancing drug for endurance sports.

Doping is featured in my book Numbers Rule Your World (link), written before Lance Armstrong finally admitted to doping after vehement denials for years. I made several surprising observations. Firstly, if you or I are running an anti-doping lab, we'd calibrate the test to reduce false positives at the expense of false negatives, since a doper who is getting away with it isn't going to expose our error but a falsely accused clean athlete will surely erode our reputation. Secondly, and consequently, we expect most positives to be true positives. Thirdly, a high proportion of dopers are getting away with it.

To Chartier's credit, he did not invent excuses but admitted to the offence. As reported in the article, other triathletes have paraded out the usual unlikely excuses - one even claimed that a "tainted burrito" and a "tainted Covid-19 vaccine" were responsible for the positive tests.

Now, if we believe Chartier (and many other athletes who got caught doping), he only took EPO once and only after the two greatest wins of his career. He also claimed that no one else but him knew about the doping, and his coach, who is related to the world champion of the sport, said he was blindsided by his own athlete.

***

I'm still waiting for the day when an athlete shakes up the establishment by admitting all the negative tests s/he got while doping. That would take courage. Wait - Armstrong frequently said he was the most tested athlete on Earth while he was getting away with it. Does that count?

Reading Gelman's take on election polling

The spector of Covid-19 has pushed U.S. elections off the headlines. Nonetheless, political scientist Andrew Gelman has just offered his take on the state of election polling (link to PDF). This article is perfect for those of us who have a keen interest in but no deep knowledge about polling as Andrew covers how such polls are conducted. Andrew, by the way, was the brain behind the election forecasts by the Economist in the 2020 U.S. presidential elections so he's not merely lecturing from the pulpit.

If you've followed recent U.S. elections, polling inaccuracy has been one of the popular themes. Funny enough, before Trump's stunning victory over Hillary Clinton, the media was trumpeting the exact opposite, how election polls were so damn accurate. Those were the glory days of Nate Silver. Recently, the narrative has flip-flopped: how pollsters could have given Clinton 90-99.9% chance of winning, how polls over-estimated Biden's victory margin, etc.

The following are notes from reading Andrew's article.

***

p. 1

Pollsters are aware that "recent pre-election polling errors have been an underrepresentation of choices favored by lower-education white voters" but "adjusting for ethnicity and education... did not solve the problem... this past November".

This gets at something important: making "adjustments" is necessary but may not be sufficient to make the problem go away. This is a very common issues I see all over the place. One has an observational dataset on people. One then shoves a pile of demographic data (age, sex, income groups, etc.) into the statisticial model, now known as the adjusted model, and voila, heaven opens up and all our sorrows have gone away! If someone asks about biases in the observational dataset, one points to the adjustment factors, and moves on. An analogous situation is peer review in a published journal. Is the research finding credible? In some corners, if it is published in a peer-reviewed journal, there should be no further questions. Reality is much more complicated. Are the right variables in the adjustment? Are the variables structured in the right way? Are there confounding variables that haven't been measured?

p. 2

the moment of truth when poll-based forecasts are compared to election outcomes.

The election prediction problem is unusual in that there is an inescapable, quick, definitive evaluation of the pollster's performance. Election forecasting is the most accountable type of data science. Most data science problems do not have this property. Sometimes, only certain types of errors are visible. In Chapter 4 of Numbers Rule Your World (link), I discussed the anti-doping labs: their false negative errors are unlikely to be discovered, while false positive errors elicit immediate outcry from the harmed athletes.

The key challenges [of polling] are (a) attaining a representative sample of potential voters, and (b) predicting turnout.

The progression is from eligible voters to potential voters to actual voters. There isn't much uncertainty in eligible voters, nor actual voters once the election is over. But turnout is uncertain, which is the link between potential voters and actual ones. Eligible voters are not usually a representative sample from the underlying population.

We cannot expect to eliminate non-sampling errors, because conditions for new elections are always changing. Unlike sampling errors, non-sampling error cannot be reduced simply by conducting more and larger surveys.

The last part debunks a Big Data myth. Collecting more data can't solve all problems. It reduces sampling error but does not eliminate non-sampling errors. Another way to think of non-sampling errors is that the people in the sample aren't representative of those in the population. It is helpful to extend our progression: eligible voters -> potential voters -> surveyed voters -> survey respondents -> actual voters. There are two sources of inaccuracy here: some actual voters may not be surveyed, or if they received the survey, may not have responded.

p. 5

If we are estimating opinion in 4 age categories, 2 categories of sex, 4 ethnicity categories, 5 education levels, and 50 states, that’s 8000 cells. Even if you are analyzing a set of polls with a total of 20,000 respondents, this will still leave with many cells with 0, 1, or 2 respondents, hardly enough to get any sort of estimate.

Each "cell" is a demographic subgroup. Polls rarely have 20,000 respondents so this problem is clearly pervasive. Andrew's statistical approach pools data from "similar" cells so there are enough respondents to come up with an estimate. Such a model is, however, vulnerable to being wrong about "similarity". Nevertheless, any method will be vulnerable to one issue or another because fundamentally, the available data are incomplete. You can, for example, produce just 8 cells based on age and sex alone, and then in each cell, you have enough respondents; but this model misses the effect of all the other variables, and therefore is inaccurate in a different way.

The adjustments only adjust for the variables included in the model.

The beauty of these models is that they are not black boxes. We know exactly what factors are being adjusted for, and the functional form. But if the form is incorrect, the adjusted model will be inaccurate. This points back to the point I made above: be skeptical when someone claims to have solved all biases by throwing a bunch of standard demographic variables into a model - be doubly skeptical if they only include one factor at a time, or only "main effects" - be triply skeptical if the outcome is affected by non-demographic factors, such as behavior.

 

p. 8 He goes into the crux of the matter:

Why were the polls as imprecise as they were? Some possible explanations are differential nonresponse, differential turnout, changes in opinion, and insincere survey responses. These last two explanations, which can also be phrased as “last-minute swings” and “shy Trump voters,” are natural explanations, but I doubt they are a major part of the story in 2020.

By "shy Trump voters", Andrew specifically meant Trump voters who lied to pollsters and claimed they would vote Democratic. His view that this was a non-factor appears to be commonly held among the political science community. I wonder if there is another kind of shy voter who appears under the heading of "differential nonresponse". What if Trump voters were systematically less likely to respond to polls?

p. 9

Some polls also adjust for partisanship, using recorded party registration, stated party identification, or stated vote in the previous election. But even the surveys that made all these adjustments were off by about 2 percentage points, on average.

There is a philosophical question in there. The residual inaccuracy can be either evidence of inadequate adjustment or of partisan bias being unimportant.

p. 10

An error of 2 or 3 percentage points is a problem for predicting very close elections but otherwise is not so consequential.

This is an important observation. The closeness of recent presidential elections makes the error bar a problem.

p. 11

He turns this around and praises the unreasonable accuracy of polls. Think about it. Surveying only some thousands of people among a population of 300 milllion gets the estimates down to a margin of error of only a few percentage points.

p. 13

In retrospect I wish that we’d expressed our 2020 Economist forecast conditionally: instead of simply stating a forecast interval and probability of each candidate winning, we could’ve graphed this interval as a function of the national polling error, thus indicating the confidence we had in our forecast at different levels of survey accuracy and making this dependence clear.

Good idea. Not sure if the public will appreciate it.

well-funded campaigns and advocacy groups... can do more effective survey adjustment using the voter file, which has information including past turnout history on nearly 200 million Americans.

I wonder how much this costs. Sounds like it could be a fortune.

it would be good to see less focus on political campaigns and more on surveys of attitudes

I really like this suggestion. I don't understand the value of election forecasting (except for the campaigns themselves).

 

Why FDA says stick to the test

It pays to know some basic statistics when "experts" are putting out bad information these days. Business Insider publicized a recent study that claims that rapid Covid-19 tests may fail to catch Omicron before the infection becomes infectious unless a throat swab is used in place of / in addition to a nasal swab. The test was designed to evaluate a nasal swab, and therefore, the FDA has come out to recommend against "taking matters into your own hands".

The recommendation to use a throat swab sounds reasonable and logical but is actually flawed. It only makes sense if one ignores the statistics of testing - and with that, one is ignoring the inexactitude of testing science.

I covered a lot of this material in Chapter 4 of Numbers Rule Your World (link) using steroids testing in sports as the motivating example.

***

Imagine making a soup, and there is a bay leaf somewhere in the pot, which you want to fish out before serving. That's the wrong analogy for a steroids test or a Covid-19 test. Testing is not about fishing out a whole organism from a soup.

Realize that typical tests are not looking for the whole virus but some marker - or proxy, which may be some chemical. It's a guessing game. If the sample has an amount of that marker above some threshold, it will be marked positive.

There is tremendous variability in humans, so that the amount of that marker varies from one to another, while the test developer has to decide on one threshold that works for everyone. This just means that no test can be 100% accurate. A false positive is a sample meeting the threshold but the person is actually not infected while a false negative is a sample that comes under the threshold but the person is actually infected. Wherever the developer draws the line, some proportion of the samples will be false positives or false negatives. The stricter the threshold, the fewer samples are classified as positives, and it follows that the test produces fewer false positives, and more false negatives.

How does the test developer calibrate this threshold? It typically must find a set of definitely positive samples and another set of definitely negative samples. The former is expensive to obtain while the latter relies on swabs that were taken pre-Covid-19 and somehow preserved. Now, the lab can measure the rate of false positives and false negatives, with different thresholds. The selected threshold depends on the tolerance of false positives and false negatives.

Notice that the popular Binax rapid home test indicates that they favor having more false negatives to having more false positives. That's why they tell you a positive test is highly credible while a negative test has a chance of error.

Now that you know how tests are calibrated, you should see why the FDA says don't switch to throat swabs using the same test kits. While the conceptual principle of the test works whether you're taking throat or nasal samples, the statistics may not work! With throat samples, or mixed throat-nasal samples, we have no idea if the previously measured false positive and false negative rates apply. The test threshold was selected using the amount of markers in the nasal samples. It is likely that a different threshold would be required if the input materials are altered.

The test developers can recalibrate the tests. So let's wait.

***

The study that Business Insider cites uses a small sample (30 people) that doesn't sound like it was selected randomly, and therefore it's unclear they are representative. The report tells us, interestingly, that all 30 people have had 3 vaccine shots, and 28 out of 30 were infected with Omicron.

Does biometrics authentication work?

As your smartphone scans your face and then unlocks the device, have you ever asked how well biometrics authentication work? Does it work just as well as fingerprints? How does one go about measuring its accuracy? 

***

Biometrics started with fingerprints, expanded to face recognition, and now encompasses other types of measurements such as voiceprints.

The basic steps of biometrics authentication are: capturing the signal (image, voice, video, etc.), turning the signal into data, converting data into scores, which measure the likelihood that the biometrics data came from the device owner, and determining whether to grant access based on  exceeding a certain threshold.

As with any AI/predictive model, the software must be pre-trained using labeled datasets, e.g. images known to be those of the device owners.

The authentication software makes a binary decision (Allow/Block). Block might involve iterative Retries but I'll ignore this complication. Such a prediction system makes two types of errors: blocking someone who is the device owner (false rejection error FRR) or accepting someone who isn't the owner (false acceptance error FAR). While we've all heard anecdotes of people who've been erroneously shut out of their phones, I haven't actually seen a numeric error rate ... until now.

Reading Significance magazine recently, I came across one study that quantifies the error rates of biometrics authentication. A more detailed article by the same team is found in Communications of ACM.

We tend to think fingerprints are unique. It turns out that authenticating users with other biometrics data are much less accurate - multiple orders of magnitude less so! According to these authors, the field summarizes the two error rates with one number known as "Equal Error Rate" (EER), which is the setting under which FRR and FAR have the same values. If you've read Chapter 4 of Numbers Rule Your World (link), you'll hopefully question why FRR should equal FAR. The costs of the two types of errors are clearly different, and reflect an individual treadoff between convenience and security. (Given the lack of scrutiny of these systems, one infers that most people sacrifice security at the altar of convenience.) Note also that falsely rejecting the owner is an annoyance each and every time it happens, directly felt by the true owner, while falsely accepting an imposter may not be discovered until the owner realizes s/he has been harmed.

The researchers said that their face recognition software has EER of 4 percent, which means that 4 of 100 times the owner's face is read, the software erroneously deny entry, and 4 out of 100 times someone else requests access, their intrusion goes undetected.

Voice recognition software (voice-printing) is shown to be more than 8 times worse than face recognition: the error rates are 35% each. About a third of the time the owner requests access, the software would decide to block. (These authors are pitching a system that combines multiple sources of biometrics data.)

***

When reading reports about predictive models, we should examine how the error rates are measured.

Take the false rejection rate. One would have to present images of true owners to the phone, and then compute what proportion of these images the phone erroneously decides to be imposters. Whether the FRR is credible depends on how the investigators select the set of test images. For example, if the test images are the same images used to train the model, the chance of error is smaller.

Each face recognition system has its strengths and weaknesses. Some, for example, are fooled by glasses. People find that they have to take off their glasses to unlock their phones. If the set of test images does not include images of the true owners wearing glasses, then the FRR is under-estimated. If detecting glasses is a strength, not a weakness, of the system being evaluated, the FRR is now over-estimated.

The false acceptance rate is even harder to measure accurately because the investigators must compile a set of images of imposters. There are infinitely many ways to evade the software. The researchers of this study adopted a popular method: "We performed the testing through a randomly selected face-and-voice sample from a subject we selected randomly from among the 54 subjects in the database, leaving out the training samples." This method relies on "randomization".

Almost always, such a method of selecting test samples over-estimates the accuracy rates. That's because in real life, the bad guys are not randomly selected from the entire population - but from the subset of bad guys. The bad guys who are attempting to unlock your phones are likely to exploit weaknesses of the technology. The authors included an interesting example of one such attack strategy. Certain software assesses the quality of the images submitted for verification, and assigns higher importance to higher-quality images. This makes sense but the system is open to a form of attack: the bad guys deliberately submit poor-quality images, knowing that the software can't keep locking out true owners who provide low-quality images.

It's challenging to compile test images for measuring false acceptance rate, as it requires specifying how imposters are likely to attack the system.

The key takeaway is that error rates coming from research studies is heavily affected by the investigators' choice of test images. As modelers, we can fool ourselves by presenting easy but unrealistic images for validation. Ideally, we should employ a neutral third party to evaluate these systems.

***

This last section dives into some technical details, which you can skip if not interested.

Below is a table and a figure included in the Significance article, which provides data on the error rates:

The researchers compared three authentication schemes: face recognition, voice recognition and one that fuses both face images and voice prints (described as "feature-level multimodal fusion"). Table 1 says that the fusion scheme has the lowest EER, followed by face recognition; voice recognition has a terrible EER of 35%.

Figure 2 presents the components of the EER in the form of an ROC curve. However, the results shown in this Figure does not match what is shown in Table 1.

The ROC curve plots the "true positive rate" against the "false positive rate". According to the authors, the false positive rate is the FAR, the chance of letting an imposter through. (This definition implies that a positive result is positively identifying the true owner.) The inverse of a "true positive" is a false negative, that is to say, to mistakenly block the true owner of the device. Thus the FRR (false rejection rate) is the false negative rate, which is 1 - the true positive rate.

With ROC curves, the top left corner represents the perfect system that makes zero false positiive mistakes and 100% true positives (i.e. zero false negative mistakes). The ranking of the three authentication schemes should therefore be Blue > Red > Green, i.e. Feature/Fusion > Voice > Face. But this chart contradicts the data shown in Table 1 where the ranking is Feature/Fusion > Face >>> Voice.

The EER is "the value at which FAR and FRR are equal". The table shows a summary statistic computed from the FAR and FRR while the chart plots the two metrics separately. If the underlying values of FAR and FRR are the same, the ranking should not differ.

Let's locate the points at which FAR equals FRR on the ROC curve. Recall that the vertical axis is 1-FRR while the horizontal axis is FAR. Thus, when FAR equals FRR, y = 1-x.

As shown above, all three schemes have EERs around 20%, with the best one closer to 10%. None of these have EER under 10% and voice recognition isn't 5 times worse than face recognition. So I'm very confused by these figures. I can't find any more details about this research beyond those two articles.

A 20% error rate is a far cry from what we have come to expect from fingerprinting.

 

The unexpected gift of a well-conducted study

We're drowning in a quicksand of bad-quality data during this pandemic, and frustratingly, the situation is getting worse, as we enter the second year. It would have been a challenge to measure the real-world impact of vaccines because of the mass vaccination initiatives. We then made our lives much harder by (a) destroying the placebo groups in the original randomized clinical trials, fossilizing them at 4 months of follow-up (see this post); and (b) applying different rules of masking, testing, hospitalizing, attributing causes of death, etc. to vaccinated and unvaccinated subgroups, literally injecting massive biases into observational data.

It comes as an unexpected gift to find a study that takes on these challenges seriously. Today, I review the preprint of the REACT-1 Study from the U.K. There is a large team of people involved, many of whom from Imperial College.

***

In this study, the researchers collect swabs from randomly selected people from all over England. The study is ongoing, with monthly cohorts. The report I'm reading covers Round 13, which was conducted from mid June to mid July. This cohort is monumental because Round 13 spanned a critical point in the case curve in the UK.

In the period leading to Round 13, the U.K. government was congratulating itself on the hugely successful vaccination campaign, about to declare mission accomplished. But by July, the virus has returned with a new rigor. In Round 13, the researchers analyzed about 100K swabs, with a positivity rate of 0.6%. That is more than four times higher than the positivity rate in Round 12.

The positivity rate found in the REACT-1 Study - especially after adjustments and weighting - can be generalized to the population of England. That's because the study utilizes a random sample. The swab tests find not just symptomatic but also asymptomatic Covid-19 cases. This study gives much better information than most real-world studies - which usually tap into a database of positive test results. A severe problem with those studies is selection bias. Vaccinated people, people with mild symptoms or disease, and people who live in lower-exposure areas are less likely to get tested, so their cases are under-counted. (That said, the "good" study still has to deal with biases, which I'll get to later in this post.)

***

There are many things we can learn from the REACT-1 study. The first thing is that the chance of anyone catching the virus - regardless of vaccination status - is low. Only 6 out of 1,000 people in England were actively sick with Covid-19 during July. The chance of getting severe Covid-19, getting admitted into hospitals, or dying from Covid-19 is a fraction of that number. The danger comes from the spread of the virus, so that a large base of the infected produces a large number of deaths. (Even at its peak in England - Round 8, the prevalence was 1.6%.)

What about the 994 people out of 1,000 who did not get sick during Round 13? We are frequently misled into thinking that the fully vaccinated subset were protected by the vaccine. The fallacy is made clear when you ask about those who are unvaccinated. One can't get infected if one isn't exposed. A lot of these people - vaccinated or not - did not get exposed to the virus during the study period. Most of the calculations you see out there implicitly assume that everyone is exposed every month, month after month.

I am not saying none of those 994 people got exposed. Some proportion of those were not infected because they did not get exposed. This is a missing data problem that we haven't grappled with yet. If they do get exposed, it's likely that the vaccinated ones have better outcomes; better may mean 30-50% reduction in chance of getting sick; we now know it's not anywhere close to 90%.

***

In "explaining" the surge in cases in England in July, the study's authors pointed the finger at unvaccinated 13-17 year olds. They drew this conclusion by comparing the percentage increase in infection rate between Rounds 12 and 13 by age group. See this chart (with my edits):

The study estimates that the prevalence of Covid-19 in the 13-17 age group jumped from 0.2% to 1.6%, which is a nine-fold surge, compared to an aggregate surge of about 4 times.

Are the teenagers responsible for the entire surge? To answer this question, I looked up the age distribution of England's population (shown on the chart in the green boxes).

From this analysis, I found that 13-17 year olds accounted for 15% share of cases in this period of time. The bulk (60%) of cases affected those between 18 and 64.

The above analysis contains one hopeful insight. The 65+ age group, which enjoys the highest vaccination rates, is getting some measure of protection, as indicated by the lower share of infection. This goes a long way to explaining why deaths in England have not increased as much as infections - younger people are getting infected, and they tend to recover more readily.

(Statements about deaths must be interpreted with great caution. This study is typical: the cutoff date of Round 13 was in the middle of the surge of cases, which means the analysis covered a period of time when the deaths arising from the surge of cases have not occurred yet. The researchers would never return to update the analysis and so we are always dealing with over-optimistic, premature estimates of effectiveness against deaths.)

***

Next, I made a similar analysis breaking the study population down by vaccination status: fully vaccinated, partly vaccinated and unvaccinated.

This leads to a surprising finding that the surge in infections was even stronger among those who were vaccinated (at least one dose) than those who were unvaccinated. This problem illustrates the difficulty of observational datasets. There are so many possible explanations for this. My own favorite is that many vaccinated people have stopped taking precautions.

***

Another nice feature of the REACT-1 preprint is the careful description of how they interpreted test results. It's not a straightforward science - something I explained in Chapter 4 of Numbers Rule Your World (link) using testing for steriods and HGH as a case study. Someone has to draw a line on some continuous scale to decide who's positive and who's negative.

These UK researchers described their positivity criteria as "both E and N gene targets were detected, or if N gene was detected with CT value < 37". CT is cycle threshold: the lower the CT value, the higher the viral load.

I highly recommend reading this section (pp. 10-11) of the paper. You will learn that to figure out what positivity threshold to use, the scientists need to procure positive and negative controls (samples known with 100% certainty to be positive or negative), these controls may be synthetically created, experimenters may be blinded or unblinded, testing analysis may be done externally or internally, multiple experiments by multiple labs may be required, these results may not agree, etc. In other words, scientists don't wave their hands and write down CT < 37; there is method to the madness.

Unfortunately, there does not seem to be a global standard for what test to use and what positivity threshold to apply. This is yet another aspect of the data quality problem I mentioned up top.

***

While the selected participants represent the whole of England, the study cannot escape selection bias in the form of non-response. The researchers reported that response rates (of sending back the swabs) have declined steadily and was at 12% in Round 3. That's a very respectable number.

What's concerning is that the responders were clearly different from the non-responders. Response rates for younger people and people living in deprived neighborhoods were only in the 5% range. Those, of course, are also the least vaccinated subgroups.

As if to prove the point that biases pervade every corner of real-world data, the researchers also reported finding selection bias among those who gave them permission to link their NHS records. Ex ante, it is not obvious why there should be a difference in infection rates between those who gave consent and those who didn't.

It turns out vaccinated people who gave consent are more likely to get infected than vaccinated people who didn't give consent. The reverse is true for unvaccinated people.

(Linking is used to conduct off-book accounting as described in this post. This process disappears vaccinated, and infected people because of some arbitrary case-counting window - which cannot be applied to the unvaccinated. In the case of England, the 2D+14 window means that no cases are counted on the vaccinated side until at least 14 weeks after the first shot because of a long delay between doses, based on embarrassingly wrong analyses by advisors to the UK government. This gimmick puts our most creative corporate accountants to shame.)

***

We are drowning in bad-quality data. Don't trust any study that gets published. Be discriminating, and value those researchers who are making an honest effort at doing good analyses. It takes a lot of work to properly handle real-world data, so let's take a moment to appreciate those who tackle this task seriously.

 

 

 

 

Sizing up the pandemic

From the beginning, the U.S. has never gotten religion about accurately measuring the pandemic. In March 2020, I wrote a piece for Wired about the need for broad-based testing because selectively testing only severe cases is bound to give us a biased picture of the crisis. It was known even then that a significant portion of infections is asymptomatic - which constitutes a silent path by which the virus passes from person to person. The situation has not improved much since. Recent counting rule changes imposed selectively on vaccinated people have made it far worse but that's the subject of a different post.

I'm delighted to report on a valiant effort to correct this oversight - in Slovenia, a country of about 2 million people. The relevant papers have just been published here and here.

***

Because we want to minimize bias, randomization is our friend. I have previously explained how randomly assignment treatment in a clinical trial creates covariate balance (here) which allows researchers to assume "all else equal" except for the treatment.

In this case, we want to test a random sample of people selected from the population to avoid the self-selection bias that permeates the usual counts of "cases". The demographic and medical profiles of the sample should mimic that of the entire population. In the Slovenian study, they apply standard statistical sampling methodology, as follows: the country is divided into 300 census districts; from each district, they select 10 persons at random; a total of 3,000 Slovenians were sent communications inviting them to participate in this study; just under half agreed, leading to a sample size of 1,368.

Because of this random (i.e. broad-based) testing, the researchers uncovered not only self-reported symptomatic cases but also asymptomatic cases. All participants were given PCR tests at the start of the study in April, which was towards the end of the first wave in Slovenia. The estimated prevalence was 0.15%, or 150 per 100,000.

It's important to understand prevalence as a measure of infections at a point in time (late April-early May 2020 in this case). The PCR test reveals whether someone is currently or has recently been infected with the novel coronavirus. This metric does not answer the important epidemiological question: what proportion of the population has been infected since the start of the pandemic?

***

The research team's second paper deals with this question - the canonical approach is through serological testing i.e. testing for antibodies. Most people who have been sick with Covid-19 have developed antibodies. So we expect the proportion of population that have had Covid-19 to be larger than the proportion who are currently/recently infected. Technically, we say the seroprevalence is greater than the prevalence.

I have discussed serological testing before - at the start of the pandemic. Two research teams - one at Oxford and one at Stanford - jumped the gun when they published serological studies during the first wave of Covid-19. The Oxford team (in)famously declared that up to 40% of the U.K. population have already been infected with SAR-CoV-2 by April 2020 - a result that has proven extremely embarrassing. To be precise, they used a mathematical model to come up with the alarming headline, and called for serological studies. The Stanford team argued that total infections in California were 50 to 85 times the reported number of cases - back in April 2020. This group actually performed tests to derive their estimate. Their analysis was roundly criticized. Recent handwaving analyses tend to multiply cases by 4 times, which gives you an idea of the magnitude of the error in the Stanford study.

Meanwhile, I'm still expecting that one day, the New York State Department of Health will release the data behind the almost daily pronouncements by Governor Cuomo - again in spring and summer of 2020 - that at least 15% of New Yorkers have been infected. Those numbers were supposed to have come from antibody testing throughout the state of New York but no data or science has seen the light of day. While not as far-fetched as the two other studies, that estimate of seroprevalence is also clearly repudiated by subsequent waves of Covid-19.

In the Slovenian study, the unadjusted estimate of seroprevalence around mid-November 2020 (in the middle of a devastating second wave) was only 4 percent. At the time, the cumulative reported cases was around 27,000, or 1.3% of the population. This estimate is much more solid than the previous studies because these researchers used a random sample. The design of research studies matters a lot - when devices like randomization are utilized, fewer adjustments and assumptions are necessary to interpret the data. (See my three-part talk to learn more about research methods.)

***

Much of the second paper deals with the complications of the analysis even though the team utilized a random sample. The study interviewed participants at 3-week intervals. There were two rounds of blood samples (April 2020, and October 2020).

In April 2020, the reported cases (about 700) was 0.03% of the Slovenian population. With serological testing, they estimated seroprevalence to be roughly 0.8-0.9%, so roughly 18,000 people have been infected. More precisely, they estimated that 0.8-0.9% of their random sample have evidence of SARS-Cov-2 antibodies (that are true positives), and generalized that result to the entire population.

The raw data actually showed 3 percent, which was revised down to 0.8-0.9%. This is because of two factors that artificially inflated the unadjusted number.

The first is test inaccuracy. All tests are inaccurate to some degree - which is a key message of Chapter 4 of Numbers Rule Your World (link). In this situation, because the vast majority of Slovenians have not been infected, false-positive errors can be devastating to any analysis. Even a small false-postive error rate (e.g. 2 or 3 percent) gets amplified by the large base population into an abundance of incorrect positive test results. Correcting for test inaccuracy therefore pulls down the number of positive cases, and thus seroprevalence.

We have encountered this issue before - when pointing out the methodological weaknesses of the 2020 Stanford study. Their initial analysis did not correct for false-positive results. But it's actually a really hard problem. To know that a positive result is definitely a false positive implies that we know the true result - if we did, we would not need to run a test. In practice, the researchers must obtain an estimate of the probability of false positives from other studies. 

For example, they gather blood samples collected for other reasons - preferably from before first half of 2019 when they are sure the blood samples did not have any SARS-Cov-2 antibodies (or RNA, depending on what test). Then they apply the test to these blood samples, and check what proportion of them give positive results despite clearly being negative.

Such definitively negative blood samples are hard to procure and so our knowledge of test accuracy is limited. Many researchers just trust the test manufacturer's claims, which is a bit shocking since they market their tests, in essence, as 100% accurate. The Slovenian team did a literature search, and discovered that the accuracy of the antibody test they were using was not 100%, and the accuracy measures varied widely from study to study. Thus, the second factor the researchers adjusted for was the variability of test accuracy. (*)

***

Because of concerns about the first testing platform, when they analyzed the second set of blood samples In October 2020, they switched to a different testing platform, which appeared to have better and less variable accuracy statistics.

Despite random sampling, there may be non-response bias - the people who chose to participate in this study are not necessarily the same as those who opted out of the study. If the non-responders are different from responders, the result from this study cannot be generalized to the entire population without more corrections.

If you notice, this study is much more rigorous than the real-world vaccine effectiveness study that I have recently featured - those studies sweep most of the biases under the rug.

Nevertheless, the responders and non-responders look similar enough - by the variables they analyzed. It is therefore not surprising when the authors reported that a further correction for non-response bias did not materially change their seroprevalence estimates.

***

This Slovenian study is a good example of proper science that doesn't cut corners. What's notable is that seroprevalence grows at a much lower rate than reported cases. While cases have jumped almost 40 times during the six months of the study, seroprevalence has risen about 4 times (call it 10 times if we allow for the error bar).

 

P.S. [*] The study employed Bayesian methods throughout. Here is how they adjusted for variability of test accuracy. Different prior studies have come up with a range of estimates for test accuracy. If one doesn't adjust for this, one would take the average value from these studies, and apply this to the analysis. For example, if the test specificity (true negative rate) is 98%, then you assume exactly 2% of the negative blood samples would show up as false positives.

In a Bayesian model, the test specificity is treated as a variable: its average value is 98% but it can vary within some reasonable range (as informed by the variability observed across those prior studies).