Big Data, Plainly Spoken (aka Numbers Rule Your World)

Does biometrics authentication work?

As your smartphone scans your face and then unlocks the device, have you ever asked how well biometrics authentication work? Does it work just as well as fingerprints? How does one go about measuring its accuracy? 

***

Biometrics started with fingerprints, expanded to face recognition, and now encompasses other types of measurements such as voiceprints.

The basic steps of biometrics authentication are: capturing the signal (image, voice, video, etc.), turning the signal into data, converting data into scores, which measure the likelihood that the biometrics data came from the device owner, and determining whether to grant access based on  exceeding a certain threshold.

As with any AI/predictive model, the software must be pre-trained using labeled datasets, e.g. images known to be those of the device owners.

The authentication software makes a binary decision (Allow/Block). Block might involve iterative Retries but I'll ignore this complication. Such a prediction system makes two types of errors: blocking someone who is the device owner (false rejection error FRR) or accepting someone who isn't the owner (false acceptance error FAR). While we've all heard anecdotes of people who've been erroneously shut out of their phones, I haven't actually seen a numeric error rate ... until now.

Reading Significance magazine recently, I came across one study that quantifies the error rates of biometrics authentication. A more detailed article by the same team is found in Communications of ACM.

We tend to think fingerprints are unique. It turns out that authenticating users with other biometrics data are much less accurate - multiple orders of magnitude less so! According to these authors, the field summarizes the two error rates with one number known as "Equal Error Rate" (EER), which is the setting under which FRR and FAR have the same values. If you've read Chapter 4 of Numbers Rule Your World (link), you'll hopefully question why FRR should equal FAR. The costs of the two types of errors are clearly different, and reflect an individual treadoff between convenience and security. (Given the lack of scrutiny of these systems, one infers that most people sacrifice security at the altar of convenience.) Note also that falsely rejecting the owner is an annoyance each and every time it happens, directly felt by the true owner, while falsely accepting an imposter may not be discovered until the owner realizes s/he has been harmed.

The researchers said that their face recognition software has EER of 4 percent, which means that 4 of 100 times the owner's face is read, the software erroneously deny entry, and 4 out of 100 times someone else requests access, their intrusion goes undetected.

Voice recognition software (voice-printing) is shown to be more than 8 times worse than face recognition: the error rates are 35% each. About a third of the time the owner requests access, the software would decide to block. (These authors are pitching a system that combines multiple sources of biometrics data.)

***

When reading reports about predictive models, we should examine how the error rates are measured.

Take the false rejection rate. One would have to present images of true owners to the phone, and then compute what proportion of these images the phone erroneously decides to be imposters. Whether the FRR is credible depends on how the investigators select the set of test images. For example, if the test images are the same images used to train the model, the chance of error is smaller.

Each face recognition system has its strengths and weaknesses. Some, for example, are fooled by glasses. People find that they have to take off their glasses to unlock their phones. If the set of test images does not include images of the true owners wearing glasses, then the FRR is under-estimated. If detecting glasses is a strength, not a weakness, of the system being evaluated, the FRR is now over-estimated.

The false acceptance rate is even harder to measure accurately because the investigators must compile a set of images of imposters. There are infinitely many ways to evade the software. The researchers of this study adopted a popular method: "We performed the testing through a randomly selected face-and-voice sample from a subject we selected randomly from among the 54 subjects in the database, leaving out the training samples." This method relies on "randomization".

Almost always, such a method of selecting test samples over-estimates the accuracy rates. That's because in real life, the bad guys are not randomly selected from the entire population - but from the subset of bad guys. The bad guys who are attempting to unlock your phones are likely to exploit weaknesses of the technology. The authors included an interesting example of one such attack strategy. Certain software assesses the quality of the images submitted for verification, and assigns higher importance to higher-quality images. This makes sense but the system is open to a form of attack: the bad guys deliberately submit poor-quality images, knowing that the software can't keep locking out true owners who provide low-quality images.

It's challenging to compile test images for measuring false acceptance rate, as it requires specifying how imposters are likely to attack the system.

The key takeaway is that error rates coming from research studies is heavily affected by the investigators' choice of test images. As modelers, we can fool ourselves by presenting easy but unrealistic images for validation. Ideally, we should employ a neutral third party to evaluate these systems.

***

This last section dives into some technical details, which you can skip if not interested.

Below is a table and a figure included in the Significance article, which provides data on the error rates:

The researchers compared three authentication schemes: face recognition, voice recognition and one that fuses both face images and voice prints (described as "feature-level multimodal fusion"). Table 1 says that the fusion scheme has the lowest EER, followed by face recognition; voice recognition has a terrible EER of 35%.

Figure 2 presents the components of the EER in the form of an ROC curve. However, the results shown in this Figure does not match what is shown in Table 1.

The ROC curve plots the "true positive rate" against the "false positive rate". According to the authors, the false positive rate is the FAR, the chance of letting an imposter through. (This definition implies that a positive result is positively identifying the true owner.) The inverse of a "true positive" is a false negative, that is to say, to mistakenly block the true owner of the device. Thus the FRR (false rejection rate) is the false negative rate, which is 1 - the true positive rate.

With ROC curves, the top left corner represents the perfect system that makes zero false positiive mistakes and 100% true positives (i.e. zero false negative mistakes). The ranking of the three authentication schemes should therefore be Blue > Red > Green, i.e. Feature/Fusion > Voice > Face. But this chart contradicts the data shown in Table 1 where the ranking is Feature/Fusion > Face >>> Voice.

The EER is "the value at which FAR and FRR are equal". The table shows a summary statistic computed from the FAR and FRR while the chart plots the two metrics separately. If the underlying values of FAR and FRR are the same, the ranking should not differ.

Let's locate the points at which FAR equals FRR on the ROC curve. Recall that the vertical axis is 1-FRR while the horizontal axis is FAR. Thus, when FAR equals FRR, y = 1-x.

As shown above, all three schemes have EERs around 20%, with the best one closer to 10%. None of these have EER under 10% and voice recognition isn't 5 times worse than face recognition. So I'm very confused by these figures. I can't find any more details about this research beyond those two articles.

A 20% error rate is a far cry from what we have come to expect from fingerprinting.

 

Do vaccines have halo effects?

Today, I read an unusual clinical trial that tests a theory called "heterologous immunity". This theory postulates that humans have a type of general immunity that is not specific to the antigen. Therefore, administering existing non-Covid19-specific vaccines may give a kind of halo effect that offers protection against Covid-19. The benefits of such an approach is that we'll be able to use proven vaccines that have better safety profiles, are widely available, are cheaper for less developed nations, and have more solid years or decades of research behind them.

The study was conducted at the Federal University of Santa Catarina in Brazil. About 400 health workers at the University Hospital enrolled in the double-blind, placebo-controlled clinical trial. The vaccine arm got two shots of the MMR vaccine 8 weeks apart; the placebo group received two shots of saline solution on the same schedule.

The result is fascinating: the MMR vaccine had efficacy of 10%, that is to say, the infection rate of the vaccine arm was 90% that of the placebo arm. The VE of 10% is not statistically different from 0%.

If you've been reading this blog, you'd know that such a VE number means nothing unless you know exactly how they count cases. I'm happy to report that the above number is completely devoid of any kind of tricks. They are counting all infections, based on positive PCR test results, regardless of symptoms. They are counting every case from first shot, thus no case counting window. There is no secretive adjudication committee to inject subjective criteria into case counting.

What is fascinating is that about half of the cases in the vaccine arm were asymptomatic, compared to 20% in the placebo arm. In other words, the MMR vaccine does not prevent infection but among those infected, the health workers were more than twice as likely to experience no symptoms. You have heard this story before - that sounds like what they are saying about the mRNA vaccines.

So, the MMR vaccine - if deployed against Covid-19 - would be a prototypical "leaky" vaccine. The mRNA vaccine also appears to be leaky but not to this extent. Whether this is a bad thing depends on whether asymptomatic people have high enough viral loads to transmit the disease. What's certain is that - if they are infectious - they are even more likely to be spreading the disease since they won't know they are infected. The study did not measure viral loads, so this question is left unanswered.

***

The researchers now compute the outcome measures we have come to expect from these vaccine trials. They reported a 50% VE for symptomatic, PCR-positive Covid-19, and a 78% VE for "Covid-19 requiring treatment". 

These alternative VE measures each involve discounting some of the cases in the earlier number, typically resulting in higher VE values. To get from the 10% VE to 50% VE for symptomatic Covid-19, about 50% of the cases are removed from the vaccine arm, and 20% from the placebo arm. Then, going to 50% VE to 78% VE involves removing mild cases.

It's frustrating that the research community refuses to standardize key definitions. The so-called Covid-19 requiring treatment sounds like hospitalizations but it's more than that to this Brazilian team. To count as requiring treatment, the participant must have a positive PCR test, respiratory or clinical symptoms, and treatment such as "anti-coagulation, corticosteroid and antibiotic therapy or hospitalization".

Statistics has become collateral damage during this pandemic. I find yet another proof in this study. These researchers overlook the huge margins of error associated with the above estimates: the margin of error on the symptomatic VE is 31% to 78%, and that on the "requiring treatment" VE is - gasp - from 6% to 82%. When ignoring these, they are acting just like the average vaccine researcher.

***

These researchers - to their credit - did not apply a 2D+14 day case counting window. Doing so removes even more cases from the count, resulting in an even higher VE. If they have added the case-counting window, plus subjective adjudication, it's likely that the MMR vaccine would become competitive with the lower end of Covid19-specific vaccines.

That can be interpreted as evidence of "heterologous immunity" or "grade inflation". By grade inflation, I mean, the metric may have be defined in such a way that it sets a low bar.... just like in many colleges in the U.S., it's much harder to not get an A than to get an A.

***

Let's contemplate that a bit more. In the Brazilian trial, the average rate of infection was  25% (remember this is about the same across both arms). The median follow-up time was 5 months (unclear whether this is counted from first dose or second). Roughly speaking, the Pfizer study reported cumulative case rate in the placebo arm of 6-7% about 5 months counting from 1st dose. Why such a huge gap?

I have already mentioned two factors contributing to the wide divergence in case rates. The Brazilian number counts asymptomatic cases while the Pfizer number excluded cases by adjudication.

The authors believe that the high infection rate is due to high exposure at the hospital. If true, it provides a little information on something we have so far been blinded to: how many of the uninfected participants in any trial have been exposed to the virus, and how many simply have not gotten exposed during the study period?

The higher exposure factor is balanced by the favorable demographics: the Brazilian participants are generally young (less than 10% over 50 years old), and healthy (most have no co-morbidities). 

***

On the topic of comorbidities, the table excerpted below includes some intriguing data: (remember the allocation is 1.5 vaccine to 1 placebo)

The two co-morbidities that exhibit the highest infection rates (almost 80%) - hypertension and diabetes - are skewed towards the placebo participants while the MMR vaccine arm has more smokers, who surprisingly show a 12% infection rate, half that of the average in the trial. None of these observations are statistically significant because of small sample size. What the table shows is that in small samples, randomization does not guarantee balance on all variables.

***

The result from this study makes me wonder whether such vaccines can be used as booster shots.

 

The unexpected gift of a well-conducted study

We're drowning in a quicksand of bad-quality data during this pandemic, and frustratingly, the situation is getting worse, as we enter the second year. It would have been a challenge to measure the real-world impact of vaccines because of the mass vaccination initiatives. We then made our lives much harder by (a) destroying the placebo groups in the original randomized clinical trials, fossilizing them at 4 months of follow-up (see this post); and (b) applying different rules of masking, testing, hospitalizing, attributing causes of death, etc. to vaccinated and unvaccinated subgroups, literally injecting massive biases into observational data.

It comes as an unexpected gift to find a study that takes on these challenges seriously. Today, I review the preprint of the REACT-1 Study from the U.K. There is a large team of people involved, many of whom from Imperial College.

***

In this study, the researchers collect swabs from randomly selected people from all over England. The study is ongoing, with monthly cohorts. The report I'm reading covers Round 13, which was conducted from mid June to mid July. This cohort is monumental because Round 13 spanned a critical point in the case curve in the UK.

In the period leading to Round 13, the U.K. government was congratulating itself on the hugely successful vaccination campaign, about to declare mission accomplished. But by July, the virus has returned with a new rigor. In Round 13, the researchers analyzed about 100K swabs, with a positivity rate of 0.6%. That is more than four times higher than the positivity rate in Round 12.

The positivity rate found in the REACT-1 Study - especially after adjustments and weighting - can be generalized to the population of England. That's because the study utilizes a random sample. The swab tests find not just symptomatic but also asymptomatic Covid-19 cases. This study gives much better information than most real-world studies - which usually tap into a database of positive test results. A severe problem with those studies is selection bias. Vaccinated people, people with mild symptoms or disease, and people who live in lower-exposure areas are less likely to get tested, so their cases are under-counted. (That said, the "good" study still has to deal with biases, which I'll get to later in this post.)

***

There are many things we can learn from the REACT-1 study. The first thing is that the chance of anyone catching the virus - regardless of vaccination status - is low. Only 6 out of 1,000 people in England were actively sick with Covid-19 during July. The chance of getting severe Covid-19, getting admitted into hospitals, or dying from Covid-19 is a fraction of that number. The danger comes from the spread of the virus, so that a large base of the infected produces a large number of deaths. (Even at its peak in England - Round 8, the prevalence was 1.6%.)

What about the 994 people out of 1,000 who did not get sick during Round 13? We are frequently misled into thinking that the fully vaccinated subset were protected by the vaccine. The fallacy is made clear when you ask about those who are unvaccinated. One can't get infected if one isn't exposed. A lot of these people - vaccinated or not - did not get exposed to the virus during the study period. Most of the calculations you see out there implicitly assume that everyone is exposed every month, month after month.

I am not saying none of those 994 people got exposed. Some proportion of those were not infected because they did not get exposed. This is a missing data problem that we haven't grappled with yet. If they do get exposed, it's likely that the vaccinated ones have better outcomes; better may mean 30-50% reduction in chance of getting sick; we now know it's not anywhere close to 90%.

***

In "explaining" the surge in cases in England in July, the study's authors pointed the finger at unvaccinated 13-17 year olds. They drew this conclusion by comparing the percentage increase in infection rate between Rounds 12 and 13 by age group. See this chart (with my edits):

The study estimates that the prevalence of Covid-19 in the 13-17 age group jumped from 0.2% to 1.6%, which is a nine-fold surge, compared to an aggregate surge of about 4 times.

Are the teenagers responsible for the entire surge? To answer this question, I looked up the age distribution of England's population (shown on the chart in the green boxes).

From this analysis, I found that 13-17 year olds accounted for 15% share of cases in this period of time. The bulk (60%) of cases affected those between 18 and 64.

The above analysis contains one hopeful insight. The 65+ age group, which enjoys the highest vaccination rates, is getting some measure of protection, as indicated by the lower share of infection. This goes a long way to explaining why deaths in England have not increased as much as infections - younger people are getting infected, and they tend to recover more readily.

(Statements about deaths must be interpreted with great caution. This study is typical: the cutoff date of Round 13 was in the middle of the surge of cases, which means the analysis covered a period of time when the deaths arising from the surge of cases have not occurred yet. The researchers would never return to update the analysis and so we are always dealing with over-optimistic, premature estimates of effectiveness against deaths.)

***

Next, I made a similar analysis breaking the study population down by vaccination status: fully vaccinated, partly vaccinated and unvaccinated.

This leads to a surprising finding that the surge in infections was even stronger among those who were vaccinated (at least one dose) than those who were unvaccinated. This problem illustrates the difficulty of observational datasets. There are so many possible explanations for this. My own favorite is that many vaccinated people have stopped taking precautions.

***

Another nice feature of the REACT-1 preprint is the careful description of how they interpreted test results. It's not a straightforward science - something I explained in Chapter 4 of Numbers Rule Your World (link) using testing for steriods and HGH as a case study. Someone has to draw a line on some continuous scale to decide who's positive and who's negative.

These UK researchers described their positivity criteria as "both E and N gene targets were detected, or if N gene was detected with CT value < 37". CT is cycle threshold: the lower the CT value, the higher the viral load.

I highly recommend reading this section (pp. 10-11) of the paper. You will learn that to figure out what positivity threshold to use, the scientists need to procure positive and negative controls (samples known with 100% certainty to be positive or negative), these controls may be synthetically created, experimenters may be blinded or unblinded, testing analysis may be done externally or internally, multiple experiments by multiple labs may be required, these results may not agree, etc. In other words, scientists don't wave their hands and write down CT < 37; there is method to the madness.

Unfortunately, there does not seem to be a global standard for what test to use and what positivity threshold to apply. This is yet another aspect of the data quality problem I mentioned up top.

***

While the selected participants represent the whole of England, the study cannot escape selection bias in the form of non-response. The researchers reported that response rates (of sending back the swabs) have declined steadily and was at 12% in Round 3. That's a very respectable number.

What's concerning is that the responders were clearly different from the non-responders. Response rates for younger people and people living in deprived neighborhoods were only in the 5% range. Those, of course, are also the least vaccinated subgroups.

As if to prove the point that biases pervade every corner of real-world data, the researchers also reported finding selection bias among those who gave them permission to link their NHS records. Ex ante, it is not obvious why there should be a difference in infection rates between those who gave consent and those who didn't.

It turns out vaccinated people who gave consent are more likely to get infected than vaccinated people who didn't give consent. The reverse is true for unvaccinated people.

(Linking is used to conduct off-book accounting as described in this post. This process disappears vaccinated, and infected people because of some arbitrary case-counting window - which cannot be applied to the unvaccinated. In the case of England, the 2D+14 window means that no cases are counted on the vaccinated side until at least 14 weeks after the first shot because of a long delay between doses, based on embarrassingly wrong analyses by advisors to the UK government. This gimmick puts our most creative corporate accountants to shame.)

***

We are drowning in bad-quality data. Don't trust any study that gets published. Be discriminating, and value those researchers who are making an honest effort at doing good analyses. It takes a lot of work to properly handle real-world data, so let's take a moment to appreciate those who tackle this task seriously.

 

 

 

 

The children

A quarter of cases reported in the U.S. are now in children

This is what the data are showing. The talking heads tells us there is nothing to think about - it's simply because children are not vaccinated.

But there is plenty to think about:

1) Children return to school. Especially where kids are not vaccinated, schools may be testing kids regularly. As our politicians made famous, more testing leads to more cases.

2) As testing ramps up among children, the number of tests conducted in vaccinated adults has dropped because public health officials discourage vaccinated people with mild symptoms from getting tested. Vaccinated people with mild disease are inadvertently hurting our fight against Covid-19 in two ways: many are spreading the virus, and many are not getting tested, thus biasing our datasets. I'd love to see testing statistics split by vaccinated vs unvaccinated. Have anyone seen such data?

This previous post explains why it is very hard to interpret case trends when seleciton bias permeates the dataset. I present a framework to think about these issues.

3) It's also weird that we should worry about cases in children since for months, the same talking heads have said these vaccines have never been designed to prevent mild cases (a claim that is debunked just by opening the first page of any of the clinical trial protocols, or perhaps finding the videos from 2020 in which the same talking heads can be found extolling 95% efficacy which was computed using cases.). As is well known, the risk of severe disease and deaths in children is tiny so if we should be alarmed by cases in children, then we must be worried about spread. If we are worried about spread by infected children, why are we not worried about spread by infected, vaccinated adults?

4) A quick trip to the Census reveals that about a quarter of the U.S. population are under 20.

5) There are many possible reasons why children is an increasing share of cases. Many of these reasons are artifacts of selection biases that pollute all of our very poor-quality datasets. These are artifacts - decoys that confuse us. We will continue to make very bad decisions if we continue to take biased datasets and gloss over their flaws.

 

 

 

 

 

 

 

Primer on regression adjustments 3

This is the third post in a series on using regression to correct for biases in observational data.

Let's review what we've learned so far. In the first post, I gave an example that shows regression will correct bias in a biased dataset. We looked at a sample of height data, in which women are over-represented relative to the underlying population. Without adjustment, the sample average height under-estimates the population average since women on average are shorter than men. After the regression adjustment, the gender-adjusted sample average height is almost identical to the population average.

In the second post, I expose a hidden assumption in the so-called regression-adjusted estimates, that everything is evenly distributed. In a gender adjustment (with only males and females in the population), the assumption is 50% males, 50% females. In an age-group adjustment (with 5 levels), the assumption is 20% in each level. If age is entered as a continuous variable, the age-adjusted estimate concerns persons of average age in the sample.

Needless to say, most real-world populations are not evenly balanced, nor do they contain only average persons. Therefore, if the goal is to estimate the population value, the regression adjustment may only correct the bias partially, and may even backfire, moving the sample average further from the population average. In other words, don't assume that dumping data into a regression software is a magic wand that cures all biases. And don't trust "experts" who wave this magic wand.

***

Chapter 3 of Numbers Rule Your World (link) is titled "the dilemma of being together." The chapter explores a fundamental problem in statistical analysis: the tension between capturing complex relationships in the data and finding generalizable features. The analyst resolves this question by deciding how much to aggregate the data. In regression models, this relates to how many averages do we include in the equation.

In the most naive model, the analyst summarizes the entire sample in one number, the sample average, and asserts that it is the best estimate of the population average. Thus:

[Naive] Population Average Height = Sample Average Height

Recall the statistics of our sample B of 900 people:

The naive model simply states:

[Naive] Population Average Height = 66.1

The average height of Population B is 67.1 inches, and the error of 1 inch is massive, equal to the width of a 95% confidence interval, as explained in post #1. This model makes systematic mistakes: the errors for males tend to be negative while the errors for females tend to be positive. A good model is one in which errors are as if random.

The underlying problem is that this naive model is too simple. On average, males are taller than females. So we err by summarizing everyone to one average.

To deal with this issue, we add complexity to the regression model. The so-called gender adjustment adds a gender term:

[Gender-stratified] Population Average Height = 66.6 + 2.6 if Male - 2.6 if Female

This is the regression output if I throw the data into software. It's the same equation I showed in post #2. People then take the constant term (66.6) and call that the "gender-adjusted" sample average height.

Note that the above equation is equvalent to saying:

[Gender-stratified 2] Population Average Height for Males = 69.2 while Population Average Height for Females = 64.0.

The reason for calling this "gender-stratified" should now be clear. In fact, this is a better representation of the model because the value of the constant term is not unique.

A mathematically equivalent formulation of the above model is:

[Gender-stratified 2b] Population Average Height = 69.2 - 5.2 if Female

This little wrinkle catches many software users by surprise. Depending on how the software developer codes the data, the regression outputs are different, even though they represent the same model. If you don't believe it, check that the new formula is also identical to:

[Gender-stratified 2] Population Average Height for Males = 69.2 while Population Average Height for Females = 64.0.

***

The standard way of thinking about gender adjustment is that it makes gender irrelevant. A gender-adjusted estimate removes the effect of gender, allowing us to focus on the outcome of interest.

A different - in my view, better - way of thinking about gender adjustment is that it acknowledges gender's effect on our outcome of interest. The regression adjustment involves building gender into the structure of the model.

The gender-stratified model finds that males and females have different heights: for males, we estimate, an average of 69.2 inches while for females, we estimate an average of 64.0 inches. By contrast, the naive model treats everyone as average, and we estimate the average height to be 66.1.

Gender adjustment does not prove that gender is unimportant. It's the opposite: building gender into the model means gender cannot be ignored. If gender does not matter, then we expect the gender-stratified model to reduce to the naive model. In other words, the coefficient of the gender term is zero (statistically speaking).

[Gender-stratified 3: Gender not important] Population Average Height for Males = 66.1 while Population Average Height for Females = 66.1.

And, formulated with a gender term:

[Gender-stratified 3: Gender not important] Population Average Height = 66.1 + 0 if Male - 0 if Female

So the average women in this special place is as tall as the average men.

What about the gender-adjusted sample average height of 66.6? This is the number that researchers in certain fields would have extracted from the stratified models -- recently, they have featured prominently in studies of vaccine effectiveness, including those by CDC scientists.

Mechanistically, these researchers have built gender-stratified regression models, then they ignore the gender coefficient, and report the constant term as gender-adjusted. As explained in post #2, this procedure is equivalent to averaging the gender-level height estimates, placing equal weights on male and female.

In statistics textbooks, we routinely warn readers that the coefficients of a regression model ought to be interpreted holistically. Should this rule apply to vaccine studies? If not, why not?

I'll pick this question up in the next part of this series.

***

If you missed the previous posts, you can still catch up on post #1 and post #2.