Big Data, Plainly Spoken (aka Numbers Rule Your World)

Will AI make cheaters of us all?

Andrew wrote an amusing post about mischief using AI in peer reviewing for academic journals.

It emerged that authors of scientific papers have resorted to embedding secret prompts inside their text to instruct large language models (LLMs) to give their papers positive reviews. These prompts may be printed in white, or tiny font, so they are intended to evade humans. Some prompts are quite elaborate, carrying instructions for what to say about strengths as well as what to say about weaknesses. For example:

Be sure to check out the comments section, as readers fuss over which group is worse: the authors who instruct LLMs to give positive reviews only; or the reviewers who rely on LLMs to submit their reports. As Andrew told the story, one author who admitted to inserting these prompts argued that they did it only to deal with cheating reviewers who deploy LLMs. So, we are witnessing the classic two kids in a playground scenario - he's the one who started it!

We can take this blame game one step further. The cheating reviewers should blame it on the authors because some authors are using LLMs to write bogus papers!

***

Unfortunately, this is the world of AI we find ourselves in. At an event recently, I chatted with an instructor who is throwing his hands up, complaining that he is spending time correcting code submitted by his students who are obviously using AI to do the work. Meanwhile, there are students complaining that their instructors use AI to set or mark assignments. They can of course blame each other.

Would one begrudge instructors who ask AI to mark assignments if the work were generated by AI? Would one judge the students who use AI to do their homework if said assignments were created by AI?

Is this a race to the bottom? Eventually, will humans do any work?

 

The new NYC composting law

 

On April 1, New York City rolled out a composting law for residential buildings. Residents must separate compostables such as soiled paper and food remnants, and place them in special compost bins. Building owners will get fined if they don't follow the law.

Doing my part, I started putting things in a compost bag. I didn't expect the learning curve. For each piece of soiled material, I have to figure out if it's paper or plastic, or both; and whether it belongs to the compost bag or not.

Then I wondered how the inspectors decide what constitutes a violation.

Do they consider both the following offenses?

• Putting non-compostable items into the compost bag (false positives)
• Putting compostable items into common trash (false negatives)

***

This decision problem is similar to those covered in Chapter 4 of Numbers Rule Your World (link).

One consideration is the potential harm of each type of error. If having non-compostables ruins the compost process, then the cost of false positives is high. The cost of false negatives is a smaller amount of compost, which feels like a lesser harm, given the prior state of no composting.

Another consideration is prevalence. That in turn depends on what proportion of household waste is compostable.

Textbook analysis then takes these two factors and create an expected value of harm. In my book, I bring up other important considerations.

Yet another consideration is the cost of inspection (thus, the discoverability of the error). Looking for false positives involves digging into compost bags while looking for false negatives requires searching through the common trash. The latter seems far more onerous; I doubt they would spend the time to do it.

If the inspectors focus on uncovering false positives, then those errors become more visible. In my book, I illustrate this using steroids testing of elite athletes. A false negative is when a doper receives a clean test. Will we ever find out about a false negative? Is the doper going to announce that the anti-doping lab screwed up?

Because the testers don't get into trouble for missing dopers but do get a lot of negative publicity for falsely accusing an athlete of doping, one should expect that they should err on the side of minimizing false positives, which leads to more false negatives. In the case of composting, if those inspectors are more concerned about false positives, then we may get away with throwing some compostable items into the common trash.

Given these considerations, I'd err on the side of more false negatives. That is to say, I should put things into the compost bag only if I'm highly certain they are compostable.

I'm not really sure how the inspection works. Anyone who knows, please make a comment.

***

In the meantime, the city has already issued thousands of violations (link), but the penalty is laughably small ($25-100 per ticket).

 

Know your data 45: permanence

With the bankruptcy announcement of the once-highflying genetic testing company, 23andme, its customers are scrambling to delete their DNA data (link).

These people have little understanding of how the Internet, and especially modern, cloud-based services, work.

In most, perhaps all, cases, a user who deletes data has severed oneself from the data. The data persist at the company, in the cloud, at the company's various business partners, etc.

It is very hard to delete anything in the digital age.

***

Even before the arrival of computers with effectively infinite storage, when we delete a file from a PC, we have just severed the "pointer" to the locations on the hard drive where the file was stored. There are recovery software that allows us to retrieve the "deleted" file. The existence of such software proves that the file wasn't truly deleted.

Advanced software exists to supposedly truly delete files from computers. These would not be necessary if the files were truly deleted when users execute the "delete" function. Even these advanced software has blind spots. That's why some experts recommend physically destroying hard drives before disposal.

The cloud is a network of computers outside of the user's control. This network provides resiliency and efficiency by making copies of the data and strategically scattering them around the network. Thus, if someone wants to delete a file, one would have to find every copy of the file, and then thoroughly delete them from each computer using advanced software.

A business might use a cloud that is managed by some other entity (e.g. Amazon Web Services). Then, this business may not even know how many copies of a file have been created, and where they are stored. The business pays someone else to manage all of that, and the point is to wash their hands off of those details. 

A business like 23andme makes money by selling user's data. Once the data change hands, they are now replicated in computers that are not controlled by 23andme. Those other businesses may also store their data in the cloud, which means the files are duplicated and distributed to yet another network of computers. 23andme has no ability to verify that its customers have thoroughly removed data from all of their computers using advanced software. It probably doesn't want to either.

Even if a business doesn't base its business model on selling customer data, they may still share customer data with business "partners." For example, if a hospital hires a third party to analyze CT scans, the CT scans of its patients will find their way to the computers of that third party. That third party may also send the images to its business partners, for exactly the same reason. If a patient requests the hospital to delete a file, the hospital would have to remove all of its copies, plus ask the third party to delete all of its copies, and so on down the tree of partnerships. The chance that all copies of that file are removed from all computers of all entities involved is exactly zero.

***

The above is not merely speculation. In your everyday usage of the Internet, you may inadvertently discover that data uploaded to some app are permanent.

Many years ago, I used an online email provider to send a single email to a list of people. This requires uploading the names and emails of those people. After sending the email, I deleted the contact information and closed my account. I specifically did this in the hope of preventing the vendor from taking the private data and using it for other purposes without my knowledge. The people on the list consented to receiving an email from me, but not more than that.

After I closed my account, surprise surpise, I got lots of marketing emails imploring me to return to the service.

One such email said that if I reactivated my account that day, I'd be able to recover all the contacts in my previous account. I could see how this would be a great convenience if I indeed wanted to continue from where I stopped. 

However, it also shows that when I pushed the button to "delete" the contact list, it wasn't deleted at all! Even after I closed my account, the "deleted" contact list was still there.

***

In other words, data are permanent. It's delusional to think that going to the 23andme website and clicking on the "delete" button will remove one's DNA data from prying eyes.

Sure, doing it is better than not doing it. It's most likely a placebo - that makes one feel better but in reality, does not make a difference.

Pressing the delete button certainly detaches you from your data but there is no way to verify that your files have been permanently and thoroughly removed from all of 23andme's computers (not forgetting all the computers of employees who downloaded your files as part of some larger analyses of their database.) The DNA file would already have been sold many times over during your time as a customer, and will live on forever in many computers not owned by or accessible to 23andme. It would also have been shared with business partners who provided relevant services to 23andme. The most valuable asset that 23andme could sell during its bankruptcy proceedings is the DNA database, so you can bet that the leadership team has made sure that the data are transferred to the eventual buyer.

And remember, DNA data are itself immutable. It belongs to the class of data (like date of birth, social security number) that only needs to be stolen once. (See this previous post.)

The only effective way to ensure your DNA data don't fall into the wrong hands is to not have them stored at 23andme in the first place, i.e. don't be their customer.

 

 

The dangerous story-first data analysts

The spectacle of Elon Musk and his proteges seizing the people’s data has brought uncomfortable truths about data collection to the fore. This day could hardly have arrived sooner. The people must now confront the perils of entrusting our data to corporations run by smooth-talking profit-maximizers.

According to media reports, the Musk team targeted specific federal agencies, demanding access to vast amounts of data (link, link), and in short order, fired boatloads of government employees, claimed to have found fraud everywhere, cancelled contracts, etc. They make announcements on social media and put up "receipts" on a hastily put together website. One such tweet was “Maybe Twilight is real and there are a lot of vampires collecting Social Security,” with an accompanying frequency table showing ages up to 369.

These circumstances do not inspire confidence for producing high-quality data analyses.

1. The team is made up of “story-first” analysts. From the name of the department, and every word they have since published, it's evident that these data analysts have already written their headlines before they even touched a single row of data. They even set a target bounty of $2 trillion of "savings" before they began their work. Is there any chance that this group finds any data that show some parts of the federal government is run efficiently?
2. Ironically, when all evidence points in the same direction, it's panic time for data analysts, not party time. If the Musk team weren't "story-first", by now we should have learned at least one finding that debunks an alleged source of fraud or waste. Wake me up when they make such a discovery. "Story-first" analysts behave differently from "data-first" analysts. They pursue different questions, and thus harness different methods. As a trivial example, consider the widely-publicized claim of fraudulent social security payouts to dead people (link). If that's the answer, what is the question? The appropriate workflow for this analysis starts with defining dead people, for example, as those whose age is above 120; extracting a list of such people; removing duplicates; and computing the total amount of payouts to them. By contrast, if the analyst wants to estimate the extent of fraud, she won't get anywhere by honing in on just one segment of alleged fraudsters.
   
   
3. "Story-first" analysts are at the gravest risk of being fooled by confirmation bias, i.e. they skillfully find evidence for whatever it is they want to find. In the trivial example introduced above, they fail to inquire why there are "dead" people in the database with positive payouts (other than malfeasance). I don't have access to the data so I can only speculate. If I found dead people in the data, I'd seek answers to many questions before making accusations on social media. Are those payouts final? Could they have been caught by a downstream system? Are the birth dates of those people accurately entered? Are fake birth dates used as placeholders for missing data? Is that field the right one to compute age from, or is it an obsolete field? Any data analyst run the risk of self-delusion but the danger is amplified by a "story-first" mindset. Story-first promotes cherry-picking evidence, discarding or disregarding anything that disagrees with the story, as well as running down paths that lead to that story.
   
   
4. "Story-first" analysts are also at great risk of being careless. At every stage of the analysis, they may fail to exercise self-doubt. Within days, the Musk team listed as a top catch on their website a saving of $8 billion by cancelling a project. They didn't bother to check that the maximum contract value of the cancelled project was $8 million (link). Of course, they also failed to pro-rate that number by the amount that has already been paid out for work that has already been delivered and accepted. And surely, they wouldn't think to check the contract to see if it mandates a breakup fee. Hard to imagine they had included any litigation expenses and if appropriate, settlement fees, that would apply when the other party sues the government for breach of contract.
   
   
5. By many accounts, the Musk team has obtained access to data that are not anonymized. This creates the opportunity for malfeasance. The blackmail happening at Columbia University, in which the government rescinded certain funding to coerce compliance with a list of demands (link), can happen to any entity that receives federal funding, not necessarily in the public's eye. This is what I mean when I said "The people must now confront the perils of entrusting our data to corporations run by smooth-talking profit-maximizers." If we let corporations amass our data, the data may land in the laps of our enemies. Our enemies may not be known at the time of data collection; they may reveal themselves in the future. Witness how fast the likes of Mark Zuckerberg switch allegiance. Recall Musk was once allied with the Democrats.
   
   
6. "Story-first" analysts who are aware of their cherry-picking often resort to hiding the data. It's always a red flag when analysts refuse to release the data. The Musk team has not released any data that can be independently vetted. Pharmaceuticals also play this game of control. These entities frequently lean on the privacy excuse, but in fact, there are ways to provide the data without violating people's privacy; and if these entities truly cared about privacy, they would have stopped much of their privacy-busting marketing activities. Not releasing the datasets has serious consequences: not only is there no way to independently verify the conclusions, we aren't even sure if the findings were completely made up.
   
   
7. The speed at which the Musk team is moving precludes the possibility of deeply understanding the data they're looking at, or gathering the background information required to interpret them properly. Take for instance the possibility that the 150-year-old people in the Social Security database may have missing age information. Missing age may have been coded as some very large number. The media has tried to blame this on the Musk team not being familiar with old programming languages. That's nonsense. The missing age problem also affects new computer code; it is a problem that exists in every database that collects age information; and the value chosen to represent missing age depends on the designer of the database. To be sure what this value is, the team would need to find someone who can look up the source code. Well, they probably fired most of the people who have this knowledge.
   
   
8. Don't be distracted by "read" vs. "write" access (link). Read access is sufficient for the previously mentioned blackmail. Write access - which means the ability to edit the database - merely opens up more avenues for inflicting harm (e.g. by planting fake data or deleting inconvenient data).
   
   
9. Don't celebrate bad analyses even if you agree with the mission of the Musk team. Everything I described so far can ricochet, should the roles be reversed. Further, you may be an ally today; you could become their enemy tomorrow.
   
   
10. Don't believe the "we haven't done it" defense. If something is technically feasible, it will likely be done. Many tech companies have been caught with their pants down. The large-language model developers claim they did not use stolen copyrighted materials to train their models; ask any insider if they believe these words (link).

Here's the situation we're facing. A gaggle of story-first data analysts have seized the people's data. They are pursuing a pre-conceived agenda of uncovering and uprooting "wasteful spending" in the federal government. Laser-focused on this one idea, they publish cherry-picked evidence, everything pointing to the same conclusion. Their analyses show no sign of self-doubt, and exhibit amateurish carelessness. They are also shielded from third-party scrutiny, hiding the data. It is hard to take this group seriously but these conditions allow them to do serious harm, intentionally or not.

 

P.S. [4-1-2025] On point #8, they were just joking. In the latest news (link), Musk claimed to have "corrected" Social Security records. This proves that they have "write" access to the database. This development also proves point #10. "If something is technically feasible, it will likely be done." Gaining all access to databases is technically feasible, and now it's done.

 

Self-checkout keeps growing despite theft

Finally, I came across some hard statistics about self-checkout, published by Capital One Shopping Research (link).

The report has two main findings:

• Self-checkout technology contributes to the worsening theft problem in U.S. retail
• Companies worldwide are continuing to expand the self-checkout footprint

***

An estimated 20 million Americans have admitted to shoplifting via self-checkout kiosks. We presume this is an underestimate since some shoplifters probably didn't want to admit a crime to a pollster.

Apparently 20 percent of the self-checkout thefts are described as "accidental" (i.e., the shopper fails to notice when an item does not scan). If we assume this information was obtained through response to a poll question, then the shopper did at some point notice the item did not scan and then chose to not pay for the item.

Further, 44 percent intended to steal again using self-checkout kiosks.

Shockingly, the value of stolen items for each shopping trip is $60 on average. That's a lot of groceries. If one item didn't scan, it may have been an accident; if many items didn't scan, I'll let you decide.

The most predictable finding: it is much easier to steal from self-checkout kiosks than from human cashiers. Also, the amount of "shrinkage" from self-checkout is much higher than from traditional checkout.

***

One wonders why companies continue to push in this direction. They obviously save labor costs by not having to pay wages, and associated benefits.

Could it be that the saved labor cost pays for the increased stealing? Or do the stores have to raise prices on everyone to recoup what's stolen?

Let's pretend to be a mom-and-pop corner store manager.

According to the above research, each shoplifter using self-checkout steals $60. The actual loss depends on whether those items would have been sold to paying customers or not. If other customers would have paid for those items, then the store owner lost $60. If the items would not have been sold, and could have been returned to the manufacturer, then the store owner lost $60 minus the profit margin. Let's say the store typically makes $10 profit from selling the $60 worth of goods, then the store would lose $50 when the goods were stolen.

As the profit margin at a corner store is probably even lower, I'll just use $60 loss per shoplifter to make the math simpler. How does this compare to the savings on wages?

Let's say the store pays a cashier $30 per hour (including benefits, etc.).

If there are more than one shoplifter every two hours, then the loss due to theft exceeds the pay of a single cashier. The report says 15 percent of shoppers are stealing via self-checkout, so if the store has more than 7 shoppers during those two hours, the threshold has been breached! The foot traffic at this store had better be over 3 people per hour.

In this calculation, we aren't even taking into account the cost of the self-checkout machines, and any maintenance fees.

Is this another reason for inflation at the grocery store? Are these stores recouping losses by raising prices on every shopper? Why are they continuing to expand self-checkout? That's a head scratcher.

Know your data 41: collateral damage of data exhaust

In a previous post, I referred to a report that revealed Manhattan's congestion pricing program faced a toll evasion crisis right after launch - the non-compliant drivers didn't wait even one week.

What happened after the first week? New York Post (link) disclosed that law enforcement has already started issuing tickets. Toll evasion must have been quite rampant!

The journalist revealed that NYPD wrote over 100 tickets during a six-hour crackdown on toll evasion.

***

That's not the point of interest for this post. To me, the most informative line is this one:

In addition to the summonses, 10 vehicles were seized during the operation — eight of which had suspended registrations, and two that were impounded for other reasons. One individual was arrested at the scene, though authorities clarified the bust was for a warrant unrelated to congestion pricing.

I call this the collateral damage of data exhaust.

Because of this congestion pricing scheme, every single car entering lower Manhattan is identified. They have installed 1,400 cameras. These cameras can identify also every driver and passenger in every car - although I haven't heard whether that is being done.

This situation on the roads is wild wild West compared to what we face on the Web or on our phones. At least there we have some laws, albeit limited, concerning PII (Personally Identificable Information). This whole congestion pricing setup clearly involves PII but there doesn't seem to be any protection against abuse.

What those sentences tell us is that the PII is not just used for the tolling application for which data are being collected; the PII is then taken elsewhere, used to look people up in databases unrelated to congestion pricing, which leads to arrests and seizures of property. In the world of data science, such appropriation of data collected for one purpose to another is confusingly called "data exhaust".

In effect, every single camera is a cop who can identify someone without even asking for ID, and who has 24/7 access to all kinds of databases.

Know your data 40: omnipresent listening

Apple markets itself as the tech giant that is most protective of user privacy. One reason to believe this claim is that unlike all the other tech companies, Apple charges premium prices for its products and is not as reliant on advertising as its revenue source. Nothing in this post disputes this assertion although we will wonder how low the bar is to quality as "most".

Apple recently settled out of court a lawsuit that alleges that Apple's Siri secretly records people's conversations, then extracts "insights" which are sold to advertising customers. It's a scenario that most of us recognize is possible, and perhaps many of us thought couldn't possibly be happening, not on an Apple device.

The allegation is that some users were delivered "eerily accurate targeted ads that appeared after they had just been talking about specific items like Air Jordans or brands like Olive Garden".

***

Of course, any interactive technology requires the tech to capture the user's instructions or reactions and thus we expect the tech to be recording, whether it's audio or video. This is true of Siri, other voice assistants, any home devices you can interact with, gaming and TV monitors, etc.

So, if one chooses to use these features, one should expect omnipresent listening.

Just think about the idea that "Hey, Siri" is used to activate Siri. If the phone is not constantly listening to you, how does it pick up "Hey, Siri"?

The concern is really about the lack of user control, or the disrespect of user intention (what has been euphemistically called "unintentional" here). I laugh every time the word "unintentional" is used. There is almost nothing that is unintentional in tech. The recording has to be triggered by some instruction and that instruction is written up in the code, a deliberate act. It's possible that the recording is set to be universal regardless of user intention or controls but that's not what is being admitted.

The allegation is that Siri was supposed to record conversations only if the user utters the words "Hey Siri" but in fact, conversations were (sometimes) recorded even when those words were not spoken.

Further, one doesn't typically hand over entire recordings to advertisers. The recordings must be turned into transcripts, then insights are extracted (e.g. person A mentioned brand X), the person A has to be identified so that the advertiser can send ads to that person's devices, etc.

The amount of code needed to execute the entire targeted advertising program, starting with recording the conversations and ending with delivering the ads, is substantial and involved. In my view, if such code exists, it is intentional.

***

It just so happens that while I was writing this up, a friend called me up and told me that his Instagram (that's a Facebook property) started flooding him with ads from a particular VPN brand after he recently asked me for VPN recommendations. He was quite alarmed and annoyed. He said he only wrote down the recommended names on a notepad so the only way Instagram knows we talked about that vendor would be through listening to our phone call.

That scenario is certainly possible but hard to prove since there is so little transparency when it comes to technology. In fact, any number of trackers may be collecting the data, which eventually find their way to Instagram.

Without whistleblowers or discovery, it's challenging to prove the case against Apple in court. But since Apple is settling, we might not hear about this again.

 

 

On the second day of congestion pricing...

Here comes the most predictable story of the AI revolution in toll collection. Drivers are evading tolls, stupid! (link)

Since Sunday, New York City has imposed tolls for driving into midtown and downtown Manhattan, which can potentially reduce the volume of traffic, and thus pollution in the city. They call it “congestion pricing,” which is a misnomer, since the price is not zero when there is zero congestion (the toll is $2.25 for passenger cars after 9 pm).

In any case, "congestion pricing" is powered by omnipresent surveillance, as AI image recognition technology is used to recognize car license plates, and then via a database lookup to establish their identities, the car owners are sent bills. You need lots of cameras all over the city to make this possible. According to this report (link), they installed 1,400 cameras in 110 “detection points”.

It used to be that cars lined up to pass by toll booths where each driver must hand over cash to the human toll collector before the car was allowed to pass through. This means drivers paid first, then used the roads. Then came AI image recognition technology, which snaps photos of cars, extracting the license plate numbers, and via a database lookup, sends bills to people’s homes. As a result, drivers get to use the toll roads first, and pay later.

It doesn’t take a genius to figure out that one can’t get charged if the AI failed to discover one’s license plate number. One can use any number of ways to obscure the license plate, or not even show it. One can fake the plate numbers, or simply cover it up. It appears that the toll evasion problem is so serious that on the second day of congestion pricing, NYC officials have proposed new rules to crack down on such tactics.

The next thing they will discover is more revenue leakage – what proportion of plate numbers are incorrectly read, what proportion of bills are sent to the wrong addresses, and what proportion of people don’t pay the bills.

***

On a related matter, my own experience is that banks are trusting the handwriting recognition software used by ATMs to read checks less and less; I feel like more and more, I'm asked to enter the amount manually, which suggests that the software is not sure about the number, or that the banks are extra cautious. 

Was Debrina Kawam homeless?

The New York Times published a short entry on the life of one Debrina Kawam, the woman burned alive while sleeping in a New York subway car recently (link).

In the initial reporting, the victim was described as "homeless" but eventually, when she was identified, the police used language that suggests otherwise: they said she had a "brief stint" in the New York homeless shelter system. This CNN article even said:

The victim was initially believed to have been homeless, which complicated efforts to identify her, law enforcement sources previously told CNN.

The NYT reporter added details, saying:

In the fall of last year, Kawam was homeless. After an outreach team encountered her at Grand Central Terminal, she entered the New York City shelter system and was assigned to a facility in the Bronx. But she never showed up.

***

Since I have been digging into how the U.S. HUD (Department of Housing and Urban Development) counts the homeless population (here, here), Kawan's case presents an opportunity to put our knowledge to practice.

If she had been alive in late January, she would have been found by the homeless counting team to have been sleeping in the subway system, which means she is part of the "unsheltered" homeless. If she had registered with the assigned Bronx shelter, she would have been counted as "sheltered homeless."

Thus, the reporting that suggests she was not homeless at the time of her death appears inaccurate.

 

P.S. Another tidbit from the NYT article was her futile attempt to find her mother in New Jersey. If her mother had allowed her to stay even temporarily, Kawam would not be included in the homeless count (neither in New York nor in NJ) as this situation fits one of the exclusion criteria set out in the HUD Guide.

The forbidding math of finding terrorists

New Yorkers have been traumatized by sensational stories of subway riders being pushed into the tracks, randomly stabbed, etc. Personal security is a hot topic. So, the MTA, which runs public transportation, has been initiating various efforts to enhance security. Mayor Eric Adams in particular loves technological solutions.

One such effort is hiring an AI company that claims they could detect who's carrying weapons. Evolv's technology reportedly uses scanners.

The city conducted a pilot program in 20 stations for 30 days. They have not released full results, and, after multiple requests by media outlets, only issued a four-sentence statement, so some of the following is necessarily speculative. (link)

It seems like passengers were selected and instructed to walk through the scanners, and a total of 2,749 scans triggered 130 positive signals.

The limited released data are incoherent in many ways.

The rate of scanning is surprisingly low. 2,749 scans in 30 days is about 90 scans per day on average, which means fewer than 5 scans per station per day. That makes no sense to me.

On top of that, who was responsible for picking which people to send through scanners, and how? They were definitely not scanning everyone. Let's say they do five scans a day, and since this is a small number, let's say they do all five during one hour of the day. Which five of the passengers do they stop and scan?

Ideally, they should pick five random passengers but the base rate of carrying guns in the NYC subway is probably low, so a proper test would have to scan a lot of people, much much larger than five per station per day.

The algo flagged 130 out of 2,749 scans, which is just under 5%. If we assumed all positives are correct, then at least 5% of New York subway riders carry guns, one in 20, so on average there are several (concealed) guns in each car of the subway during rush hours.

But, how many of those 130 positive signals were correct? Presumably, the police then searched those suspects, and they found ... ... ... found ... ... ... zero guns. Oops. So, the experiment yielded a positive predictive value of 0% (0/130). In plain English, a positive scan result holds zero value as an indicator of gun carry. Additionally, if any of the 2,749 people who were scanned carried guns, they were not detected. The false negative rate is 100% (all true positives were predicted to be negative.).

Wait, that's not what the city officials said in those four sentences. They disclosed that Evolv found 12 knives, so they claimed a positive predictive value of just under 10% (12/130). I guess that sounds better than 0%! I highly doubt the MTA (or any other establishment) would invest in scanners that can detect only knives but no guns.

Several reports I have seen computes the ratio 118/2,749 = 4.3% and called it the "false positive rate". This is the wrong definition of the FP rate; the denominator should be the number of true negatives, not the total number of scans. In any case, the 4.3% is not a useful metric.

I discussed the kind of math that underlies all security-related statistical detection problems in Numbers Rule Your World (Chapter 4; link); refer to that chapter for more details.

***

For this blog post, let's focus on a simple mental model for this type of problem. We start with a guess of the base rate of gun carry (Bayesians call this the prior). Let's say it's 1%.

If the base rate is 1%, and we picked 2,749 people randomly, then we should expect to find 28 guns. The first thing to realize is that if the predictive algo were perfect, it would flag exactly 28 scans as positive. In this case, the positive rate for the scanning program should be 28/2749 = 1%.

But perfection is illusory. All models make mistakes. To allow for mistakes, a realistic algo ought to flag more than 28 scans as positive, which means it must make false positive mistakes.

The Evolv model tagged 5% of the scans as positive but at most 1% of the 5% can be positive so its accuracy is upper bounded at 1/5 = 20%. If we demand a higher positive predictive value, the algo cannot spit out that many positives.

Moreover, the accuracy can dip below 20% because it might not catch all of the gun carriers (false negatives). The lower bound is 0%, as demonstrated by the MTA experiment. If we require the algo to declare fewer positives, the chance of false negatives increases.

If we were evaluating these algos, one of the most important metrics should be the false negative rate i.e. of those who were carrying guns, what proportion went through the scanners undetected?

***

What if the base rate were much higher, like 10%, or 280 gun carriers?

Because the algo only tags 5% of the scans as positives, it "gave up" on 5% of the gun carriers, so at most it could catch half of them. The other half could also be incorrect.

If the harm of a single gun is intolerable, then the algo has to flag more subway riders, annoying them. But if the goal isn't to banish all guns from the subway, then the inconvenience of getting scanned can be reduced by lowering the frequency of positive signals. With Evolv, they can't lower the frequency since they are not finding any guns.

In this Wired article (link), the reporter found that Evolv's technology when installed at schools or hospitals has also underperformed expectations.