Big Data, Plainly Spoken (aka Numbers Rule Your World)

Is this data science professor a victim or a villain?

Apparently, some secretive entity is trying to maximally embarrass Forbes magazine as much as possible. One by one, they are exposing people on Forbes' 30 under 30 or 40 under 40 list of young entrepreneurs as fraudsters. Elizabeth Holmes has already been found guilty of fraud in her medical tech startup. Sam Bankman-Fried is being sued for running a type of Ponzi scheme in his crypto startup. And now, Charlie Javice is being sued by JPMorgan for faking customer lists in her college financial aid app startup (named Frank).

According to the Wall Street Journal (link with paywall), an unnamed data science professor played a role in this fraud, and that definitely caught my attention.

The unnamed professor, who was paid $18,000, used computer-generated information to show Frank’s purported user base, including details like names, birthdays and schools they attended, the suit alleged...“Our plan was to sample first name and last name independently and then ensure none of the sampled names are real,” the suit alleges the professor wrote to Ms. Javice.

This sounds very bad. Apparently, JPMorgan alleged that about 90% of the data were faked.

However, I don't think we can read from this excerpt whether the professor has also been duped, or the professor was part of the con. It could be the case that Javice contracted the professor to generate simulated data, supposedly for some legitimate purpose.

The other reason this case strikes a chord with me is that many of the emerging data science/AI technologies can easily be exploited to deceive and hurt people, and there does not appear to be much serious discussion of this problem.

 

P.S. [1/19/2023] My friend Mark P. sent me a note saying that some people on Twitter who has read the lawsuit learned that Excel is featured in it. Apparently, JP Morgan got suspicious when they received an Excel spreadsheet with the customer data, which cut off at ~1 million rows, which is the row limit for Excel files.

I have to say I don't really understand this point because I'd have thought that their IT people didn't realize that 4 million names don't fit into an Excel spreadsheet; I wouldn't have immediately assumed they had fewer customers than they claimed.

Beneath the surface of recent Google, Elon, FCC news

Several recent news items caught my eye.

First, there's been endless drama coming out of Twitter since Elon Musk acquired it for an ungodly sum. One of his latest bombshells is to promise an email drop that "makes public all internal discussions about the decision to censor @NYPost's story on Hunter Biden's laptop" in order to "restore public trust" (link). It's just one example of how corporate executives can weaponize data.

Second, Wired reported that Google handed over the personally identifiable information (PII) and detailed location data of at least 1,500 people to FBI investigators (link), through "geofence warrants", which is a dragnet. Geofencing is a favorite technique also of marketers for finding targets to deliver their marketing messages: the marketer draws an area on the map, and the data providers hand over contact information of every cellphone that is known to have entered the geofenced area. In this case, the FBI requested information on anyone who entered the area around the Capitol with their cellphones on January 6, 2020.

Third, the FCC blocks several leading Chinese technology manufacturers from the U.S. market (link) in the name of protecting national security. This action is taken based on the "possibility" of "espionage, IP theft, blackail, foreign influence campaigns, and other nefarious activities".

***

These stories are all related to our data. What happens to our data. Who has our data. What can people do with our data. What have they done with our data.

Americans have allowed corporations to be custodians of our private data. This was originally cast as a grand bargain, in which corporations get our data for free while they offer us some service of value for free. Think back to the beginnings of Google and Facebook... scratch that, not the humble beginnings when the entrepreneurs lured us in by offering their services for free without a quid quo pro. It was after these companies went public when they realized they must make money, and eventually, every one of these outfits has become a hoarder of private data, which are sold to the highest bidders. Every new digital product now is conceived as a way to augment the profiles of the users. This includes paid services (e.g. Amazon and any number of subscription services) which also collect the same types of data, and trade them in an obscure market... without the grand bargain.

***

Many - most, perhaps - of us have decided to trust Google, Twitter and similar companies with our data. Some of us prefer to trust our corporations rather than our governments. But can we really trust corporations to act in our best interests?

What is our best interest? It depends. If you're one of those people who entered the Capitol on January 6, your best interest is for Google not to send your location data to the FBI. If you happen to be around the Capitol on January 6 but not involved in the election politics, you don't want investigators to be probing you just because you were in the neighborhood.

If you are one of the anti-government protestors in China, Iran, Russia, etc., your best interest is for Google not to send your location data to the investigators. Now, is it any different if you're protesting against a Western democratic government?

Closer to home, if you happen to live in one of those states that have banned abortion, and also encourage your neighbors to report on you to authorities, your best interest is for Google not to sell your location data to the highest bidder. If your neighbors are pro-life activists, their best interest is for Google to release geofenced location data around known abortion clinics so you can fulfil your civic duty to report on people and doctors who skirted the abortion ban.

Do we really believe that corporations are making these decisions based on our individual desires, or based on what their lawyers think is the best course of action for the corporate interest?

If the government entity is collecting our data, there are constitutional rights that govern what this entity can do. There are Freedom of Information Act (FOIA) provisions that citizens can use to discover if laws have been broken. There are no FOIA requirements that regulate corporate behavior. There is no punishment for companies issuing false statements when not under oath.

***

Elon Musk's behavior provides a glimpse at the answer. His threat to release a load of internal emails relating to Twitter's moderation process is a direct threat to weaponize data. He's not the first corporate executive to do this sort of thing: remember how a Uber executive casually told a reporter that she's being tracked during a time when another Uber executive threatened to dig dirt on journalists who wrote critical articles about Uber (link).

If Twitter will release internal emails to make a political point, it certainly has the means to release user data if an executive finds a reason to do so. This is selective, targeted disclosure of potentially private data. Musk is not releasing all emails about all banned accounts, but emails about one particular political figure that happens to belong to a political party he is on the record of opposing.

There is no way to verify that all emails related to that one situation would be released. No one outside of Twitter has access to this data. If there are any emails that could place Twitter in legal jeopardy, what's the chance that the corporate lawyers would allow them to be released? Would these lawyers play government and redact the documents?

Remember, besides the messages that Twitter users post publicly, there is a whole hidden universe of direct messages which are private conversations between people. Activists in authoritarian countries are known to use platforms like Twitter to hide communications from prying eyes.

Musk's announcement is much more than about Hunter Biden's laptop. It is an admission that private messages are not really private. They are in the hands of the companies which own these platforms. It only takes one CEO or executive with a grudge to weaponize someone's data. (Junior staff can also wage their own small-scale vendettas as many people have access to data as part of their work.)

***

The banning of Chinese tech companies is based on the possibility that the Chinese government may get access to private data from Chinese companies, and the possibility that such data would be used to harm people.

In Twitter's story, the CEO articulated a motive and a plan to selectively release data. He just hasn't pushed the button yet.

By contrast, Google pushed the button when they transferred data to the FBI. It's not anonymized data, either. It's personally identifiable information plus detailed maps of these people's locations. (While the Wired article singled out Google, it's likely that other big tech companies, like Microsoft and Apple, and telecoms companies, may have done something similar.)

 

P.S. [12/4/2022] In the evening of the day I put up this post and the next day, Elon Musk pushed the button and released the data so we now have two instances in which prominent U.S. corporations released data with the express purpose of harming the people who generated the data, plus an allegation that Chinese companies may do similar things.

 

 

Data flows behind "secure" debit cards

The tech industry constantly promises "security". Is it possible for anything to be secure?

Here's something to ponder.

One of my debit cards recently got scammed. I am not sure what the scammer was after because the fraudulent transactions came in two sets, each set consisting of the identical transaction repeated 8 to 10 times on the same day for the same amount from the same merchant name. In other words, it would not take a genius to recognize these transactions as fraudulent.

What is interesting are the following facts:

• the account was less than a year old
• the debit card has never left my wallet except for when I used it at ATMs
• the only ATMs I ever used were the ones belonging to the bank that issued the debit card
• I have never presented this card to any merchant
• I have never used this card at any merchant (self checkout, etc.)

Remember, just a few years ago, we were promised that the new chip cards are so secure they will end the fraud problem. And so on for every new tech going back decades.

in order for these charges to appear on my card, presumably the scammer has stolen my card number, my address, the various codes on the card, etc. The bank's system must have approved those transactions without flagging them as fraudulent.

How did the scammer know those numbers? Why didn't the bank's fraud detection software reject those transactions? Where are the holes in the system?

 

Great data journalism should have destroyed Djokovic's story but it didn't

If you follow sports, you could not avoid the Novak Djokovic saga at the Australian Open, which is scheduled to start this week. In brief, Australia, having pursued a zero Covid policy for most of the pandemic, only allows vaccinated visitors to enter. Djokovic, who's the world #1 male tennis player, is also a prominent anti-vaxxer. Much earlier in the pandemic, he infamously organized a tennis tournament, which had to be aborted when several players, including himself, caught Covid-19 (link). He is still unvaccinated, and yet he was allowed into Australia to play the Open. People are upset. Some players who got themselves vaccinated in order to play in the tournament are not cool. Spectators who also must be vaccinated in order to watch the matches in person are not amused.

When the public learned that Djokovic received a special exemption, the Australian government decided to cancel his visa. Djokovic's camp, however, proceeded to fight his case in court. This then became messier and messier, as the superstar told his side of the story. His parents, his fans, and the Serbian government aggressively supported the player. [Djokovic lost in court for the second time this Sunday, and was deported and no longer could play in the tournament.]

In the midst of it all, some enterprising data journalists uncovered tantalizing clues that demonstrate that Djokovic's story used to obtain the exemption is full of holes. It's a great example of the sleuthing work that data analysts undertake to understand the data.

***

A central plank of the tennis player's story is that he tested positive for Covid-19 on December 16. This test result provided grounds for an exemption from vaccination, although the Australian government tightened entry requirements due to the Omicron surge. The timing of the test result was convenient, raising the question of whether it was faked. Intriguingly, Djokovic attended a children's event the day after he said he tested positive, and also gave an in-person interview to a French reporter two days after. His team maintained that the test was authentic, and offered evolving explanations and apologies for his not isolating after testing positive.

Digital breadcrumbs caught up with Djokovic. As everyone should know by now, every email receipt, every online transaction, every time you use a mobile app, you are leaving a long trail for investigators. It turns out that test results from Serbia include a QR code. QR code is nothing but a fancy bar code. It's not an encrypted message that can only be opened by authorized people. Since Djokovic's lawyers submitted the test result in court documents, data journalists from the German newspaper Spiegel, partnering with a consultancy Zerforschung, scanned the QR code, and landed on the Serbian government's webpage that informs citizens of their test results.

The information displayed on screen was limited and not very informative. It just showed the test result was positive (or negative), and a confirmation code. What caught the journalists' eyes was that during the investigation, they scanned the QR code multiple times, and saw Djokovic's test result flip-flop. At 1 pm, on December 10, the test was shown as negative (!) but about an hour later, it appeared as positive. That's the first red flag.

Since statistical sleuthing inevitably involves guesswork, we typically want multiple red flags before we sound the alarm.

The next item of interest is the confirmation code which consists of two numbers separated by a dash. The investigators were able to show that the first number is a serial number. This is an index number used by databases to keep track of the millions of test results. In many systems, this is just a running count. If it is a running count, data sleuths can learn some things from it. (This is why even so-called metadata can reveal more than you think. Djokovic may have become the latest victim.)

Djokovic's supposedly positive test result on December 16 has serial number 7371999. If someone else's test has a smaller number, we can surmise that the person took the test prior to Dec 16, 1 pm. Similarly, if someone took a test after Dec 16, 1 pm, it should have an serial number larger than 7371999. There's more. The gap between two serial numbers provides information about the duration between the two tests. Further, this type of index is hard to manipulate. If you want to fake a test in the past, there is no index number available for insertion if the count increments by one for each new test! (One can of course insert a fake test right now before the next real test result arrives.)

The researchers compared the gaps in these serial numbers and the official tally of tests conducted within a time window, and felt satisifed that the first part of the confirmation code is an index that effectively counts the number of tests conducted in Serbia. Why is this important?

It turns out that Djokovic's lawyers submitted another test result to prove that he has recovered. The negative test result was supposedly conducted on December 22. What's odd is that this test result has a smaller serial number than the initial positive test result, suggesting that the first (positive) test may have come after the second (negative) test. That's red flag #2!

To get to this point, the detectives performed some delicious work. The landing page from the QR code does not actually include a time stamp, which would be a huge blocker to any of the investigation. But... digital breadcrumbs.

While human beings don't need index numbers, machines almost always do. The URL of the landing page actually contains a disguised date. For the December 22 test result, the date was shown as 1640187792. Engineers will immediately recognize this as a "Unix date". A simple decoder returns a human-readable date: December 22, 16:43:12 CET 2021. So this second test was indeed performed on the day the lawyers had presented to the court.

Dates are also a type of index, which can only increment. Surprisingly, the Unix date on the earlier positive test translates to December 26, 13:21:20 CET 2021. If our interpretation of the date values is correct, then the positive test appeared 4 days after the negative test in the system. That's red flag #3.

To build confidence that they interpreted dates correctly, the investigators examined the two possible intervals: December 16 and 22 (Djokovic's lawyers), and December 22 and 26 (apparent online data). Remember the jump in serial numbers in each period should correspond to the number of tests performed during that period. It turned out that the Dec 22-26 time frame fits the data better than Dec 16-22!

***

The stuff of this project is fun - if you're into data analysis. The analysts offer quite strong evidence that there may be something smelly about the test results, and they have a working theory about how the tests were faked.

That said, statistics do not nail fraudsters. We can show plausibility or even high probability but we cannot use statistics alone to rule out any outliers. Typically, statistical evidence needs physical evidence. That's one of the key takeaways in Chapter 5 of Numbers Rule Your World (link).

***

Some of the reaction to the Spiegel article demonstrates what happens with suggestive data that nonetheless are not infallible.

Some aspects of the story were immediately confirmed by Serbians who have taken Covid-19 tests. The first part of the confirmation number appears to change with each test, and the more recent serial number is larger than the older ones. The second part of the confirmation number, we learned, is a kind of person ID, as it does not vary between successive test results.

One part of the story did not hold up. The date found on the landing page URL does not seem to be the date of the test, but the date on which someone requests a PDF download of the result. This behavior can easily be verified by anyone who has test results in the system.

Because of this one misinterpretation, the data journalists seemed to have lost a portion of readers, who now consider the entire data investigation debunked. Unfortunately, this reaction is typical. It's even natural in some circles. It's related to the use of "counterexamples" to invalidate hypotheses. Since someone found the one thing that isn't consistent with the hypothesis, the entire argument is thought to have collapsed.

However, this type of reasoning should be avoided in statistics, which is not like pure mathematics. One counterexample does not spell doom to a statistical argument. A counterexample may well be an outlier. The preponderance of evidence may still point in the same direction. Remember there were multiple red flags. Misinterpreting the dates does not invalidate the other red flags. In fact, the new interpretation of the dates cannot explain the jumbled serial numbers, which do not vary by the requested PDFs.

***

Statistical investigations can be very powerful, and have gained strength in the Big Data era due to digital breadcrumbs. Nevertheless, statistical arguments suggest plausibility or probability, never certainty. Short of a confession or whistle-blowing or leaking, those who are inclined to disbelieve can always find reasons to disbelieve. Similarly, interested investigators can easily fool themselves.

 

 

Better late than never

Quick quiz. Who wrote this?

I did not test the data for irregularities, which after this painful lesson, I will start doing regularly.

Of course, you won't know. But which of the following people is the most likely to have said such a thing?

A) A Stat 101 student

B) A data scientist working in industry during the first year of employment

C) A graduate student research assistant

D) An assistant professor in the publish-or-perish game

E) A senior tenured professor with multiple best-selling books and tens of thousands of citations

***

If you think the least likely is the most likely, then you'd be right. The answer is (E).

The person who said the sentence (in 2021) is Dan Ariely, currently a Duke professor of economics but probably known to my readers as author of a series of best-sellers in behavioral economics, starting with Predictably Irrational. (You can find the quotation in this PDF.)

Andrew has documented a series of scandals in which several of his seminal studies have been called into question. The latest one involves data obtained from an insurance company by Ariely. He now claims to have no role in collecting, and processing the data.

Neither did the people who uncovered this potential research fraud. Hence, the ability to suss out data problems does not require first-hand involvement in collecting or processing data.

To read about how the fraud was detected, see here.

***

Next time you hear about a publication in a "peer-reviewed" scholarly journal, think about this case. It is entirely possible to publish data analysis in a peer-reviewed journal without "testing the data for irregularities". In fact, researchers with hundreds of peer-reviewed publications may have never once tested data for irregularities! Better late than never, right?

 

P.S. (1) If you need something light for the holidays, here is an advice column Andrew, hmm, wrote for the WSJ.

(2) One of Ariely's best-sellers is "The Honest Truth about Dishonesty". He is an expert on honesty.

 

Instant Classic for Science Communications Students

When I blogged about Pfizer's adolescent trial (here), I was wondering when and if they will release the data or the scientific report to support those press releases (one by Pfizer, and one by the FDA). A report has just been published in the prestigious New England Journal of Medicine.

I predict this will become a staple of future science communications textbooks.

The 12-15 year old age group presents extreme difficulties for conducting clinical trial research - the case rate is expected to be extremely low, and there is a strong desire to fast track approvals. Those two issues are incompatible. You need a huge sample size to detect low rates of events, let alone differences between low rates of events. From the start, the FDA has allowed a head fake: the primary endpoint of the adolescent trials will be immunogenecity results, i.e. test of antibody levels in the vaccinated. This is no ordinary trial, as we will not be comparing adolescents to adolescents, vaccinated against placebo. We will be comparing adolescents to 16-25 year olds, vaccinated against vaccinated.

So what does the highly respected NEJM say about the 16-25 year olds, the designated "control" group? I searched for every mention of 16-25 year olds in the report. Below are some representative examples of the matching sentences (Scroll to the very end of the post to see the entire list of mentions).

• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.

You can read every word of the above sentences, or every word of the entire report, including every word in the footnotes, and you walk away thinking this adolescent trial included two age groups (12-15 and 16-25). What if this weren't true? What if the number of 16-25 year olds in this adolescent trial is exactly zero?

As I was reading the prestigious NEJM report, I was feeling a sense of dread, which worsened as page after page repudiated what I thought was true: that the 16-25 year olds data came from the previous adult trial. Indeed, one of the key issues I flagged in my previous blog post is that the analysis plan involved comparing one age group in one trial to another age group in a different trial. Did I mislead my readers? Was I so blinded by my own biases that I made up this detail?

It surely was not possible that scientists and peer reviewers and the editors of the prestigious NEJM would allow such a basic but important detail to be omitted. It has got to be my brain fog... am I losing grip of my sanity?

So I checked my notes... The very first line of the Pfizer press release from March 31 stated unmistakably: "In participants aged 12-15 years old, BNT162b2 demonstrated 100% efficacy and robust antibody responses, exceeding those reported in trial of vaccinated 16-25 year old participants in an earlier analysis, and was well tolerated." (their italics)

Next, I checked the FDA press release on May 10. The source of data for the 16-25 year olds has been wiped out. In the safety section, the FDA stated "The side effects in adolescents were consistent with those reported in clinical trial participants 16 years of age and older. " As if the adolescent trial contained participants 16 years of age and older. In the efficacy section, the FDA stated "The immune response to the vaccine in 190 participants, 12 through 15 years of age, was compared to the immune response of 170 participants, 16 through 25 years of age. " Unless you know otherwise, you're going to assume the trial contained participants of both age groups.

I warned you up top that you're witnessing an instant classic in science communications.

It takes discipline and skill to write an entire report that dances around a key detail. A true masterclass in deception. This is an instance of "story time," lulling readers in with clinical trial data but presenting conclusions from analyses that do not enjoy the benefits of an RCT. But "story time" trivializes what has happened here. I hate to use the F word. I'm not thinking the four-letter F word. The five-letter one. F R A U D.

***

Also of interest is the other side of the equation. How does one develop a nose for detecting fraud in data? I have to admit that it would have been very hard for anyone to notice the deception in this NEJM paper - that's why it belongs on the syllabuses of science communications courses.

There were just two tiny cracks. The first was when they disclosed that all 12-to-15-year-old participants came from the U.S., while 16-to-25-year-old participants were globally recruited. This would have been unusual to do in a single trial enrolling 12-25 years old, stratified into two age groups.

The second crack was in the way the waterfall chart was presented.

Notice the two age groups were presented in two separate panels. What we expect to see if the trial actually enrolled both age groups is something like this:

The waterfall starts with the entire set of enrollees and then there is a split by age groups. This chart came from the Phase 2 Moderna study of two half-doses that I discussed in this post.

As I said, these are subtle signs, easily missed.

***

Is it never allowed to pull a control group from a different place? Doing so does not automatically disqualify the analysis but such an analysis does not enjoy the exalted gold-standard status of a randomized controlled trial. Such workarounds must be scientifically and rigorously justified. Work must be presented to convince readers that any biases coming from different times, different places, different ages, different demographics, different anything have been measured and corrected for.

Here, the authors did not present any evidence about possible biases. They didn't even acknowledge the source of the control group data.

***

Listed below is every single mention of the 16-to-25-year-old control group in the Pfizer adolescent study in the NEJM scientific report. Each mention represents an opportunity for the researchers to inform readers that this control group came from a different trial.

• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was an immunogencity objective.
• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The immunogenicity objective was to show noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants.
• Noninferioirty was assessed among participants who had no evidence of previous SARS-CoV-2 infection with the use of the two-sided 95% confidence interval for the geometric mean ratio of SARS-CoV-2 50% neutralizing titers in 12-to-15-year-old participants as compared with 16-to-25-year-old participants 1 month after dose 2.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was assessed on the basis of the geometric mean ratio of SARS-CoV-2 50% neutralizing titers.
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Severe injection-site pain after any BNT162b2 dose was reported in 1.5% of 12-to-15-year-old participants and in 3.4% of 16-to-25-year-old participants.
• After BNT162b2 injection, severe headache and severe fatigue were reported in a lower percentage of 12-to-15-year-old participants than of 16-to-25-year-old participants.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• The use of antipyretic agents was slightly more frequent among BNT162b2 recipients who were 12 to 15 years of age than among those who were 16 to 25 years of age.
• [Footnote of Table 1]: Results are for the reactogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and a subset of participants in the 16-to-25-year-old cohort.
• Among the 16-to-25-year-old BNT162b2 recipients, 11% reported any adverse event and 6% had vaccine-related adverse events. Among BNT162b2 recipients, severe adverse events were reported in 0.6% of those who were 12 to 15 years of age and in 1.7% of those who were 16 to 25 years of age.
• One BNT162b2 recipient in the 16-to-25-year-old cohort discontinued vaccination because of severe vaccine-related injection site pain and headache, both of which were reported 1 day after dose 1 and resolved within 1day.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.
• Appendicitis was reported in 2 participants: 1 BNT162b2 recipient in the 16-to-25-year-old cohort and 1 placebo recipient in the 12-to-15-year-old cohort.
• The immune response to BNT162b2 in 12-to-15-year-old adolescents was noninferior to that observed in 16-to-25-year-old young adults.
• The geometric mean ratio of the BNT162b2 neutralizing geometric mean titer in 12-to-15-year-old participants to that in 16-to-25-year-old participants 1 month after dose 2 was 1.76.
• Among all participants regardless of serologic evidence of previous SARS-CoV-2 infection, the serum-neutralizing geometric mean titer 1 month after BNT162b2 dose 2 was 1283.0 in the 12-to-15-year-old cohort and 730.8 in the 16-to-25-year-old cohort.
• Substantial increases in the 50% neutralizing titer from baseline were observed, with GMFRs from baseline to 1 month after dose 2 of 118.3 among 12-to-15-year-old participants and 71.2 among 16-to-25-year-old participants.
• [Caption of Figure 2]: The results shown are for the reacogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and the subset of participants in the 16-to-25-year-old cohort who had electronic diary data available.
• [Footnote of Table 2]: The geometric mean ratio and two-sidded 95% confidence intervals were calculated by exponentiating the mean difference of the logarithms of the titers (the 12-to-15-year-old cohort minus the 16-to-25-year-old cohort) and the corresponding confidence intervals (based on the Student's t distribution).
• Antipyretic use after both dose was slightly higher in the 12-to-15-year-old cohort than in the 16-to-25-year-old cohort.
• All 12-to-15-year-old participants in this trial were enrolled at U.S. sites, whereas the 16-to-25-year-old participants were recruited globally.

U.S. regulators asleep at the wheel

My friend Mark Palko has been on Tesla's case for a while. He just commented on the latest fatal accident involving a Tesla vehicle. Amusingly, at this point in time, we are supposed to believe that a ghost crashed the car killing two passengers - as we're being told neither a human driver nor Tesla's software was directing the car.

Mark rightfully laid the blame on regulators. It's pretty clear that U.S. regulators put business profits above citizens' interests. We learned how Boeing was allowed to inspect themselves. The Europeans, not U.S. regulators, are reigning in the likes of Google and Facebook. Tesla's case is particularly telling.

Observation #1

U.S. regulators are allowing Tesla to lie about its "self-driving" technology. The "Auto-pilot" software does not auto-pilot - in fact, drivers are told to have their hands on the wheel at all times. The "full self-driving" mode is not what researchers in the field mean by full self-driving. Tesla customers are used as test subjects for an immature technology. The regulators should have stepped in to stop the false advertising but they have been absent.

Observation #2

After an accident happens, it appears that regulators rely on Tesla to report on itself. It seems that only Tesla has data on whether Auto-pilot or other features were turned on or off, whether the driver's hands were on the wheels, etc. Imagine the Chauvin trial, and the only video for the incident came from the police's bodycams. How do we know the footage hasn't been edited, and that the best perspectives were submitted?

Observation #3

The entire conceit of requiring the drivers to have their hands on the wheel at all times while using "Auto-pilot" not only makes a mockery of the term "auto-pilot" but also creates a liability-free zone for Tesla. If the driver's hands were determined not to be on wheel (and the only source of this information is Tesla), any accident is blamed on the driver. In past accidents, they even faulted drivers for momentarily not touching the wheel. If the driver had hands on the wheel, the accident is also not caused by Tesla because this accident could not have been prevented.

Observation #4

Some Tesla fans argue that everyone should know "Auto-pilot" is false advertising, and so any dead or injured Tesla customers have themselves to blame since they chose to use the feature. Using this logic, anyone who died from opioids deserves to die since they chose to ingest.

 

 

 

 

If people would buy the real thing, there would be no counterfeits

In Chapter 3 of Numbersense (link), I told the story of the software industry complaining that pirates stole billions worth of software each year. This estimate of the value of software embeds the fantasy that people who use pirated software would have purchased legal software at list price should piracy be eradicated. It's a failure to think counterfactually. Many people who use pirated software cannot afford, or do not want, to pay for the legal version.

The tech industry invokes this logical flaw frequently. Earlier this week [a month ago by the time this post is published], news broke that the U.S. Customs and Borders Protection (CBP) intercepted "counterfeit" Apple Airpods but quickly, tech-savvy observers pointed out that those earbuds are made by OnePlus for Android phones, an Apple competitor. The photos included in the CBP press release even showed the seized products in OnePlus packaging.

CBP estimated that the seized products had a value of $389,000, by multiplying the number of products with the list price of Apple Airpods. Such an estimate assumes that if OnePlus earbuds were banned, consumers would have all purchased the much more expensive Apple product. The list price for OnePlus earbuds is less than half that of Airpods.

In this ArsTechnica article, the CBP claims $1.5 billion worth of counterfeit products are seized in a year. This number appears to suffer from the same counterfactual fallacy. What would consumers do if OnePlus earbuds were banned? A proportion of them will go back to wired headphones. Another group will migrate to other earbuds that are in a similar price range as OnePlus. These are not merely speculation - we do know that these customers elected to buy OnePlus instead of paying double for Apple.

If everyone who purchased the counterfeits would buy the real thing at list price, why didn't they buy the real thing in the first place?

 

The absurdity of predictive policing, part 2

This is a follow-up post to Part 1. Read that post first to catch up. In Part 1, I set up a scenario, in which 1,000 people are flagged by a predictive-policing model and sent to jail for their future crime of burglary.

At the end, I asked how many of the 1,000 people were accurately flagged by the predictive model.

***

At the crudest level – and this is the argument made by those who sell predictive policing technologies, the model was 100 percent accurate. Look, it flagged 1,000 people, and subsequently, none of these people committed burglary! Amazing! Genius! Wow!

The absurdity of this calculation is that such perfection could have been obtained with any model – even a random model. If I randomly selected 1,000 people from the list, and convinced the courts to impound them because of their future crimes, those 1,000 would also not commit burglary. Therefore, the fact that the predicted burglars didn’t commit the crime proves nothing about the model’s effectiveness! It just states the obvious fact that if we lock someone up, that person couldn't commit burglary.

For each person flagged by the predictive model, the relevant question is: if we didn’t throw this person in jail (or take other actions that would alter the outcome), would s/he have committed burglary? If the answer is yes, then the model has made one accurate prediction. The trouble is it's counterfactual. Taking action using this model removes our ability to measure its accuracy.

***

In practice, a few strategies can be deployed to assess such a predictive model.

Strategy 1 is back-testing. Apply the model to historical data. Train the model using data before a cut-off date. Use it to make predictions after the cut-off date. Run time forward, and compare the burglary rate between the flagged and unflagged populations.

This is easy to do but fraught with dangers. It only works if the modeler does not cheat by using data after the cut-off date. The most common way of cheating is to tinker your model after learning that it isn't accurate enough.

Since every analyst building models tinkers the model upon learning it isn't accurate enough, every analyst "cheats". (Textbook authors, though, have memberships to the private clubs where analysts only use their test sets once.) Thus, basing any decisions purely on back-testing is foolish. Buying predictive technologies from a vendor who only reports back-tested results is foolish.

Strategy 2 is dry run. Run the predictive model, flag the predicted offenders but take no action. After sufficient time, look at flagged versus not flagged and the burglary rates between the two groups. If the model is any good, the burglary rate amongst those flagged should be substantively higher.

People who don't want you to know about accuracy counter with these objections: (a) it takes too long (depending on the burglary rate of the neighborhood, this could well take 6-12 months to collect enough cases); and/or (b) you're letting crime happen while this experiment is running - this critique is usually delivered with a contempt for science: "you're not being practical, you're being a scientist!".

Reporting results from a dry run builds confidence in the accuracy of the predictive models - on average. It still doesn’t change the fact that for any individual flagged and arrested based on the model’s prediction, we would never know if the model was correct.

***

Further reading:

Chapter 4, Numbers Rule Your World: discussion in the context of predicting terrorists (link)

Chapter 5, Numbersense: discussion in the context of Target's pregnancy prediction model (link)

The absurdity of predictive policing

Big (surveillance) data is changing everything - that is undeniable. But is Big Data changing everything for the better? That is what you have to think about.

As someone who's done data analytics for a long time, I can say confidently that the potential for harm is at least as large, but probably larger, than the potential for good.

A case in point is data as related to crime and policing. This post picks up from the perceptive article in the New York Times, "How Reporting Practices can Skew Crime Statistics" (link).

The article points out something really important - data is as data is defined. Different definitions result in different analytical results, frequently contradictory. No one should interpret analytical results without first learning the data definitions.

The author describes two extremely popular styles of "big data" analytics: top N ranking, and trend analysis. In crime statistics, sample insights are (1) "South Bend, Indiana is the 27th worst city for violent crime rate among all cities with over 100,000 people." and (2) "Violent crime has worsened in South Bend since mayor Pete got elected in 2012."

With the volume of data available today, those two simple analyses still account for the vast majority of "analytics" out there.

South Bend, Indiana is the 27th worst city for violent crime rate among all cities with over 100,000 people.

Here is a list of definitions that affect the interpretation of this statement, most of which are described in the NYT article:

• whether the suburbs are included as part of the "city", or more broadly, how the boundaries of the city are drawn. This matters because the crime rates in the suburbs are different from the crime rates in the city centers.
• whether the error in the count of population is comparable across all cities under consideration
• what is considered a crime
• what proportion of crime is reported
• what crime is classified as "violent"
• are there any exceptions

You might think the population of a city is an objective fact but it isn't. Definitions matter, and they matter a lot when you're comparing different cities.

Violent crime has worsened in South Bend since mayor Pete got elected in 2012.

In addition to the points listed above, we also need to know if any of those definitions changed between 2012 and 2019. Changing how something is defined almost always breaks the continuity of any analysis, thus undermining any trend analysis. I raised this point in my book Numbersense (link) in the chapter on obesity - if the medical community changes how we measure obesity, then all historical data are rendered useless so such proposals must be considered using great caution.

Further, we should find out whether violent crime was increasing or decreasing prior to 2012 in South Bend, whether violent crime was increasing or decreasing in other cities comparable to South Bend since 2012, whether violent crime was increasing or decreasing in the nation as a whole since 2012, etc.

***

Some readers of my book Numbersense (link) might be mystified why I spent so much energy discussing definitions - of grad school admissions statistics, of U.S. News ranking, of obesity measures, of unemployment rates, of inflation rates, of fantasy football metrics, etc., etc.

A key to numbersense - to judging the statistics and data thrown at you - is to understand clearly how the data are collected, and in particular, how any metric is defined.

The discussion of crime statistics by the New York Times makes this case cogently.

***

Now let me get to the absurdity of predictive policing.

A particularly popular and cited application of Big Data "for social good" is predictive policing. The goal of predictive policing is to prevent crime from happening.

The ideal of "Minority Report" is often hoisted on us. Wouldn't it be great if some AI or machine learning or predictive model can figure out who will be burglarizing your home next year, and have the police lock the would-be burglar up before s/he could commit the crime?

I have done a good amount of predictive modeling in my career so I am not saying what I'm saying because I'm a luddite. It's because of that experience that I am leading you down the following rabbit hole:

There are not many burglaries. According to Statista, the worst state (New Mexico) suffered 770 burglaries per 100,000 residents in 2018. This means that if the analyst is given a list of 100,000 New Mexicans chosen at random, a perfect model should label 770 of those as potential burglars. That is to say, we want the model to pick out 0.77 percent of the list.

A perfect model does not exist. Any model generates false positives. Because burglars are rare, any reasonably predictive model would need to red-flag many more than 0.77 percent of the list to have any chance of capturing all 770 potential burglars. Let's say the model points the finger at 1 percent of the list. That would mean 1,000 potential burglars. Since there should be only 770 burglars, the first thing we know is that at least 230 innocent people would be flagged by this model.

Next, think about what happens to the 1,000 flagged people. In the ideal of predictive policing, police would knock on their doors and arrest them for their future crimes. These people who haven't burglarized anyone would be duly jailed by courts who buy into predictive policing. (How would they defend themselves against an allegation of future crime?) 

We could say with certainty that those 1,000 did not subsequently commit burglary - given that they were incarcerated.

So humor me, and try to answer the following question: Of the 1,000 people flagged by this model, how many of these were accurately predicted?