Big Data, Plainly Spoken (aka Numbers Rule Your World)

Numbers indeed rule our world, 10 years out

Statistics is the subject of generalizing data. In statistics, we study how to use imperfect data to draw general conclusions. We often have to wait for the future to reveal the evidence of success.

Ten years ago, McGraw-Hill published my first book, a best-seller, Numbers Rule Your World (link). I’ve hoped to explain five of the most profound concepts from my field in a way that anyone can understand and apply to their daily lives. I attempted the balancing act of featuring specific stories while bringing out the general principles. In the last couple of months, the Covid-19 pandemic has generated immense attention to data and statistics, and provided me confirmation, ten years on, that the contents of Numbers Rule Your World (link) have general applicability. It affirms that statistics is a great subject!

If you’re approaching my book for the first time, or re-reading it, here’s how you can work your way through it, making connections at various points to the Covid-19 crisis.

***

Chapter 2: Bad Spinach / Bad Scores

This is your perfect starting point. Half of the chapter details a disease outbreak investigation. You learn about the information networks the CDC puts in place to monitor unusual patterns of disease around the country. You discover data analysts whose job is to set off the alarm. How do they decide if the first handful of cases foretell an epidemic?

You encounter “shoe leather,” the painstaking collection of data to pinpoint the origin of an outbreak. It’s much harder than you might think. The U.S. politicians are making much noise about the early days of Wuhan. An epidemiologist may learn that 10 out of 15 first patients all visited the Wuhan seafood market before they got sick. But the Wuhan market might be a hugely popular destination, like Central Park in New York City. Finding 10 out of 15 people went to Central Park over a weekend could be more common than you think. If all 10 touched a particular rare species of wildlife, that's another story.

Another controversy of the moment is the huge cost of shutting down the economy, against the benefit of saving lives. I invite readers to think about this cost-benefit tradeoff. Because the disease outbreak was traced to a spinach processing plant in California, the FDA ordered a recall of all spinach, which depressed sales of salad greens for six months or so. The cost was much lower than for this pandemic, but the bacteria is well-known and thus predictable, and the lives saved were most likely zero, and even harder to pin down than in the current crisis.

People are already complaining that social distancing and other measures have not improved our numbers, by saying deaths were “unexpectedly low”. You discover how hard it is to prove the effectiveness of a policy decision because you need the counterfactual – what would have happened if the FDA did not recall the spinach.

Chapter 4: Timid Testers / Magic Lassoes

Everything you should know about the statistics of diagnostic testing and predictive models in one place. The motivating examples are (a) the fight against performance enhancing drugs in sports via anti-doping tests and (b) the false promise of terrorism prediction algorithms.

I wrote several posts (1, 2, 3) about the flawed Stanford antibody study that reported 50 times more infections in Santa Clara county in California than reported cases. Despite this headline, the raw data is also consistent with a scenario of zero proportion having antibodies. The statistical reason is the low underlying prevalence of antibodies, which caps the number of true positives in any sample. The number of false positives will be large relative to true positives, because the false positive rate applies to those without antibodies, which is the bulk of the population.

An extreme case of this phenomenon is terrorist prediction using a dragnet surveillance dataset. The set of terrorists is a drop in the ocean of people under mass surveillance. For a predictive model to capture most of these terrorists (low false negative rate), it must flag a lot of people – even a tiny false positive rate when applied to a huge population results in a lot of false accusations.

You see the distinction between a “chemical” false negative (positive) and a “practical” false negative (positive). To evade steroids testing, dopers use all kinds of tactics, such as handing in someone else’s urine. The diagnostic test finds a true negative but in practical terms, it’s a false negative. For swab testing, one may test negative if the swab doesn’t reach a virus-rich area, or if the patient is tested at the “wrong” time.

All tests have errors. What are the consequences of errors? Also, how likely will the error be discovered? For anti-doping, do dopers inform the test lab they made a mistake?

Chapter 1: Fast Passes / Slow Merges

The first chapter gives important insights on managing capacity, which is a pivotal issue for hospitals during the coronavirus crisis. I cover two case studies: (a) Disney’s management of theme-park capacity, and (b) state’s management of highway congestion.

The strategy of building more capacity is unavailable during a crisis, and wasteful in general. In this extraordinary time, some localities were able to bring up temporary hospitals. Transportation experts discovered that policies of regulating congestion are more effective. Highway ramps slow down the influx of new vehicles onto the highway, and are turned on before congestion arrives. The key is to delay as much as possible the onset of congestion. Sounds familiar? It’s the analogue to “flattening the curve”. Social distancing is supposed to reduce contacts, slowing down community spread. This tactic delays the peak demand for hospital resources. The chapter explains the scientific support for why such tactics work.

Chapter 5: Jet Crashes / Jackpots

Statistical significance is the topic of this chapter, one of the most common and also most commonly mis-used concepts in statistics.

In the Stanford antibody study referenced above, statisticians argued that the observed level of positive results is consistent with zero prevalence. This is a way of saying the result is not statistically significant. What we mean is that the study’s result (50 positives out of over 3,000 tests) would be considered normal even if nobody in the population actually had antibodies. This conclusion does not assert that the true prevalence is zero. The topic of statistical significance is made inaccessible to non-mathematicians by textbook authors piling on equations, but it need not be so. I even made a video on explaining "not statisitical significant" (link).

Before you apply a test of statistical significance, you must make sure you’re comparing apples to apples. In the context of air safety, you shouldn’t compare U.S. and foreign airlines on all routes; you should only compare them on routes they share, in which case, you’ll find no statistical difference. One of the bad stats that is circulating during this pandemic is the purported harm of ventilators. This is usually stated as the vast majority of patients put on ventilators eventually die (presumably, relative to those patients not put on ventilators). But only the most serious patients are placed on ventilators. The proper analysis should focus on patients who qualify for ventilators, and compare those who were put on ventilators to those who got other treatments.

We have already seen a host of invalid comparisons. Comparing complete data from years ago with incomplete data this year. Comparing regions that are in different phases of the epidemiological timeline. Comparing countries that have taken different approaches to managing the crisis.

Chapter 3: Item Bank / Risk Pool

This chapter addresses a deeper issue in using statistical averages. An illustrative example is educational testing like the SAT. Critics pointed to the gap in the average scores across racial groups as evidence of bias. But the racial grouping masks a third factor, which is differential ability between students. Due to various socio-economic factors, white students have higher average ability than black students. So the real question is: Among those students with comparable abilities, does the test still show bias toward one racial group? (The answer is not straightforward.)

During the pandemic, the “denier” faction has argued that the mortality rate of Covid-19 is “on a par with” that of seasonal influenza. Are those two rates comparable? The simple answer is no. Comparing mortality rates masks a third factor, which is the level of immunity in the population. Even at the most optimistic, the current level of immunity to the novel coronavirus is less than 20 percent (This statement generously presumes that (a) having antibodies confer immunity without question and (b) the New York City test result is not a scam, and the test is 100% accurate). By contrast, because of vaccination and recurrence, a good chunk of people have immunity to influenza. The mortality rate of Covid-19 should be compared to that of influenza among a population that has no immunity to either, or a population with similar levels of immunity to each.

The other example in this chapter explains how the insurance industry is an outgrowth of the law of large numbers. In any given year, health or auto insurers cannot predict with high accuracy which covered individuals will file claims, but the magic of statistics allows them to predict with great accuracy the aggregate claims. Disaster insurers run into trouble when rare events happen, such as when Class 5 hurricanes hit Florida. Suddenly, potentially all covered individuals tap into the insurance pool all at once. Insurers try to pool together risks of different types and different areas. The global pandemic is a huge threat to the insurance industry. Perhaps there are lawyers reading this, and they might tell me there are natural disaster exclusions. If those don’t apply, we will see a gigantic number of covered entities filing claims all at once.

Florida’s hurricane insurance market was dysfunctional also because the risk is predictable along some parameters. Those living inland realized they were subsidizing the coastal dwellers and wanted out. The coronavirus also creates different risk subgroups. The younger generations have much lower risks than the elderly. When social-distancing and other practices are applied to the whole community, the subgroup with lower risk peels away. A key difference is community transmission – the risk of hurricane damages does not spread by contact.

***

The above covers less than half of the book. Almost everything in those pages has direct relevance to the Covid-19 world. It’s quite heartening to find out ten years later numbers indeed rule our world.

Here’s a link to get your copy of Numbers Rule Your World. There are also translated copies in Chinese (simplified and traditional), Japanese, Korean, and Portuguese, as far as I know.

Excess deaths: why we want the cake and why we can't eat it yet

In my early March preview of statistical issues relating to coronavirus, I mentioned "people are counted as infected only if they are tested", and I also asked "how are deaths confirmed?"

For a while, the media ignored how meaningless the case counts were. Eventually, they got it, and that was great. But some experts convinced them that the savior is the death count. And now, in the last week, the reporters finally learned that death counts are subject to similar problems of definition.

The Economist just launched a tracker to look at excess all-cause mortality, using data from EuroMOMO. If during this pandemic, the total number of deaths from any cause is higher than normal, then it provides evidence (though not irrefutable) that Covid-19 is not a new name for influenza. Of course, we are finding excess deaths everywhere.

The next puzzle is that the number of deaths attributed to Covid-19 on death certificates is frequently lower than the excess deaths from all causes. This raises the possibility that Covid-19 deaths are under-reported. By itself, this is not necessarily true, because it is also possible that deaths from other causes are being under-counted.

However, it has also emerged that in many countries and regions, the published counts of fatalities so far have only included deaths occurring at hospitals for whom the deceased had returned a positive test for the virus. A good number of deaths occurred in care homes, hospices, and even private residences, and these have been ignored. In fact, many hospitals refused to admit patients who do not report severe symptoms, and told them to isolate at home.

The excess all-cause mortality data take a bit more time to collect, and perhaps not by accident, as these statistics come to light, many countries announced that they are revising how they count Covid-19 deaths, essentially admitting they were under-reporting.

***

The lesson related to data integrity is being taken up very slowly. Even as the Economist published the valuable tracker of all-cause mortality, they wrote:

... people have become grimly familiar with the death tolls... these numbers give a better indicator of a country's trajectory than do counts of confirmed cases, which largely measure how many people have been tested. Nonetheless, official covid-19 death tolls still under-count...

This is better than most but still repeats several fallacies:

a) Given the unknown fatality rate of Covid-19, the death count is a proportion of the case count. If data integrity is not at issue, the death count is a worse indicator of a trajectory than the case count because it just gives the same information after a time delay. If death tolls are better than confirmed cases, it's because we believe the data on deaths are less bad than the data on cases.

b) The emerging evidence shows that the unadjusted death counts are wildly inaccurate, so I'm not sure you can say the data on deaths are "the lesser evil".  In Lombardy, the Economist estimated the "true toll was about 120% higher" than the confirmed deaths. In Spain, the true number might be 60% higher (off a much larger base).

c) Confirmed cases is not a simple function of how many people got tested. More importantly, it is measuring who got tested. I know I have been hounding this point to death. In statistics, we care not only about the size of the sample but the composition of the sample! We could test lot of people with no symptoms and still have few confirmed cases.

d) The under-counting of deaths is still partially due to lack of testing. It is true that Covid-19 testing is not required in those all-cause mortality statistics but without testing, the count of deaths is clearly less reliable, just like the count of cases.

e) Deaths can also be over-counted, especially without testing to confirm coronavirus. How well can a doctor tell if the patient who died of pnuemonia had Covid-19 or some other illness?

f) If it's broke, you can't fix it. Countries change the definition of deaths mid-way but won't (and can't) change the past. This renders the death count series useless, unless you're willing to adjust the data. This was a key point I made about new metrics of obesity in Chapter 2 of Numbersense (link).

***

So, if you've been reading this blog, you know interpretation of statistics is no simple business.

While compiling these excess mortality numbers is necessary and important, here is something you should know, and I'm not going to sugarcoat it.

Imagine wearing glasses or sunglasses in the summer. You walk into an air-conditioned shop. You must wait for the fog to clear.

These excess mortality numbers are shrouded in a fog right now. It is actually somewhat meaningless to discuss them. This is because of a substitution effect. Some of the excess deaths we're seeing are accelerated deaths. (Of course, all excess deaths are accelerated deaths.) What proportion of the excess deaths are relevant will be up for debate once we wait long enough to see clearly.

A store usually runs an annual sale on New Year's Day only. This year, the store starts the discounts on the day before. After New Year's Eve, the store reports 50% jump in sales versus last year ("excess"). Was the promotion particularly effective this year? You don't know until after New Year's Day. When you compare the total sales of both days against the two days last year, you'll find that the jump is less than 50%. Some customers just showed up one day earlier.

 

 

 

 

Statistical evidence is both powerful and limited: fraud in baseball and lotteries

In Chapter 5 of Numbers Rule Your World (link), I told the story of how Professor Rosenthal used statistical evidence to uncover a cheating scandal in a Canadian lottery. Statistics can be very powerful in identifying suspicious data - in this case, the winning probability of store owners who sell lottery tickets was found to be way higher than normal.

In a textbook presentation, you can simply do the math and compute the relevant probabilities and invoke statistical "significance" arguments. Hey, we've found evidence of cheating. End of story.

In real life, that would just signal the beginning of a long story. The statistical conclusion is merely that "someone cheated" or at best, that "someone cheated almost surely". But you can't catch the cheater(s) based on that evidence! The question left unanswered is: who cheated?

In the example from the book, we know that as a group, store owners cheated. That doesn't mean every store owner cheated. Who cheated? For that, you need old-school shoe leather. You need to find the receipts.

***

I was inspired to write this post after a friend asked me whether the Astros were the only baseball team that stole signs. Here is a summary of the cheating scandal engulfing professional baseball right now.

The statistical argument goes like this: the kind of sign-stealing using video can only happen at home stadiums. Teams who cheated in this way would have a performance advantage in their home games compared to the away games. But of course, every team enjoys a home-field advantage so we're talking about an abnormally high home-field advantage.

Where statistics demonstrates its power is here. We have guidelines for what level of home-field advantage should be considered abnormally high.

The question my friend asked is: If there was a second team who also did remarkably better at home games than away games - similar to the Astros, would that be sufficient evidence to convict that team as well?

Sadly, the power of statistics is also limited. We think cheating might be behind the abnormally high home-field advantage - but we can't be sure! We are not unsure about the observed outcome; the uncertainty concerns the cause of such an outcome.

Similar to the lottery fraud scheme described in Numbers Rule Your World (link). We are very confident that too many winners were store owners but nothing in the data tells us that the cause is cheating! The case didn't get solved until specific store owners were caught. You need to find receipts. 

Unless there is a confession by the cheaters, or a reporter or detective discovering incriminating evidence, we don't know what caused the anomaly in the data. Suspicion is not proof of wrongdoing.

In baseball's case, it was a former Astros pitcher who confessed to the scheme. Without receipts, the cheaters can always offer other explanations for the performance advantage. The most powerful alternative explanation is coincidence.

***

Here's an example of an alternative story. Someone could claim that their hitters started making "superman" poses before they get on the field, and that explained the advantage.

But spinning such stories is to run "story time". We have some data - the performance advantage - that show something. Then, we seamlessly move to a story that has nothing to do with the data just presented. Just because there is a presentation of data doesn't mean the data support the story!

 

 

Quick hits on data issues in the Democratic primaries: cancelled poll, and cost of complexity

The Democratic primaries are generating - unexpectedly - stories with good data angles.

First, there was the "cancelled" Selzer poll leading up to Monday's contest (link). While we still don't have a proper explanation, what was reported in the press is that there was a single instance in which Pete Buttigieg's name might have not been read, possibly due to the long list of candidates, some of whom will be "off screen". In this case, the alleged instance is due to someone increasing the font size on the screen, thus moving names off the viewable area.

I find this explanation inadequate. In any survey involving thousands, there will be a few issues - kind of like dropping bad ballots. What if, for example, a phone connection got momentarily mangled just as Buttigieg's name was read? If the Buttigieg's camp found one bad case, they surely went around looking for more examples - and as far as we're told, they didn't find any more.

Further, any competent survey designer - and Selzer is a good one, according to Nate Silver - knows that you always randomize a list of responses. In other words, the order in which the candidates are listed for the same question will be randomly assigned for every phone call. So if Buttigieg's name got missed because it's at the bottom of the list for some calls, his name would be at the top of the list for other calls. If x percent of the calls are flawed because of omitting names, each candidate will be affected by the same amount. The randomization equalizes the impact of such unpredictable errors.

Cancelling such an important poll is a decision that changes the dynamics of the race, and a much more thorough accounting is needed.

Selzer's explanation is an "abundance of caution". If true, this instance opens up a flood gate for delegitimizing polls. If a complaint based on a single anomaly is allowed to topple a poll, then any candidate can conspire to preempt poll results that s/he doesn't expect to like.

The decision to cancel the poll is not fair to all parties. Turns out someone leaked partial results of that poll on the day of the Iowa primary (I saw this on FiveThirtyEight, which was not the source of the leak). It clearly shows that some candidates would be happier with the poll result than other candidates.

***

In the turmoil around the Iowa caususes, we must differentiate between what the data show and what might explain the data. There is no doubt that this screwup hurts the winner, and any other candidate that did unexpectedly better while it helps any candidate who did unexpectedly worse. This outcome is also not fair to all parties.

The problem is when we impute a cause, or motivation to explain the effect. We don't know if someone conspired to help or hurt any particular candidate - unless specific corroborating evidence is subsequently found.

Yet, we don't need a motivation to agree on the positive/negative effects on specific candidates.

***

Further, people are underestimating the cost of complexity. Having two candidates in 2016 and having 10 candidates now makes things much more complicated. (I'm using two and ten for illustration. The list is longer if we include every fringe candidate.)

This increase in candidates is magnified by having multiple rounds of voting, and realignments. With two candidates, the realignment decision is either to join the other side or go home. With 10 candidates, there are so many possibilities!

Then, apparently, they also promised to provide vast amounts of data. This policy is itself commendable. With two candidates, validating the numbers is easy. There is really only one number per precinct to check. With ten candidates, there are nine numbers per precinct to check. There are over 1,000 precincts. The resulting tabulation has lots of missing values (holes) because by definition, most candidates become non-viable by the end of the night for each precinct but the set of candidates that remain viable vary by precinct.

A simple way to think about the cost of complexity is this: each number has some unknown error rate, say 1 percent. If we need an accurate result, we need the probability of no errors. That is, 1 minus the probability of one error minus the probability of two errors ... minus the probability of all errors. The 1-percent error gets magnified very fast as the number of entities grow!

 

 

In times of paranoia, seek out data and stick to facts

All around the world, people are getting very afraid of the coronavirus. What's happening now is a live version of a disease outbreak investigation, which is discussed in Chapter 2 of Numbers Rule Your World (link).

To understand what's going on, you should see this as the exact opposite of "big data". We have a new flu virus. There is almost no direct data on this virus. The data are dribbling in as cases are being reported. We have a couple hundred deaths.... in a country with almost 1.5 billion people. Public health officials must make key decisions now before more data become available. But we have already passed the most dangerous phase, because they already found what causes the disease.

I saw a graphic last night, shown during an interview with an expert who investigated the SARS epidemic. Supposedly the mortality rate for the coronavirus is about 3 percent. This number has a large margin of error at this stage because of the shortage of data. This shortage is due to the newness of the phenonemon.

The closest relative to the coronavirus is the SARS virus which had a mortality rate of about 10 percent, which is three times higher than what we currently know about the coronavirus.

We also know that the coronavirus spreads faster than SARS. Most of the news reports I read mention the number of infected but rarely the number of deaths. This is a major oversight.

What's fearful is a deadly disease. A disease that infects lots of people but acts like the common cold is not a public-health crisis. So as data roll in, we should keep an eye on the mortality rate.

The mortality rate is the number of deaths divided by the number of infected. Both counts are growing each day. If the growth of infected is faster than the growth of deaths, the mortality rate will go down (not up!). The mortality rate will go up only if the growth rate of deaths exceeds the growth rate of infected.

In fact, by the time I finish this post, the latest report said the coronavirus death rate is 2 percent, so it has already been adjusted downwards because the flu is spreading fast. (The death rate of regular flu is about 1 percent, according to the graphic that was shown last night. This level acts as a control for interpreting the mortality rate.)

That report contains a typically alarmist quote by a medical professional: "There are likely to be many times more cases in Wuhan than officially confirmed". Okay, but if that is true, the real mortality rate is many times smaller than what is official! In terms of missing data, it's the unreported deaths that are more significant than unreported infections.

Also note the network effect. It's very unlikely that the growth rate of deaths can be higher than the growth rate of infected because infection grows exponentially due to transmissions. Deaths don't transmit.

It would also be useful to know what the cure rate is - because some of the infected will be in the treatment stage. During the report last night, the journalist wrongly asserted that 97 out of 100 infected will survive. This interpretation is correct only if we exclude those infected that are currently in treatment. (The technical term is right-censored data. We don't yet know how many in this subgroup will survive or not.)

While one should be cautious because this coronavirus is new, in times of paranoia, one should seek out the data. Ignore the unfounded rumors, innuendo and folklore. Stick to the facts. So far, the data aren't that alarming.

***

I n Chapter 2 of Numbers Rule Your World (link), I describe two modes of statistical modeling. Disease outbreak investigation illustrates one mode, in which the chief concern is establishing cause-effect relationships. This is always hard if you have observational data - without controls. But it's even more challenging in these crisis situations, with small drips of data. If you understand what statistical tools are available, and what the limitations are, you have a better grasp of what's going on right now.

 

Image credit: By https://www.scientificanimations.com - https://www.scientificanimations.com/wiki-images/, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=86358105

 

P.S. [2/3/20] See also my post on Junk Charts about a graphic used by the New York Times in a report on the virus.

The absurdity of predictive policing

Big (surveillance) data is changing everything - that is undeniable. But is Big Data changing everything for the better? That is what you have to think about.

As someone who's done data analytics for a long time, I can say confidently that the potential for harm is at least as large, but probably larger, than the potential for good.

A case in point is data as related to crime and policing. This post picks up from the perceptive article in the New York Times, "How Reporting Practices can Skew Crime Statistics" (link).

The article points out something really important - data is as data is defined. Different definitions result in different analytical results, frequently contradictory. No one should interpret analytical results without first learning the data definitions.

The author describes two extremely popular styles of "big data" analytics: top N ranking, and trend analysis. In crime statistics, sample insights are (1) "South Bend, Indiana is the 27th worst city for violent crime rate among all cities with over 100,000 people." and (2) "Violent crime has worsened in South Bend since mayor Pete got elected in 2012."

With the volume of data available today, those two simple analyses still account for the vast majority of "analytics" out there.

South Bend, Indiana is the 27th worst city for violent crime rate among all cities with over 100,000 people.

Here is a list of definitions that affect the interpretation of this statement, most of which are described in the NYT article:

• whether the suburbs are included as part of the "city", or more broadly, how the boundaries of the city are drawn. This matters because the crime rates in the suburbs are different from the crime rates in the city centers.
• whether the error in the count of population is comparable across all cities under consideration
• what is considered a crime
• what proportion of crime is reported
• what crime is classified as "violent"
• are there any exceptions

You might think the population of a city is an objective fact but it isn't. Definitions matter, and they matter a lot when you're comparing different cities.

Violent crime has worsened in South Bend since mayor Pete got elected in 2012.

In addition to the points listed above, we also need to know if any of those definitions changed between 2012 and 2019. Changing how something is defined almost always breaks the continuity of any analysis, thus undermining any trend analysis. I raised this point in my book Numbersense (link) in the chapter on obesity - if the medical community changes how we measure obesity, then all historical data are rendered useless so such proposals must be considered using great caution.

Further, we should find out whether violent crime was increasing or decreasing prior to 2012 in South Bend, whether violent crime was increasing or decreasing in other cities comparable to South Bend since 2012, whether violent crime was increasing or decreasing in the nation as a whole since 2012, etc.

***

Some readers of my book Numbersense (link) might be mystified why I spent so much energy discussing definitions - of grad school admissions statistics, of U.S. News ranking, of obesity measures, of unemployment rates, of inflation rates, of fantasy football metrics, etc., etc.

A key to numbersense - to judging the statistics and data thrown at you - is to understand clearly how the data are collected, and in particular, how any metric is defined.

The discussion of crime statistics by the New York Times makes this case cogently.

***

Now let me get to the absurdity of predictive policing.

A particularly popular and cited application of Big Data "for social good" is predictive policing. The goal of predictive policing is to prevent crime from happening.

The ideal of "Minority Report" is often hoisted on us. Wouldn't it be great if some AI or machine learning or predictive model can figure out who will be burglarizing your home next year, and have the police lock the would-be burglar up before s/he could commit the crime?

I have done a good amount of predictive modeling in my career so I am not saying what I'm saying because I'm a luddite. It's because of that experience that I am leading you down the following rabbit hole:

There are not many burglaries. According to Statista, the worst state (New Mexico) suffered 770 burglaries per 100,000 residents in 2018. This means that if the analyst is given a list of 100,000 New Mexicans chosen at random, a perfect model should label 770 of those as potential burglars. That is to say, we want the model to pick out 0.77 percent of the list.

A perfect model does not exist. Any model generates false positives. Because burglars are rare, any reasonably predictive model would need to red-flag many more than 0.77 percent of the list to have any chance of capturing all 770 potential burglars. Let's say the model points the finger at 1 percent of the list. That would mean 1,000 potential burglars. Since there should be only 770 burglars, the first thing we know is that at least 230 innocent people would be flagged by this model.

Next, think about what happens to the 1,000 flagged people. In the ideal of predictive policing, police would knock on their doors and arrest them for their future crimes. These people who haven't burglarized anyone would be duly jailed by courts who buy into predictive policing. (How would they defend themselves against an allegation of future crime?) 

We could say with certainty that those 1,000 did not subsequently commit burglary - given that they were incarcerated.

So humor me, and try to answer the following question: Of the 1,000 people flagged by this model, how many of these were accurately predicted?

 

 

 

A student skipping class might be grievously depressed. Claims like these discredit data science

This is Part 3 of a multi-part post on student surveillance technologies. You many want to read Part 1 and Part 2 first.

In the Washington Post article on student surveillance, one of the featured use cases is predictive analytics. In short, the location data are merged with other data to “score” students. Scoring is a form of rating by means of a formula.

The most famous example of predictive scoring is the ubiquitous and consequential credit scoring system. The credit score is a prediction of whether an individual is likely to repay the credit card company which is lending him or her money to purchase something today (without any collateral). I consider it a fundamental example of big data analytics, and have an extensive discussion of credit scoring in Chapter 2 of Numbers Rule Your World (link).

[Image credit: Frankieleon on Flickr]

***

Misunderstanding predictive analytics is flooding the marketplace with lots of defective products that harm people. I'm quite concerned by the language used by the business folks in the Post article to sing the praises of predicting student well-being. These statements reveal unfounded optimism about the accuracy of predictive analytics, and a shallow grasp of cause-effect relationships.

One vendor presented how their predictive model works: “a student avoiding the cafeteria might suffer from food insecurity or an eating disorder”. This kind of overly simplistic analysis is giving data science a bad name.

In reality, an extreme minority of people who avoid the cafeteria will be suffering from an eating disorder. Most scoring algorithms work with not one but multiple predictive factors, and even then, such an app will trigger many false alarms.

Even more shocking is the deduction that “a student skipping class might be grievously depressed”. It’s not the lack of modeling skill that concerns me here; it is the lack of common sense. For each student who skips class because of “grievous” depression, there are tons of non-depressed students who skip class. The insertion of “grievous” makes it even worse.

Behind any such deduction is a cause-effect model. It should be clear that Skip Class -> Grievously Depressed does not fly.

This may be a fallacy of transitivity. Grievously depressed -> Skip class is likely to hold. But reversing the causal order does not work.

***

Flawed data science products have consequences. Welfare checks are wasted on large numbers of healthy students. Students rightfully demand to be treated as adults. The truly grievously depressed students meanwhile may be missed.  

***

In Chapter 4 of Numbers Rule Your World (link), and Chapter 5 of Numbersense (link), I discuss ways to examine predictive models, and dealing with their inaccuracy.

New data on vaping

In my video about vaping (link to video, link to blog post), I explained why the e-cigarette industry’s claim that vaping saves lives is a half-truth. Their argument assumes people who vape would have died from smoking traditional cigarettes if they hadn't chosen to vape instead.

That’s true only when all of the following conditions apply:

1. The person would have smoked cigarettes if e-cigarettes were not available
2. The person switched completely to e-cigarettes
3. The person would have died from smoking-related illnesses had s/he smoked cigarettes
4. The person would not have died from vaping-related or other illnesses had s/he vaped
5. The person would not have undertaken any other measure, available now or in the future, to mitigate the risk of death

(1) fails for every vaper who would never have smoked cigarettes.

We expect this category to be sizeable – precisely because powerful marketing asserts that vaping is safe. The potential harm of a new product is hard to quantify at the start but we now know that people have died from it.

What’s easier to establish is how big this problem is. A recent Wall Street Journal article gives us the data: according to a federal survey, 20% of all middle- and high-school students in the U.S. “recently used e-cigarettes”, compared to 4% who used cigarettes. If the prevalence of cigarettes is stable, then four out of five vapers in school would never have shouldered the risk of smoking-related death but are newly exposed to possible harm through e-cigarettes.

***

In thinking about point (3), keep in mind that not all smokers die from smoking-related illnesses. You should only count the preventable deaths, that is to say, the person would have lived longer had s/he not smoked. What proportion of smokers actually die from smoking-related illnesses?

This question is harder to answer than one thinks. The basic analysis would be identifying who's the smoker, and whether the person died from a disease associated with smoking such as lung cancer. However, there are few diseases with a single cause so even if a smoker died from lung cancer, it's not assured that smoking is the cause of death. The counterfactual question we're interested in is whether this same person would have died from that disease had s/he not smoked. To complicate this further, research showed that if someone quit smoking early enough, the risk factor goes down by 90 percent. CDC has this page that shows some relevant numbers.

***

My video takes a deeper look into the different groups of people that may be impacted by vaping.

 

The research on SAT bias is great data science

I just put up a follow-up video on the SAT controversy, including a speculation about what might happen if the SAT were to be toppled from college admission. Link to video.

***

One of the key arguments against the SAT is its alleged bias against certain subgroups of test-takers. There has been a lot of research into ways to measure and combat such biases. Chapter 5 of Numbers Rule Your World (link) gets into the science behind this.

In the video, I discuss the following example featured also in the book

Is this test item biased against some subgroup? If so, which subgroup, by how much, and for what reason? It turns out that it is basically impossible to answer these questions purely by intuition. Test designers run experiments to measure the bias, and are often surprised by what they find.

This whole area of research is a great example of a practical application of data science. The issues range from how to collect the right data, and how to analyze such data to make causal conclusions. You can read about it in my book (link).

Or try the video first.

 

The Harden-Miami link: thinking about indirect metrics in an analysis plan

A co-worker pointed me to a Reddit thread that someone put up to showcase a small data-science side project - to establish whether NBA star James Harden's performance is affected by his "affinity for ... strip clubs" (thereafter, SC).

The analytical plan used in this project is typical of many data science projects. In this post (and possibly a few more), I raise issues that analysts should ponder when conceptualizing this type of analysis. These comments are designed to help those who are interested in advancing the sophistication of their analytical work. I am not re-analyzing the data, or debunking what's been done. I am impressed by what "AngryCentrist" has put together - starting with how AC gathered the data to how AC executed an analytical plan resulting in some conclusions.

***

Let me summarize the analytical game plan as follows: (you can read the gory details in this google doc)

1. The outcome variable is John Harden's on-court performance, expressed in an inverse scale (i.e. the larger the metric, the worse his performance). The explanatory variable is the quality rating of SCs in the cities where Harden played away games. The analysis looks at the correlation between these two variables.
2. Only the last four seasons of away games in which Harden played are used. In each season, there are usually one or two away games against any given team. Team and city are synonymous except for Los Angeles which fields two teams. (Brooklyn and New York are treated as separate cities.)
3. On-court performance is measured as the number of metrics on which Harden's in-city performance is worse than his season average. Only six metrics (e.g turnovers) are considered over four seasons so this performance metric is a number between 0 and 24. Zero means Harden out-performed his season average for each of the 6 metrics over each of the 4 seasons. By contrast, if Harden did worse than his season average for every metric over those four seasons, then the performance metric attains the maximum value of 24. The redditor, AngryCentrist, called this the "number of sub-par performances".
4. SC ratings are based on the first 10 results with customer ratings when doing a search on Google of the type "[city] strip clubs".
5. The analyst concluded that the two variables were positively correlated based on fitting a linear regression model. Higher SC quality rating is associated with worse on-court performance. The r-squared was 20 percent (which is not impressive).

***

For this post, I focus on the explanatory variable, the SC average rating, and the concept of "noise" in our data.

The SC average rating is an example of an indirect metric, a "noisy" proxy for what we want to measure. Notice that the outcome variable is about a particular person (Harden) while the explanatory variable measures a large set of people, of which Harden is allegedly a member.

The "noise" comes from several sources. If Harden was not a member of the set, then the explanatory variable has no direct connection with the outcome. So we hope that Harden not only visited SCs in all those cities, but also reviewed his experiences on Google, thus contributing to the average ratings.

If Harden did enjoy himself at the SCs and submitted reviews to Google, his ratings would still have been immaterial, just one vote in the average. Since the analysis only extracted "top 10" SCs based on a Google search, one expects the selected SCs are the more popular ones, which means that Harden's ratings would be even more insignificant.

If we don't require Harden's ratings to be included in these SC averages, we would at least require that his ratings - if revealed - to be highly correlated with the average ratings among Google reviewers. This assumption has two components - Harden would have to patronize the popular SCs, rather than ones that are more exclusive, or discreet; and Harden's opinion of his experiences there would have to be similar to that of the average patron.

Let's pretend that Harden visited one club prior to playing an away game in Miami. The average rating of the top 10 clubs in Miami is then not quite up to the task. That specific club might not even be part of the ten. This gap is also part of the "noise" of indirect measurement.

Timing is yet another source of noise. What if Harden engaged in the adult activity after the game rather than before the game?

Timing enters the picture in another way. Adult entertainment venues come and go, just like any other businesses. If Harden had visited a club before a match early in the NBA season, and that club subsequently shut down, then this club would not feature in the Google search, and would be absent from the average rating. Besides, new clubs that opened after Harden's visit would enter the average rating, if they become popular enough. Either of these situations causes the indirect metric to stray away from its proper value.

***

Here's the big picture. The indirect measure of average quality rating of SCs is a good proxy only under a large set of presumptions. None of the above conditions is directly observable, so each is presumed.

In general, the further away the measurement is from what is supposed to be measured, the more "noisy" is the measurement.

A workaround is to change our theory of the link between SC rating and Harden's performance. Maybe the link does not require physical visits to these establishments and subsequent reviews. Perhaps it is Harden's knowledge of a vibrant SC scene, regardless of active participation, that is doing the trick. This link though cannot be proven or disproven by the data, and thus is also presumed.

Assumptions are not avoidable. Verifiable assumptions are better than unverifiable ones. A good analysis plan should make the fewest, and weakest assumptions.