Big Data, Plainly Spoken (aka Numbers Rule Your World)

Exercising fails to stop obesity, doh

Axios has an informative article about obesity, and the various remedies such as exercising, diets, and so on. Their headline is: "Health and wellness are booming, but we're fatter than ever." They have compiled some data, shown in a triplet of graphs:

The problem of obesity is complex, and fascinating from a data perspective. I devoted an entire chapter of Numbersense (link) to issues around measuring obesity.

There is much more underneath the surface than what is presented here. Let me unpack the layers of complexity.

Correlation is not Causation

The simplest issue to explain - just because statisticians have been screaming about it forever. If you look at the obesity chart and the gym chart, it is entirely accurate to say that gym membership has been rising in lock step with obesity rate during this decade. Both metrics rose by roughly 20%; and so it is very tempting to argue that going to gyms makes you fatter.

Of course, if you draw that conclusion, you've just been disinvited from the party of statisticians.

Ecological Fallacy

Here's the disturbing bit: the charts are also compatible with the opposite conclusion - that gym membership reduces obesity. This is an example of why it's so hard to interpret observational data.

Note that the data analyst collapsed a 2x2 matrix into two aggregate rates. Imagine four types of people: those with or without gym membership, crossed with those who are obese or not obese. When you're aware of the four types, you should realize that the rate of obesity, aggregated across gym membership, is not a great metric. It's pretty obvious that the obesity rate of those who are gym members is lower than that of those who do not have membership. The average rate paints them with the same brush.

In the same way, gym membership, aggregated across obese and not obese people, is not a great metric.

You can reasonably assume that obesity rate for the gym members should be lower than the average obesity rate, for example, if the average is 25%, then perhaps the obesity rate for non-gym members is 15%.

It's possible that the 15% rate has not changed over time but if the obesity rate of the non-gym-members increases, the overall obesity rate will increase (note that there are five times as many non-gym-members as there are gym members). The 15% rate for gym members could even have improved, and the overall obesity rate could still decline to 30% - it just requires the non-gym-members to get even more obese.

When aggregating the rates, some information is lost, and that weakens our ability to draw conclusions about individuals.

Indirect Metrics

Gym membership is not the same as gym usage. The gym's ability to influence obesity would require usage, not just membership.

 

CDC Diet Recommendation

The bit about the CDC complaining that people don't consume the recommended levels of fruits and vegetables makes me wonder if their problem formulation is overly simplistic. The dietary guidelines appear to be an optimization of nutritional benefits. But the real problem is to maximize nutritional benefits under a budget constraint. Each item in the basket of recommended foods delivers an amount of benefits at a level of cost. The total cost can't exceed the household budget.

For anyone taking a traditional class on optimization, "the diet problem" is often the first problem discussed. Here is one exposition of the diet problem.

 

Why stock X is falling today

In this tumultous market, there is a flash flood of journalists and pseudo-journalists explaining "why stock X is falling today". On a simple web search, the first bunch of results have X equaling Atlassian, Hanesbrand, WWE, Overstock.com, Southwest.

These authors all follow the same script. They look at each company's financial statements, or recently announced earnings report summaries, pull out some negative bits, and claim that those items are the causes of the sharp stock declines.

It's disturbing how little science there is in these pieces. Even the problem statement is completely warped. For example, the Motley Fool article takes the recently-popular format of a synopsis, starting with a one-liner on "What Happened." Eventually, the reader expects to hear an explanation of why it happened, and what does it mean for you.

So what happened according to the Motley Fool reporter?

"After the company reported third-quarter financial results that beat industry watchers' top- and bottom-line forecasts, shares in Southwest Airlines Co. (NYSE:LUV) are trading 8.2% lower at 2:15 p.m. EDT today."

This is one loaded statement. The causal link between the Southwest's 3rd quarter results and the drop in its stock price is already baked in. The reporter isn't asking if the 8.2% price drop is caused by the financial results, or how much of the drop can be explained by the results; the entire problem is framed such that the only possible conclusion is that the stock drop is caused by and only by those results!

PS. There is one scenario under which that problem statement is justified - if the stock price chart shows a steep plunge right after the earnings were announced. From the chart below, it appears that the drop started around noon, in fact, following a few hours in which the stock was trending up. (I'm not seeing where that 8.2% came from.)

***

How would a statistician approach this question?

One starts with the overall market decline. Then estimate any excess decline of airline stocks versus the overall market. And if the decline hasn't been fully explained, then look for factors directly related to Southwest.

 

 

New Facebook data breach, and the importance of Business Intelligence

 

On Friday, Facebook announced that hackers have gained access to personal data of at least 50 million users (see here for example). Analysts immediately connected this incident to the Cambridge Analytica scandal. How is this data breach different from the Cambridge scandal?

How the data was breached

One should take any announcement of how a data breach occurred with a grain of salt: it's obvious that companies will not publish anything that attracts lawsuits; plus, it's unclear that there is a penalty for lying about the real reasons for a data breach. There is no external auditing, and the companies control the narratives and statistics surrounding these events.

Based on what Facebook told us, the Cambridge Analytica scandal involves Facebook partners gaming the system to obtain data about Facebook users. At worst, it is a violation of community standards, i.e. claiming to be doing academic research. The research firm uses tools provided by Facebook to obtain the data. (The firm maintains that it disclosed to users that it was using the data for non-research purposes.)

In the present case, Facebook claims that a combination of coding bugs (i.e. unintended features) enabled unknown partners to gain access to the user's entire account. Not even that, but any accounts on third-party apps or websites that the user signs on to using Facebook.

 

How the breach was discovered

The current scandal demonstrates the value of the business intelligence/business analytics functions within companies. We were told that Facebook first realized that certain metrics were showing unusual trends, and upon investigation, they discovered the bugs.

This is entirely believable. That's what happens when you have good data reports. They surface anomalies. These then have to be investigated. These investigations are extremely tricky because all you know is the trends are different. There are a thousand reasons for the shift. The analyst's job is to establish a cause-effect. Especially since the development community adopted "agile" practices, all kinds of self-imposed changes are occurring all the time with no warnings. For a site as large and complex as Facebook, it takes a huge effort just to get a list of all site changes within some specified time window!

What complicates this situation more is that the vulnerability was traced to multiple bugs, not just one. I could imagine the twists and turns and the false alarms that were generated during the investigation.

The nature of this problem is no different from an investigator trying to chase down an e-coli outbreak, which is detailed in Chapter 2 of Numbers Rule Your World (link).

The data science community is guilty of talking down on the business intelligence function. There is a misperception that BI is for less skilled people doing boring things. The reality is there is more science in BI than in so-called data science (defined here as software engineering). Science, after all, is about figuring out why things are as they are. Engineers, by contrast, use our understanding of science to change the way things are.

 

You've been warned

In my debut Youtube video, released a few weeks ago, I explain how Facebook collects data about you. My biggest tip about protecting yourself is to not use Facebook to log onto other websites. Convenience is the drug that lures you into the trap.

Think about those one-password services. Instead of hundreds of password, you have one. So the thief needs only one password to assess hundreds of sites. Using Facebook to log onto other websites makes Facebook the centralized point of attack by the bad players.

Note that this is not limited to Facebook. Using Google, Yahoo, Amazon, etc. to log onto other services is no different!

If it's more convenient for you, it's also more convenient for your enemies. Remember that.

Also in the video, you will learn

• why did Facebook invent the Like button
• how do Facebook know what you are doing outside Facebook
• why not clicking doesn't mean you're not being tracked
• why you shouldn't use Facebook to log into other services

See the video here, and send me ideas for future episodes!

A purely mathematical theory of risk is not enough 2

,

This is the continuation of the last post on Chapter 5 of John Adams's excellent book on Risk. In the previous post, I discussed the problems of measurement that undermines the mathematical theory of risk. This post focuses on the particularly compelling example of traffic black spots, which reveals some risks in the statistical interpretation of risk data.

***

What Adams said:

Parts of the [road] network that have experienced bad spell are defined as accident black spots. When they are "treated", the numbers of accidents usually go down - but they probably would have gone down anyway.

He goes on to show a table of data used to identify traffic black spots. By convention, you list all the sites by the rate of accidents in descending order. The top sites with the most accidents are grouped together as black spots. In that table, 14 sites were identified as the worst sites with on average 3.53 accidents in a "before" period. In the "after" period, the accident rate dropped to 1.50.

Here's the catch: there was no treatment separating the "before" and "after" period. The analysis is an "A/A test," which I have frequently advocated as a way for data analysts (or our clients) to get a feel for "uncertainty". So the halving of the accident rate has zilch to do with any safety measure. One plausible explanation for the sudden drop in accident rate is the phenomenon of regression to the mean: if you select the worst sites for any given time, then in the next window of time, it is likely that those sites will experience below-average accident rates. (Vice versa, if you focus on the best sites for any given time, it is likely that those sites will suffer above-average accident rates in the after period.)

If you read the table of data carefully, you will notice that the 2000+ sites that were perfectly safe (zero accidents) in the "before" period subsequently suffered an increase in the accident rate in the "after" period. The average accident rate in the "after" period was 0.19, still low as an average number, but when multiplied by 2,000 sites, it can't be ignored. To see this even more clearly, in the "after' period, the black spots contributed 1.50x14 = 21 total accidents while the "no accident" spots contributed 2000x0.19 = 380 total accidents, 18 times more than the black spots!

The disease of the "Top X" analysis is an epidemic in our Big Data era; the OCCAM nature of these datasets worsens the disease (link). Pouring money on the "top" prospects is folly when one has not established causation of treatment and effect.

***

"Top X" type analysis is regrettably common in practice. You take a set of data and divide up the data into groups (often called deciles, quartiles, etc. depending on how many groups). Then you can compute some metric of the "lift". For example, in the "before" period, in the aggregate (> 2,600 sites), the average accident rate was 0.26. So if we predict that accidents will happen in the "black spots" (i.e. the worst class of sites), then the lift is 3.53/0.26 = 13.5 times. That is to say, the black spots are predicted to see 13.5 times more accidents than the average site.

Unexpectedly, in the "after" period, this lift number plunged to below 6 times. In other words, the predictive power is poor (when computed on data not even collected at the time of model-building). Given that no treatment was applied, if the predictive model captured the underlying dynamics, we should expect to see the lift number to be close to 13 times again. (We're assuming no strong known other factor that could cause the accident rates to collapse during the "after" period - in fact, in aggregate, the total number of accidents slightly increased.)

Here is the more subtle error: usually, the "after" period is after some safety measure is applied to the black spots (and not the other sites). The expected result is a drop in accident rate at the black spots, providing evidence that the safety measure actually caused the reduction in accidents. It is therefore very convenient for analysts (and policymakers who hire them) to conclude that the drop from 13.5 times to 6 times is evidence of the effectiveness of the safety measure. What the above analysis shows is that even in the absence of a safety measure, the accident rate could have dropped by half! So if you did apply a safety measure, it would need to produce a massive effect beyond the regression to the mean.

There are two reasons why analysts may fall prey to this type of bad analysis: one is confirmation bias - seeing in the data what you want or expect to see; the other is being what Nassim Taleb calls "fooled by randomness".

***

Adams goes on to discuss "accident migration". This is a second-order phenomenon: sites near to "treated black spots" often see an increase of accidents - the hypothesis is that since drivers take more caution when passing through black spots, they end up being more careless after the black spots are treated and so accidents spike up in nearby areas. (Adams does point out that it is possible that accident migration is not a separate thing from regression to the mean.)

Some related reading is the material on highway congestion in my book, Numbers Rule Your World (link). The problem of road congestion is slippery. If you successfully reduce the amount of traffic along one route, drivers will then change their habits to return that route to congestion - for example, they may have chosen to go along local roads before.

A purely mathematical theory of risk is not enough 1

I have been reading the excellent book by John Adams titled Risk (link). This is a geographer's treatment of a subject that is a staple of mathematics, particularly probability math. A mathematical treatment creates objects called probability distributions, which are then taken as complete representations of risk. Adams challenges that construct, bringing a social scientist's sensitivity to the table. In particular, he points out how the mathematics of risk is undermined by measurement issues (i.e. data issues) and statistical issues. He is not invalidating math, just pointing out large cracks that are often ignored.

I will provide a more comprehensive review of the book eventually. I'm very excited by Chapter 5, titled "Measuring Risk", and specifically the example of "traffic black spots". This example is very instructive for anyone who is interested in the practical implications and interpretation of risk measures.

The post got long so I have split it into two parts. The second part will be posted on Monday, and it concerns a delicious bit of analysis related to traffic black spots.

 

 

 

***

Here are some key sentences related to the quality of the collected data:

1. The connection between potentially fatal events [accidents] and actual fatal events is rendered extremely tenuous by the phenomonen of risk compensation.... avoiding action is taken, with the result that there are insufficient fatal accidents to produce a pattern that can serve as a reliable guide to the effect of specific safety interventions. As a consequence, safety planners seek out other measures of risk, usually - in ascending order of numbers but in decreasing order of severity - they are injury, morbidity, property damage and near misses.
2. It is much easier to achieve "statistical significance" in studies of accident causation if one uses large numbers rather than small. Safety researchers therefore have an understandable preference for non-fatal accident or incident data over fatality data... in exercising this preference, they usually assume that fatalities and non-fatal incidents will be consistent proportions of total casualities.
3. [Evoking some data from London] The correlation between fatality rates and injury rates is very weak. Is the weak correlation real or simply a recording phenomenon? How many injuries equal one life?
4. Uncertainty in the data increases as the severity of the injury decreases. The fatality statistics are almost certainly the most accurate and reliable of the road accident statistics... the categorization and recording of injuries is generally not informed by any evidence from a medical examination.... [A British Medical Asssociation report said that] only one in four casualties classified as seriously injured are, in fact, seriously injured and many of those classified as slightly injured are in fact seriously injured..... some 30 percent of traffic accident casualties seen in hospital are not reported to the police, and that at least 70 percent of cyclist casualties go unreported.

This last point is widely applicable in the data science/analytics world. We often have large amounts of unreliable data, and small amounts of more reliable data. What we hope to have - and we don't - is large amounts of reliable data. A lot of bad analyses results from assuming we have large amounts of reliable data.

 

Good job Bloomberg, and some comments on ad measurement

I'm continuing to see important articles as our journalists shine light on the tech companies. This Bloomberg article reveals a previously secret relationship between Google and Mastercard. It is important also as a reminder that the intense focus on Facebook is misguided - as other big tech companies engage in similar practices.

I'd recommend reading the article in full. Here I'd like to bring out a few key observations for you to chew on.

1) The payment processors have become part of the surveillance network that has been constructed by the tech industry. In the past (when I was working in that industry almost two decades ago), payment processors played the role of toll booths that happened to be a very lucrative business collecting cash as the transactions stream through. Users' purchase data were safeguarded and remained private.

Sometime in the last 5 to 10 years, the payment processors decided that the purchase data is a goldmine. They now monetize this data through mostly secret deals with all kinds of companies. Most card-users are not aware that their purchases are being disclosed to third parties.

An analogy is Gmail and other web-based email systems. Gmail in particular pioneered the concept of reading users' emails to extract information. It is now known that Gmail allows selected partners to extract all kinds of information from emails, such as shopping receipts, boarding passes, etc. We knew this because of the scandal about the Unroll.me service in which the service provider extracted shopping information from emails under the pretense of finding email lists from which users can be unsubscribed from.

 

2) Next time reporters hear the tech companies talk about "anonymized" data or hiding of "personally identifiable information", they really need to press the companies for details. Look out for clever word play. It is simply not possible to get value from these datasets without access to information about each person.

In this instance, Google is merging two datasets, the clickstream (what ads people clicked on) and Mastercard purchases, with the goal of claiming that person A purchased item X after clicking on some Google ad. The Mastercard data contain your card number, and/or your name. The clickstream data contain one or more of the following: your Google "cookie", your Google user name, or your email. You can't merge the datasets unless there is a common field. In this case, it will most likely be your name or email address. How could that be "anonymous"? Good question if you're asking it. And the reporters should be pushing on this point.

Further, if the data are fully anonymized, the potential utility of the data is much lower, and the price marketers are willing to pay for them is much lower. If I don't have your name and address, how can I send you a piece of junk mail?

 

3) While the article points out the violation of privacy of this partnership, it inadvertently legitimizes the highly dubious science behind matching the clickstream data and the Mastercard purchase data. The reporter described the program as "powerful" and "potent" but should have talked to third-party experts to understand this science better. I'll just list several major problems with this approach:

• As far as I know, Mastercard does not have line-item level details on our purchases. It may know that you spent $500 at Best Buy, but it doesn't know that that total includes an iPad and a TV, for example. So if the ad shows a TV, how does Google know that Best Buy purchase included a TV? The reporter claims that Google knows that someone bought "red lipstick" with her Mastercard. I find that dubious.
• Do you remember what ads you clicked on 30 days ago? 15 days ago? 5 days ago? This system counts any purchases made on Mastercard as long as 30 days ago as "caused" by an ad click.
• Does Google know what these users are doing on non-Google websites? For example, does Google know that the user who saw a Google ad 10 days ago also saw a Facebook ad for the same product 3 days ago?
• Does Google know anything about the users' interactions offline? For example, the user who clicked on the Google ad 10 days ago may have heard a radio ad, seen a TV ad, talked to a friend about the product, been handed a flyer at the mall, saw a piece of direct mail, etc.

The reporter suggested that Facebook, the other digital advertising behemoth, is also pursuing similar deals with payment processors. If Facebook were to adopt this same method for measuring ad performance, then both Facebook and Google would have claimed credit for the same purchase, as each would have matched that purchase to its own clickstream data. One solution to this particular issue is to merge the Facebook and Google data, which simply merges two surveillance networks into one extra-large surveillance network.

 

 

What's wrong with the Dropbox research?

In a mailing list I subscribe to, some users were not happy about the academic research using Dropbox data collected on academics, as written up by the co-authors in Harvard Business Review (link).

In a nutshell, the researchers obtained "anonymized" "project-folder-related" data from Dropbox on university-affiliated accounts, and did some simplistic bivariate correlations, and proceeded to draw several conclusions about "best practices" on "successful team collaborations." This type of research is very common in this "Big Data" age, and I have already written extensively about its many challenges.

OCCAM Data

This Dropbox dataset has all five descriptors of the OCCAM characteristics of "BIg Data". It is Observational, seemingly Complete, with no Controls, Adapted from its original use, and Merged (with data from Web of Science). These characteristics cause many problems with the analysis, which I describe below. For more on OCCAM data, check out this post, and my other posts on OCCAM data.

Found Data

Implicit in their analysis - and most other uses of "found data" - is the assumption of complete data. The authors believe that because their data consist of tens of thousands of researchers and 500,000 projects, they must have all the data. In this case, the authors knew that there are other platforms out there but they waved away the inconvenience. This implies they believe they have all of the "informative" data.

The authors also assumed that (a) all relevant collaborative research involves putting all relevant files on one of the major online platforms (e.g. nothing on Slack or emails) and (b) all project collaborators use one and only one platform. Further, they assume that everyone has highly organized and structured folder directories within Dropbox for which an external person who knows nothing about the projects, or a machine, can infer its contents. These problems arise because the researchers did not start with a research question and design the data collection. They chose to adapt "found data" to their own objectives.

Ecological Fallacy, and Story Time

A typical conclusion is "People at higher-performing universities seemed to share work more equally, based on the frequency of instances that collaborators accessed project folders." Top universities are ranked by the aggregate performance of all teams (that use Dropbox, and are identified correctly). It does not follow that every team at a top university is a top team.

This is an instance of "story time." A piece of data is offered about something related, then while the reader is dozing off, it is linked to a conclusion that is not directly supported by the data. That conclusion is elaborated by a lot of words hypothesizing why it must be true. In this case, they say "It’s likely that more frequent collaborations led to positive spillover of information, insights, and team dynamics from one project to another." But they provide no evidence at all for this last statement. That's just a story.

Proxy Unmasking

Here's another one of the conclusions: "People at higher-performing universities seemed to share work more equally, based on the frequency of instances that collaborators accessed project folders." They make a conclusion about work allocation between different collaborators but what they actually measured was the relative frequency at which collaborators accessed the project folders in Dropbox. That's a proxy measure, and it's convenient based on "found data", but not a good proxy.

Xyopia

It does not appear that a multiple regression model was run. The presentation walks through apparently a series of bivariate analyses. The word control does not appear in the entire article. So this work suffers from xyopia - in each analysis, the one explanatory variable being analyzed is presumed to be the chief and only variable that influences the outcome.

Causation Creep, and More Story Time

The authors made no attempt to establish causality at all. They just interpreted every correlation as causal. So every conclusion is "story time". They print one analysis of the data, then they draw a causal conclusion that one would believe only when half-asleep.

***

People are also upset about data privacy.

1. It does not appear that the academic users understood that using Dropbox means they are part of research studies.
2. People don't believe the data are truly anonymized. It's pretty clear that the anonymization can be easily reversed. Just take the HBR article for example. If they removed the names of the authors but retained information about the authors as: one junior faculty at Northwestern Univ. Business School, one senior faculty at Northwestern Univ. Business School, and one employee of Dropbox - I don't think you can find another article that fits those criteria. So is that anonoymous?
3. It's unclear if how they "anonymized folders" or analyzed them. There are folders with highly descriptive names, there are folders with partially descriptive names that only the collaborators may be able to decipher, and there are folders with names that do not identify the project (e.g. old_work). If they converted all folder names to alphanumeric strings, then all information about the contents of the folders is lost. If they don't convert those names, then there are clearly privacy concerns.

It's clear that some kind of IRB review is necessary to approve Big Data research projects to make sure privacy is protected.

Poll fodder inducing xyopia in elevators

Inside elevators in commercial buildings in New York, there are frequently screens that pump out commercials or infomercials all day long. I took a snapshot of this one that showed up last week:

These poll results annoy me in many ways. Let me explain.

The conclusion "free snacks is the key to happiness" is clearly wrong, even if the data were trustworthy (I'll talk about the data in a future post). I call this "xy myopia" or xyopia, in short. The writer of the poll has narrowly framed the problem: in this xyopic world, happiness hinges on just one factor - whether one's employer offers free food. In the real world, the happiness of an employee depends on a host of factors, of which free food is likely to be a minor one.

Xyopia arises from the reasonable concept of ceteris paribus ("all others equal"). Ceteris paribus is of little value in data analysis - it is often implicitly assumed by the analyst, but the assumption is false, which leads to bad conclusions.

In this case, the analyst just assumes that the two groups - employees getting free food, and those not getting free food - are equal on all other dimensions. So not true! The employer who offers free food is more likely to treat employees better across the board, such as providing more vacation days, better benefits, allowing work from home, etc. Besides, free food is only relevant in office settings (think Starbucks baristas). Certain demographic groups may value free food more than others. 

Unless it is established that the respondents who got free food are identical to the respondents who didn't in all other ways, the conclusion that free food is key results because it is the only factor you analyzed. This is xypoia.

(Not to mention, "free snacks" is not the same thing as "free food.")

***

Now, think about what question the poll writer asked that led to the headline "Free snacks is the key to happiness". It appears that they asked a pair of questions: "Does your employer offer free food?" and "Rate your happiness at work". Asking the first question frames your answer to the second question.

I'm going to make up my own poll:

A) Does your employer provide financial support for your personal development?

B) Rate your happiness at work (1 = extremely unhappy, 5 = extremely happy).

Based on this poll, I intend to conclude that paying for personal development is the key to employee happiness.

***

If you extend your vision to the basket of factors that affect employee happiness, such as their pay, benefits, treatment, work-life balance, family life, etc., you'll recognize more problems with the data analysis.

The poll result of 67% versus 56% is a strong result only if the gap is wider than for other factors, such as the poll I created above. I might find that 75% of the employees who get reimbursed for professional development answer extremely or very happy.

Also, consider a poll with three questions:

A) Does your employer provide free food?

B) If answer to A) is yes, do you consume the free food provided by your employer?

C) Rate your happiness at work (1 = extremely unhappy, 5 = extremely happy).

This creates three subsets of respondents (No - NA; Yes - No; and Yes - Yes) and leads to a better understanding of the causal mechanism.

***

Polls are easy to do but hard to do well.

 

 

They destroyed huge amounts of food again

Every year or so, another huge food recall makes the news, and it makes me sad.

The latest is the recall of 200 million eggs (link) due to salmonella "risk."

The identification of the cause of such disease outbreaks is a great case study of causal investigations, which I cover in detail in my book, Numbers Rule Your World. In that section, I also raise questions about the (in)sanity of these broad-based food recalls.

How does a food recall save a life?

All of the following have to happen:

• Some proportion of the recalled eggs has to be contaminated
• The contaminated eggs have to be purchased by consumers
• The contaminated eggs have not already been consumed or discarded before the recall (while the investigation was happening. From this epi curve, it looks like the greatest concentration of reported cases occurred in Nov/Dec 2017, four months ago.)
• Those who bought the eggs learn of the outbreak and subsequent recall
• Those who knew of the recall take action to return or dispose of their eggs
• Someone would have eaten those tainted eggs if they weren't recalled
• When eating the tainted eggs, the consumers do not cook them thoroughly
• The immune systems of those people who ate the tainted eggs fail to fight off the illness on their own
• The sick person decides to go to a hospital, thus identifying him/herself to CDC
• The sick person does not recover, typically due to having some pre-existing conditions that weakens his/her defence

The short version: the saved life has to be someone who would have died from consuming the tainted eggs were they not subject to this massive recall.

Each one of the above steps happen with some level of probability. For all these steps to happen at the same time is highly unlikely.

The economic fallout of dumping all those eggs is certain. The benefit of the recall in human lives is highly uncertain. So, it's not clear to me that these recalls are reasonable.

 

Overselling predictive models (Hint: future performance matters)

Those who have read my books or taken my courses would be the least bit surprised by the news that just came out about the naked emperor of Chicago's much ballyhooed predictive algorithm for child abuse. Chicago Tribune has the story (link).

Key points:

• Data scientists (over-)sold an algorithm to predict children "at risk for serious injury or death."
• The director of the government agency that purchased this product concluded after two years that "[the model] isn't predicting much."
• At least $366K were spent i.e. wasted on this program. This number most likely does not account for the costs of support staff and infrastructure and the actual waste described below.
• "More than 4,100 Illinois children were assigned a 90 percent or greater probability of death or injury. And 369 youngsters, all under age 9, got a 100 percent chance of death or serious injury in the next two years" [a false positive problem - not to bring up a sore subject but several infamous outlets proclaimed that Hillary Clinton would win the Presidency with 90% or higher probability!]
• "high-profile child deaths kept cropping up with little warning from the predictive analytics software; Predictive analytics (wasn't) predicting any of the bad cases" [a false negative problem]
• Selling the dream: "If it is possible to use big data to spotlight a child in trouble and intervene before he or she is hurt, then doing so is government's moral obligation, advocates for the technology say."
• The algorithm, despite being paid for by public funds, is hidden from public view but the Tribune report pieces this together: "Eckerd [the vendor] retrospectively analyzes thousands of closed abuse cases and from them draws data points that are highly correlated with serious harm. The parents' ages could be a factor — or their previous criminal records, evidence of substance abuse in the home, or the presence of a new boyfriend or girlfriend."

The story does not invalidate the practice of predictive analytics but points out pervasive problems within the industry. Most importantly, we are over-hyping this technology. I cover the statistics of such models in depth in the marketing chapters of Numbersense (link). Predictive models are impressive in a relative sense (in marketing, we use the term "lift") but not usually in an absolute sense. Every social science based predictive model I have seen misses a sizeable number of targets while falsely labeling lots of "cases".

When a vendor does not report on accuracy metrics, or when such metrics are not interpretable in practice, or when the metrics are generated by third parties, or when metrics are computed purely from historical data, one has to be very careful about separating the good products from the scams.

***

It bears repeating that the "big data" used in this type of analysis has multiple complications which I collectively describe as OCCAM.

Observational - we cannot run an experiment and assign people to have criminal records, or other characteristics at random, so it is tough to get any read on causal mechanisms. Thus, there is a good chance to suffer from spurious correlations.

Seemingly Complete - but not really. Many cases are unreported and not in the training data. One of the excuses used by Eckerd when confronted with inaccuracy is that certain children are not even scored because they do not appear in the data. That is technically correct but does not change the fact that the model did not predict those deaths.

No Controls - many predictive algorithms do not use controls explicitly. Another issue is that the possible controls all come from cases reported to the authorities. Without a doubt, if we take the profiles of the at-risk children as determined by the algorithm, we will find many other children in the general population that do not show up in the database at all.

Adapted - the data collection process was not designed specifically to support predictive modeling.

Merged - various datasets get merged during the analysis, which introduces errors.

More discussion of OCCAM here and here.

***

While I appreciate that the Chicago Tribune wrote and published this article, this is yet another media report on predictive modeling using "big data" that does not contain any data or quantified metric of predictive accuracy.

The two chapters most relevant to this post are Chapter 4 in Numbers Rule Your World (link) and Chapter 5 in Numbersense (link).