Big Data, Plainly Spoken (aka Numbers Rule Your World)

Grubhub's lead generation tactics leave bitter taste with restaurant owners

In my new 3-minute video, I discuss the controversy about the revelation that Grubhub has been playing a dirty trick on its restaurant customers.

As I explain in the video, what Grubhub did is similar to what Google has been doing with its search engine.

Most of these online businesses make money off "lead generation" - they bring prospective customers to brands. Restaurant owners hope Grubhub will bring them additional diners. These have to be incremental diners to justify the expense, which is a sizable 20-percent fee Grubhub assesses for each lead.

Grubhub has every incentive to count each order as an incremental lead. The dirty trick involves counting orders who should not be charged the lead-generation fee. Grubhub sets up shadow restaurant websites with similar URLs that pretend to be official websites. The diner usually does not know s/he is ordering from Grubhub's website and not the official restaurant website. This bit of deception qualifies the order for the lead generation fee - if the same diner had ordered from the restaurant's own website, Grubhub would have collected a much lower fee.

Similarly, lots of us search for brand names on Google, instead of going directly to the brands' websites. This leads some brand managers to buy their own brand names from Google. Google over the years has made it harder and harder to see which result is a paid ad. If we click on those ads, the brands pay Google a lead generation fee. Nevertheless, if we are searching for a brand's name, we will be visiting that brand's website, whether or not we pass through the Google toll booth!

In Numbersense (link), I talk about the importance of counterfactual thinking, and measuring incremental rather than absolute metrics.

Click here to see the video about the Grubhub controversy.

 

Know your data 25: Roundup of the fast-and-furious news

The genie is out of the bottle. The press is now reporting, almost daily, new discoveries of all the unspoken things tech companies have been doing with personal data collected from our usage of their products and services.

First up, the reporter from Washington Post wondered how much surveillance is happening on the Chrome browser. He learned that the browser loaded 11 thousand trackers in just one week. Horrified, he dumped Chrome (a Google product) for Firefox.

Traditionally, the website you visit installs a cookie (a type of tracker) to remember you: when you visit a page, and it automatically remembers who you are, that's the classic use of a cookie. Almost all of the trackers nowadays are invited guests of the website you visit, third parties who just want to follow you around.

I have more to say about the surveillance industry in this video. It's not just Chrome that you have to worry about.

***

When multiple websites like Facebook and Instagram temporarilly went down last week, news came out that Facebook automatically tags any photo you upload with "labels". During the downtime, the images did not load but the labels did. People saw things like "Photo may contain: two people, smiling, a beach". These are almost surely AI-created tags.

This is nothing new actually. In the past, you can see these labels if you are on a slow internet connection.

The real concern is the disclosure that Facebook automatically tags photos with people's names, using facial recognition technology. It appears that telling Facebook you don't want to be tagged means the tagging is moved underground, behind the scenes, out of your reach. I've been calling this the "illusion of control."

***

Another instance of illusion of control came by way of Amazon, which admitted that audio recordings of Alexa users are retained even after these users delete their accounts and data.

***

Next, a new email tool called Superhuman lets users to track whether the emails they sent have been opened by their recipients. This feature was on by default. This type of tracking uses "pixels" rather than cookies. A pixel is a tiny transparent "image" that is designed to hide from human eyes. When your browser pulls the pixel from its owner's server, you have made yourself known to that server.

This tracking feature has always been part of commercial email software. Whenever Amazon or your favorite brand sends you an email, analysts can tell when you open it, or interact with it. This data allow brands to assess the effectiveness of their emails.

Remember this the next time you pretend to your friend or colleague that you haven't seen his/her email.

 

Some things of interest

My review of Jim Albert's book Visualizing Baseball is on the sister blog. As I mentioned in the review, I thought a more appropriate title should be "Fast Intro to Baseball Analytics". Thus, the book may be of interest to some of the readers here.

***

Over the past year, I've been experimenting with the video medium. Here are some clips on various topics:

Analyzing observational data, containing a good example of using critical thinking

Alarming college admissions statistic, a nice example of data driving behavior in a feedback loop

Did you betray your friend today?, which explores how data flow through businesses

Two-part exploration of route optimizing apps like Waze: why they might not give you the fastest route, and why these apps are still great

To keep track of future releases, subscribe to our channel.

 

Inside the Black Box: analyzing observational data

 

Just posted a short video that explains one of the techniques used to work with observational data (or found data). This type of data is extremely common in the Big Data world. The data are being collected by some operational process, and in the indelible words of the late Hans Rosling, they are a bag of numerators without denominators. In this case, the database is for car crash fatalities. You only have the reported crashes linked to deaths but that database does not contain any information about crashes that do not have fatalities or safe driving.

Just like most scientific studies, the original researchers made a claim of statistical significance, i.e. they found something out of the normal. (There was an excess of fatalities on April 20.) However, a second research group took a different look at the data and demonstrated that what happened was more common than first thought.

How do statisticians measure how common something is? One takeaway is how to define the reference (control) group. Another takeaway is replications, repeating a style of analysis over different slices of the data.

Click here to see the video. Don't forget to subscribe to the channel to see future videos.

For a long-form discussion of what is covered in the video, see this blog post.

 

Deep-dive analysis goes off track: do women earn lower test scores because it's too cold?

A couple of researchers recently claimed to have found evidence of “a battle of the thermostat”: specifically, they argued that women perform significantly worse in a math test when it is administered at lower room temperatures while men’s performance does not decline. They did not find a similar gender gap in a word test.

The analysis strategy is similar to one that is employed by business analysts so readers here might be interested in how it went off track.

First, the researchers looked at the effect of temperature on test scores in aggregate. They found none. See the chart below, which is a replicate of similar charts in the research paper.

Note that in my chart, the scores have been standardized (to be explained below). For both math and word scores, the difference in scores between the extreme temperatures of 15 to 32 celsius (60 to 90 Fahrenheit) is smaller than 0.25 standard units, clearly not statistically significant by any measure.

The next step in the analysis strategy is to rotate through other factors that might influence test scores, such as gender, language spoken, and college major. In the business world, this is often called a deep-dive analysis. The question being asked is whether the effect of temperature on test scores differed by [factor X]. The researchers struck gold when splitting the data by gender. Here are the relevant charts:

In the top chart displaying math scores, it appears that the trend line for men is almost flat while women’s scores progressively increase as temperature increases. According to the usual convention of statistical significance, the effect is strong enough to be published. It is even possible to explain this observation using "battle of the thermostat".

***

Many other researchers have expressed skepticism that the reported effect can be replicated in other settings. See, for example, here.

I explored the data a bit more. Here is the raw data shown in side-by-side boxplots. I have arranged the sessions by the average temperatures, ordered from lowest to highest.

 

The following features caught my attention:

• The math scores of women (top chart, red boxplots) show unusually low variability. The dispersion of these scores is much below that of men’s math scores. The dispersion of these women’s math scores is also much below that of women’s word scores (bottom chart, red boxplots).
• Focusing only on the math scores (top chart), and inspecting the data session by session, from the right to the left, I found that the gender difference is hard to see (often obscured by the high variability of the male scores), except in a few sessions with the lowest temperatures (boxed area).

In terms of math scores, the shape of the distribution for men is clearly different from that for women. The scores for men were generally higher, and contain more extreme values on the high side. If a male and a female student score the same in absolute terms, the scores do not mean the same thing in relative terms – each relative to other students of the same gender.

***

One way to deal with this difference in variability is to standardize the scores by gender. This leads to the following chart, in which the scores are expressed relative to each gender’s distribution.

Notice how the math and word charts look almost identical. Also compare to the chart above using the raw data. This shows the fragility of statistical significance: the top chart shows marginally significant while the bottom chart shows not significant.

***

[Technical note: Here is a scatter plot of the standardized math scores against the raw scores, grouped by gender. You can see that the raw women's scores are in a narrower range, and that men's scores are more disperse and contain some high extremes.]

***

Deep-dive analysis sometimes lead to unexpected discoveries. Aggregate data sometimes mask subgroup differences. But at other times, the subgroup difference is a mirage. At the disaggregated level, the sample size is smaller, and there may also be a difference in variability.

For more discussion of subgroup analysis, see Chapter 3 of Numbers Rule Your World (link), in which I discuss how the insurance industry and the educational testing community tackle this problem.

 

 

 

 

False-positive science

The Atlantic reports on the dynamics of yet another group of scientists coming to grips with having wasted time and resources chasing down a dead end. (link)

It's a good read but long. Here is the gist of it:

Almost 20 years ago, some researchers made a huge splash by claiming to have discovered the "depression gene". The one gene eventually engendered 450 publications, and when counting related genes, over 1,000 publications. A recent large-scale "validation" study is likely to bring down the entire cottage industry - the depression gene is found to have little explanatory power for depression after all.

Gene data is an example of a type of Big Data. Big Data can be big in terms of the number of individuals in the dataset, or the number of measurements per individual. Two decades ago, the scale was attained by virtue of more measurements, not more individuals. The original study looked at about 300 or so individuals but each person's genome is vast.

The basic analysis is to compare the average depressed individual versus the average not-depressed individual in the sample. The data analyst sifts through large numbers of genes to find one or a few that are highly correlated with having depression. This is a classic fishing expedition, because of the large number of candidate genes, and also because of the large number of ways to define depression.

Such an analysis rides on top of a "model" of the world in which a single gene is responsible for depression. Over the years, the scientific community has discovered that this model is wrong. The new model assumes depression is indicated by a large set of genes each contributing a weak effect.

This type of structure is very hard to elicit from the typical datasets of the past - those that have numerous measurements on few individuals. Nowadays, we have data on lots of individuals but the sourcing of the data and other problems pose formidable challenges. It's also not clear how to use a model that spreads the blame thinly around a large number of genes for treatment.

Science is proceeding as it should - weak theories are overturned with more research. The article laments that it took 20 years to turn the tide, earlier warnings were ignored, the publish-or-perish culture in academia creates perverse incentives, retraction of scientific studies, etc.

***

I recently wrote about the challenge of Big Data expanding the variety of measurements here. Also, in writing Numbersense (link), I was concerned that the explosion of data collection causes an avalanche of false-positive science.

 

The limit of statistical understanding from adapted data

In the previous post, I described how some researchers found insights from a database of fatal car crashes. This dataset has all the markings of OCCAM data, which I use to summarize the characteristics of today's data.

Observational

• the data come from reports of crash fatalities, rather than experiments, surveys, or other data collection methods

No Controls

• the database only contains the cases, i.e. fatalities but not controls, which in this case should be drivers who did not suffer fatalities. The study design creates a type of control but as discussed in the previous post, the "controls" are still fatalities, just that they happened during different weeks. Such a study design requires the untested assumption that under normal circumstances, the frequency of fatalities to be constant within the three-week window of the study.

Seemingly Complete

• it is assumed that all crashes involving fatalities are reported accurately in the database. This assumption is frequently discovered to be wrong when the analyst digs into the data. A recent example is the Tesla auto-pilot analysis: even though in theory Tesla should have data on all its vehicles, the spreadsheet contains a large number of missing values.

Adapted

• the fatality data are collected for a number of uses, none of which is to investigate the potential effect of 420 Cannabis Day. Adapted data is sometimes called found data or data exhaust

Merged

• For this analysis, the researchers did not merge datasets. Most of the time, they do. For example, one of the commenters suggests looking at the effect of temperature. To do that requires merging local temperature data with the fatality data. Merging data creates all kinds of potential data quality issues.

***

In this post, we shall forget about the conclusion of the previous post, that April 20 may not be extraordinary. We accept that April 20 is an unusual day.

The first question to ask is: unusual in what way?

Let's look at the histogram again:

April 20 is unusual in having a higher number of fatal car crashes compared to the average of April 13 and 27.

That is what we learned from the data. Our next question is: why is April 20 worse?

According to the original study, the reason for the excess fatalities is excess cannabis consumption on April 20 because 420 is cannabis celebration day.

But at this point, we only have story time. Story time is the spinning of grand stories based on tiny morsels of data. The moment hits you in the second half of a newspaper article or research report after the author presents the data analyses, when you realize that story-telling has begun, and the report strays far from the evidence.

In this case, it's the link between excess fatalities and excess cannabis consumption that is tenuous. The problem goes back to OCCAM data, and lack of proper controls. If we could perform an experiment, the evidence would have been interpreted more directly.

The database of fatalities does not contain data on cannabis consumption. The original study has some info on "Drug police report" with over 60 percent of the cases listed as "not tested or not reported". This information is not used to argue one way or another about cannabis consumption.

The next step for this type of study is finding corroborating evidence to support the causal story. For example, are more of these accidents occurring around neighborhoods in which 420 Day is being celebrated? Can we find neighborhoods that only started celebrating 420 Day after a certain year and look at whether a jump in crash fatalities occurred after that year? Do people drive more or less frequently after they smoke weed? Are there proxies for cannabis consumption? (for example, maybe cannabis users are more likely to drive certain cars.) etc.

Harper and Palayew looked into whether the crash ratio got worse over time because cannabis consumption may have increased over time. They failed to see this, which weakens the conclusion.

 

DUIW 420: offering up 20 paper ideas pre-approved for prestigious journals

First, you have to read till the end for the 20 paper ideas. 

And if you're wondering about the acronym, it's Driving Under the Influence of Weed on 420 Day, which I learned from Andrew Gelman's blog is a day of celebration of cannabis.

Andrew's blog post is about the exemplary work done by Sam Harper and Adam Palayew, debunking a highly-publicized JAMA study that claimed that 420 Day is responsible for a 12 percent increase in fatal car crashes.

The discussion provides great fodder for examining how to investigate observational data, which is what most of Big Data is about. It is a cautionary tale for what not to do.

***

The blog begins with Harper/Palayew channeling Staples/Redelmeier, the authors of the study: "fatal motor vehicle crashes increase by 12% after 4:20 pm on April 20th (an annual cannabis celebration)."

This short sentence captures the gist of the original study but it omits an important detail: to what is the increase relative?

If we ran an experiment, we would recruit a group of drivers, and select half of them at random to smoke weed on April 20. Then, we would count what proportion of drivers suffered fatal car crashes after 4:20 pm. The analysis would be straightforward: what's the difference in proportions between the two groups? With such an experiment, it is possible to draw a causal conclusion.

Alternatively, we could conduct a case-control study. The cases are the drivers who suffered fatal car crashes on April 20. We collect demographic data on these drivers. Then, we define a set of "controls", drivers who did not suffer car crashes on April 20 but on average, have the same demographic characteristics as the cases. Next, we need data on cannabis consumption, preferably on April 20. We want to show that the level of cannabis consumption is significantly higher for cases than for controls.

(For further discussion of these analysis designs, see Chapter 2 of Numbers Rule Your World (link).)

The actual study was neither experiment nor case-control. It was a piece of pure data analysis, based on "found data". I like to call this "adapted data," the "A" in my OCCAM framework for Big Data - data collected for other purposes that the researcher has adapted for his/her own objectives. In this study, the adapted data come from a database of fatal car crashes.

So how was the adapted data analyzed? Harper/Palayew answer this question in their second description of the research:

Over 25 years from 1992-2016, excess cannabis consumption after 4:20 pm on 4/20 increased fatal traffic crashes by 12% relative to fatal crashes that occurred one week before and one week after.

The cases are the fatal car crashes that occurred after 4:20 pm on 420 Day. The comparison isn't to the drivers who did not suffer crashes on the same day. The reference group consisted of fatal car crashes that occurred after 4:20 pm on 4/13 and 4/27. The difference in the average number of crashes is taken to result from "excess cannabis consumption". 

Notice that such a conclusion requires a strong assumption. We must believe that absent 420 Day, 4/13, 4/20 and 4/27 ought to have the same fatal crash frequencies.  

***

You hopefully recognize that the analysis design for adapted data is on much shakier ground than either an experiment or a case-control study. 

Harper/Palayew's initial debunking focused on one issue: what's so special about April 20? To answer that, they repeated the same analysis on every day of the year. The following pretty chart summarizes their finding:

The red line is the line of no difference (between the analyzed day and the two reference days from the week before/after). Each vertical line is the range of estimate of the difference for a specific day of the year. The range for 4/20 is highlighted, and several other days with elevated fatal crash counts are labeled.

The chart was originally published here, with the following commentary: "There is quite a lot of noise in these daily crash rate ratios, and few that appear reliably above or below the rates +/- one week." Andrew adds: "Nothing so exciting is happening on 20 Apr, which makes sense given that total accident rates are affected by so many things, with cannabis consumption being a very small part."

While the chart looks cool, and sophisticated, the following histogram of the same data helps the reader digest the information. 

I took the daily estimates of the fatal crash ratios from Harper/Palayew's published data. Each ratio presents the crashes on the analysis day relative to the crashes on the two reference days. The histogram shows the day-to-day variability of the crash ratios, which is what we need to answer the question: how special is 4/20?

The histogram is roughly centered at 1.0 meaning no observed difference. The black vertical line shows the ratio for 4/20. It is leaning right - in fact, it is at the 94th-percentile. In classical terms, this is a p-value of 0.06, barely significant. 

The following 21 days have more extreme ratios than 4/20: 

Jul 4 Dec 23 Dec 21 Nov 21 Sep 1 Dec 20 Sep 2 Jul 3 Dec 31 Oct 31 Nov 23 Dec 18 Dec 6 Jul 14 Sep 4 Dec 22 Mar 17 May 25 Apr 1 Mar 7 Dec 19

Will JAMA editors accept one research paper for each of these days? The work is already done - the rest is story time. 

 

P.S. [4/27/2019] Replaced the first chart with a newer version from Harper's site. This version contains the point estimates that the other version did not. Those point estimates are used to generate the histogram.

How to use Kahneman-Tversky research from 1970s in the big data era

Just finished reading The Undoing Project by Michael Lewis, his bio of the Kahneman and Tversky duo who made many of the seminal discoveries in behavioral economics.

In Chapter 7, Lewis recounts one of their most celebrated experiments which demonstrated the “base rate fallacy.”

Here is one version of the experiment. The test subjects are asked to make judgments based on a vignette.

Psychologists have administered tests to 100 people, 70 of whom are lawyers and 30 are engineers.

(A) If one person is selected at random from this group, what is the chance that the selected person is a lawyer?

(B) Dick is selected at random from this group. Here is a description of him: “Dick is a 30 year old man. He is married with no children. A man of high ability and high motivation, he promises to be quite successful in his field. He is well liked by his colleagues.” What is the chance that Dick is a lawyer?

 

Those subjects who answered (A) made the right judgment, in accordance with the base rate of 70 percent.

The answer to (B) should be the same, since it shouldn't matter whether the random person is named Dick or not, and the generic description provides no useful information to determine Dick’s occupation. However, those subjects who answered (B) edited the chance down to about 50-50. The experiment showed that access to Dick’s description led people astray – to ignore the base rate. Note that the base rate here is the prior probability.

***

What are the practical applications of the KT experiment for business data analysts?

tl;dr

Before throwing the kitchen sink of variables (features) into your statistical (machine learning) models, review the literature on the base rate fallacy starting with Kahneman-Tversky experiments.

 

1. Adding more variables can make your predictions worse

Let's start with what kind of additional information is provided by Dick’s description. The sample size has not changed – it’s still one. The data expanded only in the number of variables (or features). Specifically, these eight additional variables:

X1 = age

X2 = gender

X3 = martial status

X4 = number of children

X5 = ability level

X6 = motivation level

X7 = expected level of success in field

X8 = popularity among colleagues

In today’s age of surveillance data, it is all too easy for any analyst to assemble more variables. The KT experiment shows that having more variables does not imply you have more useful information. Worse, those extra variables may distract you from the base rate, leading to worse predictions.

 

2. Machines are even more susceptible than humans

If humans are prone to such mistakes, should we use machines instead? Sadly, machines will perform worse.

Machines allow us to process even more variables at even greater efficiency. Instead of eight useless variables, you can now add 800 or even 8,000 useless variables about Dick. The machines will then inform you which subset of these variables “pop.” The more useless data you add in, the higher the chance you will encounter an accidental correlation.

 

Before throwing the kitchen sink of variables (features) into your statistical (machine learning) models, review the literature on the base rate fallacy starting with ground-breaking Kahneman-Tversky experiments.

 

 

Going bananas over this Seamless ad

Seamless, the online restaurant delivery service, has been running a series of fun ads on the New York subway that has a statistics theme. Here is a snapshot of one of them:

The text on the ad says:

The Most Potassium-Rich Neighborhood

MURRAY HILL

Based on the Number of Banana Orders

No One’s Cramping Here

***

This ad is tongue-in-cheek. But it's making a data-driven argument. So I started unpacking it.

The conclusion is “No one’s cramping here (in Murray Hill).” It’s an exaggeration so I’m going to read this as “Most people don’t cramp here in Murray Hill.”

The data behind this conclusion is much harder to nail down. One would think it should be the proportion of orders containing bananas in Murray Hill relative to the same in other neighborhoods. The ad uses the phrase “number of banana orders.” What does that mean? Is it “orders with at least one banana”? Or “orders of bananas only”? Or “total number of bananas ordered (across all orders)”?

Between the data and the conclusion is a long, windy path. Let me draw this out:

Assumption 1
All the neighborhoods have similar total populations so that by proportion of banana orders, Murray Hill also ranks #1.

Assumption 2
“Banana orders” is defined meaningfully. For the sake of argument, we’ll assume a banana order is an order that contains at least one banana.

Assumption 3
The data analyst used the appropriate address data. For the sake of argument, we'll assume that the delivery address is the source of the neighborhood data.

Assumption 4
Everyone who has a “banana order” through Seamless lives in the neighborhood to which the banana(s) were delivered. This further requires

Assumption 5
Everyone who has a “banana order” through Seamless works in the same neighborhood as they live. This distinction is important for daytime orders.

Assumption 6
Murray Hill residents who has a “banana order” through Seamless are just like other Murray Hill residents

Assumption 7
The name on each “banana order” is the one person who consumes the banana(s). No dogs ate the bananas, nor did a co-worker, family member, or anyone else not known to Seamless

Assumption 8
Eating bananas prevents or massively reduces cramping. The health effect of bananas requires

Assumption 9
Published scientific reports reach a strong consensus on the effect of bananas on cramping (highly unlikely); or, Seamless data show that those with a “banana order” report the absence of cramps (which requires primary research). The causal interpretation further requires

Assumption 10
Knowing that the people who made “banana orders” through Seamless would have suffered cramps had they not ordered and consumed those bananas. This counterfactual scenario is never observed, so instead, we accept

Assumption 10b
Knowing that the people who did not make a “banana order” through Seamless did suffer cramps. This requires

Assumption 11
The people who live in Murray Hill and did not make a “banana order” through Seamless also did not order bananas from a different shop, or otherwise consume bananas. In addition, we require

Assumption 12
No one who is part of this analysis benefited from any other anti-cramping remedy; or at the minimum,

Assumption 13
That people who have “banana orders” through Seamless, and those who don’t, are equally likely to have used other forms of anti-cramping remedy

Assumption 14
One banana is effective at stopping cramps, meaning there is no dose-response effect, the presence of which would require us to define “banana order” differently under Assumption 2.

The above assumptions fall into three groups: obviously false (e.g. Assumption 1); possibly true; and most likely true. The probability of the conclusion depends on the probabilities of these individual assumptions.

***

tl;dr

Most data-driven arguments consist of one part data, and many parts assumptions. An analyst should not fear making assumptions. Assumptions should be supported as much as possible.