Big Data, Plainly Spoken (aka Numbers Rule Your World)

Wage inequality and its sources

In the sister blog, I featured a data graphic showing the difference in pay levels between white male workers in the U.S. and women workers of various races. That post discusses purely visual matters, in effect accepting the analysis as given. In this post, I take a deeper look at the underlying data analysis.

Let us review the analysis strategy behind the chart, and then discuss why this simple strategy is not particularly insightful.

The starting point of the analyst is income data collected by the Census Bureau. Even here, income is measured in several ways. It appears that using personal income (as opposed to household or family income) is the most appropriate here. There is an additional complication of how to handle "missing values", which may arise in this context because someone is not employed and thus earn zero income. When one says "median income", does one include or exclude the zero earners? What about those who only worked part of the year?

After those questions are addressed, one can work with median incomes, computed at the right level of aggregation. The report linked to by Business Insider only contains aggregation by race, and aggregation by gender, each calculated separately. What is needed is a cross-tabulation. It is not possible to obtain the median income of Asian women from the median income of Asians and the median income of women - unless the analyst makes unwarranted assumptions.

So the analyst learns that white men made $46,000 in 2017 while Asian women made $27,000 and Hispanic women made $21,000. This data can be plotted directly, or after computing the race-gender discount (off the white male wage level).

***

The analyst wants to make this data come alive by using a different unit, the number of days worked.

One way to achieve this is by converting the annual salary to daily salary, then computing how many days the median Asian woman must work in order to earn $46,000, the median for the white man. This is roughly 611 days, which suggests that the Asian woman must work 246 extra days.

In the above description, I have seamlessly papered over several annoying details. I have assumed - without checking with reality - that everyone works 365 days a year. In fact, no one worked 365 days a year. And even if I obtained the correct value of the average number of days worked, call it X, it will also be the case that X will vary between race and gender. Thus, I made a further implicit assumption that such variance is not large enough to bother about.

I justify this lack of care because I rounded the median salaries to the nearest thousand. The questions I raised above ought to concern those analysts who insist on printing estimates in decimals. Is it possible to attain that level of precision while making simplifying assumptions?

***

Also notice that the analysis strategy is counterfactual in nature - it requires conjuring up a hypothetical scenario. It's a comparison of a white man who works exactly one year, and a woman (of any race) who works till she earns $46,000 or whatever is the current median wage for white men.

The notion of "extra days" is an invention since there are only 365 days in a year, and the women can never catch up unless the white guys stop working.

***

When comparing white men and Asian women, both gender and race are shifted. The Business Insider story leads readers to attribute the wage gap to primarily gender but unless we see the median salaries for Asian men, Hispanic men and black men, one can't be sure.

Most likely, there is a gender "effect" as well as a race "effect". The gender effect may even be differently sized for each race. This is known as an "interaction" effect.

***

Finally, there are even more factors to be considered. It is well known that at least some of the wage gap is explained by the difference in the mix of jobs and industries that men and women tend to be employed in. So one can't conclude discrimination without further investigation.

Unequal pay for equal work is discrimination but unequal pay for unequal work need not be.

***

Now, check out my comments about the calendar visualization of the wage gaps by Business Insider.

 

More Grubhub theatrics, with Yelp as supporting actor

Just recently, I made a short clip about how Grubhub is extracting lead-generation fees from unsuspecting restaurant owners by setting up a network of shadow websites (and phone numbers). Diners who thought they were ordering directly from restaurants were instead shuffled through the Grubhub toll booth. Click here to see how this works.

 

 

Now, Vice discovered yet another of Grubhub's toll booths. This time, the toll booth is set up on the Yelp superhighway. When a Yelp user clicks on the restaurant's phone number on its Yelp page, a pop-up shows up, with two options, one of which redirects the user to a shadow number owned by Grubhub. These numbers are not labelled "direct" and "Grubhub" but "general questions" and "deliveries and orders". So now both Yelp and Grubhub are making lead-generation money off the restaurant owner.

***

This dispute is about causality! And because causality is tough to establish, it creates a gray zone of disagreement.

Ideally, the restaurant owner pays for orders caused by Grubhub's marketing activities. By cause, we mean the orders would not have materialized without Grubhub's marketing. In the examples we saw, it's highly unlikely that Grubhub did anything to cause those orders. That's because those diners thought they were ordering directly from the restaurants - they are quietly re-routed to the shadow websites and phone numbers set up by Grubhub, sometimes without even the restaurants' knowledge.

It is the secrecy that gives the game away. If Grubhub's causal value is clear, it can form open partnerships with the restaurants without resorting to trickery.

As I pointed out in the video, the majority of the digital marketing industry relies on similar tactics. Search engines do not typically send you directly to the webpage you clicked on - you are often rerouted through the search engine's server, so that the search engine can "track" the click and use the paper trail to receive lead-generation money. Anyone passing through this toll booth is counted as "causal" but in reality, many of these users would have found their way to the webpage even if the search engine didn't exist. (Consider going directly to Macys.com instead of typing Macys into a search engine.)

***

This is a great example of how data, algorithms and software are silently running our lives, and often to the detriment of those who don't understand what's going on. Our video series is a small effort to help you stay in front of these data-driven technologies.

The more you know, the more you can leverage its power, and avoid its harms.

Grubhub's lead generation tactics leave bitter taste with restaurant owners

In my new 3-minute video, I discuss the controversy about the revelation that Grubhub has been playing a dirty trick on its restaurant customers.

As I explain in the video, what Grubhub did is similar to what Google has been doing with its search engine.

Most of these online businesses make money off "lead generation" - they bring prospective customers to brands. Restaurant owners hope Grubhub will bring them additional diners. These have to be incremental diners to justify the expense, which is a sizable 20-percent fee Grubhub assesses for each lead.

Grubhub has every incentive to count each order as an incremental lead. The dirty trick involves counting orders who should not be charged the lead-generation fee. Grubhub sets up shadow restaurant websites with similar URLs that pretend to be official websites. The diner usually does not know s/he is ordering from Grubhub's website and not the official restaurant website. This bit of deception qualifies the order for the lead generation fee - if the same diner had ordered from the restaurant's own website, Grubhub would have collected a much lower fee.

Similarly, lots of us search for brand names on Google, instead of going directly to the brands' websites. This leads some brand managers to buy their own brand names from Google. Google over the years has made it harder and harder to see which result is a paid ad. If we click on those ads, the brands pay Google a lead generation fee. Nevertheless, if we are searching for a brand's name, we will be visiting that brand's website, whether or not we pass through the Google toll booth!

In Numbersense (link), I talk about the importance of counterfactual thinking, and measuring incremental rather than absolute metrics.

Click here to see the video about the Grubhub controversy.

 

Know your data 25: Roundup of the fast-and-furious news

The genie is out of the bottle. The press is now reporting, almost daily, new discoveries of all the unspoken things tech companies have been doing with personal data collected from our usage of their products and services.

First up, the reporter from Washington Post wondered how much surveillance is happening on the Chrome browser. He learned that the browser loaded 11 thousand trackers in just one week. Horrified, he dumped Chrome (a Google product) for Firefox.

Traditionally, the website you visit installs a cookie (a type of tracker) to remember you: when you visit a page, and it automatically remembers who you are, that's the classic use of a cookie. Almost all of the trackers nowadays are invited guests of the website you visit, third parties who just want to follow you around.

I have more to say about the surveillance industry in this video. It's not just Chrome that you have to worry about.

***

When multiple websites like Facebook and Instagram temporarilly went down last week, news came out that Facebook automatically tags any photo you upload with "labels". During the downtime, the images did not load but the labels did. People saw things like "Photo may contain: two people, smiling, a beach". These are almost surely AI-created tags.

This is nothing new actually. In the past, you can see these labels if you are on a slow internet connection.

The real concern is the disclosure that Facebook automatically tags photos with people's names, using facial recognition technology. It appears that telling Facebook you don't want to be tagged means the tagging is moved underground, behind the scenes, out of your reach. I've been calling this the "illusion of control."

***

Another instance of illusion of control came by way of Amazon, which admitted that audio recordings of Alexa users are retained even after these users delete their accounts and data.

***

Next, a new email tool called Superhuman lets users to track whether the emails they sent have been opened by their recipients. This feature was on by default. This type of tracking uses "pixels" rather than cookies. A pixel is a tiny transparent "image" that is designed to hide from human eyes. When your browser pulls the pixel from its owner's server, you have made yourself known to that server.

This tracking feature has always been part of commercial email software. Whenever Amazon or your favorite brand sends you an email, analysts can tell when you open it, or interact with it. This data allow brands to assess the effectiveness of their emails.

Remember this the next time you pretend to your friend or colleague that you haven't seen his/her email.

 

Some things of interest

My review of Jim Albert's book Visualizing Baseball is on the sister blog. As I mentioned in the review, I thought a more appropriate title should be "Fast Intro to Baseball Analytics". Thus, the book may be of interest to some of the readers here.

***

Over the past year, I've been experimenting with the video medium. Here are some clips on various topics:

Analyzing observational data, containing a good example of using critical thinking

Alarming college admissions statistic, a nice example of data driving behavior in a feedback loop

Did you betray your friend today?, which explores how data flow through businesses

Two-part exploration of route optimizing apps like Waze: why they might not give you the fastest route, and why these apps are still great

To keep track of future releases, subscribe to our channel.

 

Inside the Black Box: analyzing observational data

 

Just posted a short video that explains one of the techniques used to work with observational data (or found data). This type of data is extremely common in the Big Data world. The data are being collected by some operational process, and in the indelible words of the late Hans Rosling, they are a bag of numerators without denominators. In this case, the database is for car crash fatalities. You only have the reported crashes linked to deaths but that database does not contain any information about crashes that do not have fatalities or safe driving.

Just like most scientific studies, the original researchers made a claim of statistical significance, i.e. they found something out of the normal. (There was an excess of fatalities on April 20.) However, a second research group took a different look at the data and demonstrated that what happened was more common than first thought.

How do statisticians measure how common something is? One takeaway is how to define the reference (control) group. Another takeaway is replications, repeating a style of analysis over different slices of the data.

Click here to see the video. Don't forget to subscribe to the channel to see future videos.

For a long-form discussion of what is covered in the video, see this blog post.

 

Deep-dive analysis goes off track: do women earn lower test scores because it's too cold?

A couple of researchers recently claimed to have found evidence of “a battle of the thermostat”: specifically, they argued that women perform significantly worse in a math test when it is administered at lower room temperatures while men’s performance does not decline. They did not find a similar gender gap in a word test.

The analysis strategy is similar to one that is employed by business analysts so readers here might be interested in how it went off track.

First, the researchers looked at the effect of temperature on test scores in aggregate. They found none. See the chart below, which is a replicate of similar charts in the research paper.

Note that in my chart, the scores have been standardized (to be explained below). For both math and word scores, the difference in scores between the extreme temperatures of 15 to 32 celsius (60 to 90 Fahrenheit) is smaller than 0.25 standard units, clearly not statistically significant by any measure.

The next step in the analysis strategy is to rotate through other factors that might influence test scores, such as gender, language spoken, and college major. In the business world, this is often called a deep-dive analysis. The question being asked is whether the effect of temperature on test scores differed by [factor X]. The researchers struck gold when splitting the data by gender. Here are the relevant charts:

In the top chart displaying math scores, it appears that the trend line for men is almost flat while women’s scores progressively increase as temperature increases. According to the usual convention of statistical significance, the effect is strong enough to be published. It is even possible to explain this observation using "battle of the thermostat".

***

Many other researchers have expressed skepticism that the reported effect can be replicated in other settings. See, for example, here.

I explored the data a bit more. Here is the raw data shown in side-by-side boxplots. I have arranged the sessions by the average temperatures, ordered from lowest to highest.

 

The following features caught my attention:

• The math scores of women (top chart, red boxplots) show unusually low variability. The dispersion of these scores is much below that of men’s math scores. The dispersion of these women’s math scores is also much below that of women’s word scores (bottom chart, red boxplots).
• Focusing only on the math scores (top chart), and inspecting the data session by session, from the right to the left, I found that the gender difference is hard to see (often obscured by the high variability of the male scores), except in a few sessions with the lowest temperatures (boxed area).

In terms of math scores, the shape of the distribution for men is clearly different from that for women. The scores for men were generally higher, and contain more extreme values on the high side. If a male and a female student score the same in absolute terms, the scores do not mean the same thing in relative terms – each relative to other students of the same gender.

***

One way to deal with this difference in variability is to standardize the scores by gender. This leads to the following chart, in which the scores are expressed relative to each gender’s distribution.

Notice how the math and word charts look almost identical. Also compare to the chart above using the raw data. This shows the fragility of statistical significance: the top chart shows marginally significant while the bottom chart shows not significant.

***

[Technical note: Here is a scatter plot of the standardized math scores against the raw scores, grouped by gender. You can see that the raw women's scores are in a narrower range, and that men's scores are more disperse and contain some high extremes.]

***

Deep-dive analysis sometimes lead to unexpected discoveries. Aggregate data sometimes mask subgroup differences. But at other times, the subgroup difference is a mirage. At the disaggregated level, the sample size is smaller, and there may also be a difference in variability.

For more discussion of subgroup analysis, see Chapter 3 of Numbers Rule Your World (link), in which I discuss how the insurance industry and the educational testing community tackle this problem.

 

 

 

 

False-positive science

The Atlantic reports on the dynamics of yet another group of scientists coming to grips with having wasted time and resources chasing down a dead end. (link)

It's a good read but long. Here is the gist of it:

Almost 20 years ago, some researchers made a huge splash by claiming to have discovered the "depression gene". The one gene eventually engendered 450 publications, and when counting related genes, over 1,000 publications. A recent large-scale "validation" study is likely to bring down the entire cottage industry - the depression gene is found to have little explanatory power for depression after all.

Gene data is an example of a type of Big Data. Big Data can be big in terms of the number of individuals in the dataset, or the number of measurements per individual. Two decades ago, the scale was attained by virtue of more measurements, not more individuals. The original study looked at about 300 or so individuals but each person's genome is vast.

The basic analysis is to compare the average depressed individual versus the average not-depressed individual in the sample. The data analyst sifts through large numbers of genes to find one or a few that are highly correlated with having depression. This is a classic fishing expedition, because of the large number of candidate genes, and also because of the large number of ways to define depression.

Such an analysis rides on top of a "model" of the world in which a single gene is responsible for depression. Over the years, the scientific community has discovered that this model is wrong. The new model assumes depression is indicated by a large set of genes each contributing a weak effect.

This type of structure is very hard to elicit from the typical datasets of the past - those that have numerous measurements on few individuals. Nowadays, we have data on lots of individuals but the sourcing of the data and other problems pose formidable challenges. It's also not clear how to use a model that spreads the blame thinly around a large number of genes for treatment.

Science is proceeding as it should - weak theories are overturned with more research. The article laments that it took 20 years to turn the tide, earlier warnings were ignored, the publish-or-perish culture in academia creates perverse incentives, retraction of scientific studies, etc.

***

I recently wrote about the challenge of Big Data expanding the variety of measurements here. Also, in writing Numbersense (link), I was concerned that the explosion of data collection causes an avalanche of false-positive science.

 

The limit of statistical understanding from adapted data

In the previous post, I described how some researchers found insights from a database of fatal car crashes. This dataset has all the markings of OCCAM data, which I use to summarize the characteristics of today's data.

Observational

• the data come from reports of crash fatalities, rather than experiments, surveys, or other data collection methods

No Controls

• the database only contains the cases, i.e. fatalities but not controls, which in this case should be drivers who did not suffer fatalities. The study design creates a type of control but as discussed in the previous post, the "controls" are still fatalities, just that they happened during different weeks. Such a study design requires the untested assumption that under normal circumstances, the frequency of fatalities to be constant within the three-week window of the study.

Seemingly Complete

• it is assumed that all crashes involving fatalities are reported accurately in the database. This assumption is frequently discovered to be wrong when the analyst digs into the data. A recent example is the Tesla auto-pilot analysis: even though in theory Tesla should have data on all its vehicles, the spreadsheet contains a large number of missing values.

Adapted

• the fatality data are collected for a number of uses, none of which is to investigate the potential effect of 420 Cannabis Day. Adapted data is sometimes called found data or data exhaust

Merged

• For this analysis, the researchers did not merge datasets. Most of the time, they do. For example, one of the commenters suggests looking at the effect of temperature. To do that requires merging local temperature data with the fatality data. Merging data creates all kinds of potential data quality issues.

***

In this post, we shall forget about the conclusion of the previous post, that April 20 may not be extraordinary. We accept that April 20 is an unusual day.

The first question to ask is: unusual in what way?

Let's look at the histogram again:

April 20 is unusual in having a higher number of fatal car crashes compared to the average of April 13 and 27.

That is what we learned from the data. Our next question is: why is April 20 worse?

According to the original study, the reason for the excess fatalities is excess cannabis consumption on April 20 because 420 is cannabis celebration day.

But at this point, we only have story time. Story time is the spinning of grand stories based on tiny morsels of data. The moment hits you in the second half of a newspaper article or research report after the author presents the data analyses, when you realize that story-telling has begun, and the report strays far from the evidence.

In this case, it's the link between excess fatalities and excess cannabis consumption that is tenuous. The problem goes back to OCCAM data, and lack of proper controls. If we could perform an experiment, the evidence would have been interpreted more directly.

The database of fatalities does not contain data on cannabis consumption. The original study has some info on "Drug police report" with over 60 percent of the cases listed as "not tested or not reported". This information is not used to argue one way or another about cannabis consumption.

The next step for this type of study is finding corroborating evidence to support the causal story. For example, are more of these accidents occurring around neighborhoods in which 420 Day is being celebrated? Can we find neighborhoods that only started celebrating 420 Day after a certain year and look at whether a jump in crash fatalities occurred after that year? Do people drive more or less frequently after they smoke weed? Are there proxies for cannabis consumption? (for example, maybe cannabis users are more likely to drive certain cars.) etc.

Harper and Palayew looked into whether the crash ratio got worse over time because cannabis consumption may have increased over time. They failed to see this, which weakens the conclusion.

 

DUIW 420: offering up 20 paper ideas pre-approved for prestigious journals

First, you have to read till the end for the 20 paper ideas. 

And if you're wondering about the acronym, it's Driving Under the Influence of Weed on 420 Day, which I learned from Andrew Gelman's blog is a day of celebration of cannabis.

Andrew's blog post is about the exemplary work done by Sam Harper and Adam Palayew, debunking a highly-publicized JAMA study that claimed that 420 Day is responsible for a 12 percent increase in fatal car crashes.

The discussion provides great fodder for examining how to investigate observational data, which is what most of Big Data is about. It is a cautionary tale for what not to do.

***

The blog begins with Harper/Palayew channeling Staples/Redelmeier, the authors of the study: "fatal motor vehicle crashes increase by 12% after 4:20 pm on April 20th (an annual cannabis celebration)."

This short sentence captures the gist of the original study but it omits an important detail: to what is the increase relative?

If we ran an experiment, we would recruit a group of drivers, and select half of them at random to smoke weed on April 20. Then, we would count what proportion of drivers suffered fatal car crashes after 4:20 pm. The analysis would be straightforward: what's the difference in proportions between the two groups? With such an experiment, it is possible to draw a causal conclusion.

Alternatively, we could conduct a case-control study. The cases are the drivers who suffered fatal car crashes on April 20. We collect demographic data on these drivers. Then, we define a set of "controls", drivers who did not suffer car crashes on April 20 but on average, have the same demographic characteristics as the cases. Next, we need data on cannabis consumption, preferably on April 20. We want to show that the level of cannabis consumption is significantly higher for cases than for controls.

(For further discussion of these analysis designs, see Chapter 2 of Numbers Rule Your World (link).)

The actual study was neither experiment nor case-control. It was a piece of pure data analysis, based on "found data". I like to call this "adapted data," the "A" in my OCCAM framework for Big Data - data collected for other purposes that the researcher has adapted for his/her own objectives. In this study, the adapted data come from a database of fatal car crashes.

So how was the adapted data analyzed? Harper/Palayew answer this question in their second description of the research:

Over 25 years from 1992-2016, excess cannabis consumption after 4:20 pm on 4/20 increased fatal traffic crashes by 12% relative to fatal crashes that occurred one week before and one week after.

The cases are the fatal car crashes that occurred after 4:20 pm on 420 Day. The comparison isn't to the drivers who did not suffer crashes on the same day. The reference group consisted of fatal car crashes that occurred after 4:20 pm on 4/13 and 4/27. The difference in the average number of crashes is taken to result from "excess cannabis consumption". 

Notice that such a conclusion requires a strong assumption. We must believe that absent 420 Day, 4/13, 4/20 and 4/27 ought to have the same fatal crash frequencies.  

***

You hopefully recognize that the analysis design for adapted data is on much shakier ground than either an experiment or a case-control study. 

Harper/Palayew's initial debunking focused on one issue: what's so special about April 20? To answer that, they repeated the same analysis on every day of the year. The following pretty chart summarizes their finding:

The red line is the line of no difference (between the analyzed day and the two reference days from the week before/after). Each vertical line is the range of estimate of the difference for a specific day of the year. The range for 4/20 is highlighted, and several other days with elevated fatal crash counts are labeled.

The chart was originally published here, with the following commentary: "There is quite a lot of noise in these daily crash rate ratios, and few that appear reliably above or below the rates +/- one week." Andrew adds: "Nothing so exciting is happening on 20 Apr, which makes sense given that total accident rates are affected by so many things, with cannabis consumption being a very small part."

While the chart looks cool, and sophisticated, the following histogram of the same data helps the reader digest the information. 

I took the daily estimates of the fatal crash ratios from Harper/Palayew's published data. Each ratio presents the crashes on the analysis day relative to the crashes on the two reference days. The histogram shows the day-to-day variability of the crash ratios, which is what we need to answer the question: how special is 4/20?

The histogram is roughly centered at 1.0 meaning no observed difference. The black vertical line shows the ratio for 4/20. It is leaning right - in fact, it is at the 94th-percentile. In classical terms, this is a p-value of 0.06, barely significant. 

The following 21 days have more extreme ratios than 4/20: 

Jul 4 Dec 23 Dec 21 Nov 21 Sep 1 Dec 20 Sep 2 Jul 3 Dec 31 Oct 31 Nov 23 Dec 18 Dec 6 Jul 14 Sep 4 Dec 22 Mar 17 May 25 Apr 1 Mar 7 Dec 19

Will JAMA editors accept one research paper for each of these days? The work is already done - the rest is story time. 

 

P.S. [4/27/2019] Replaced the first chart with a newer version from Harper's site. This version contains the point estimates that the other version did not. Those point estimates are used to generate the histogram.