Big Data, Plainly Spoken (aka Numbers Rule Your World)

Football and statistics, on HBR!

I was asked to adapt my earlier post for the HBR audience, and the new version is now up on HBR. Here is the link.

I'm happy that they picked up this post because most business problems concern reverse causation. A small subset of problems can be solved using A/B testing, but only those in which causes are known in advance and subject to manipulation. Even then, Facebook got into trouble for running such an experiment (not in my eyes though).

Thanks to the editing team at HBR. I like the new version a lot.

***

Neil Paine at Fivethirtyeight did a round-up of the key articles that attempt to use data to make arguments about Deflategate.

***

PS. It is a little hard to find the HBR article right now. Seems like it takes time for their website to update the search engine and navigational tools.

PPS. For those in California, come meet me next week. Here are the details (look also on the right sidebar). The talk is about visualizing data.

Limits of statistics, and by extension data science, as illustrated by Deflate-gate

A number of readers sent me Warren Sharp's piece about the ongoing New England Patriots' deflate-gate scandal (link to Slate's version of this) so I suppose I should say something about it. For those readers who are not into American football, the Superbowl is soon upon us. New England, one of the two finalists, has been accused of using footballs that are below the weight requirements on the rulebook, hence "deflate-gate".

The Slate piece is a good example of the brand of data journalism that is possible in today's world where anyone can get a hold of a lot of data. The quality of the analysis is above average as far as these pieces go. I like the use of different visualizations to understand the variability of the plays per fumble across teams. It's also clear to me that the histogram is easily the best of the bunch.

 

The chart is just missing a label: the one team standing far right is the New England Patriots.

***

When reading these pieces, pay attention to the structure of the statistical argument. Here is how I would summarize the argument:

1. New England is an outlier in the plays per fumbles lost metric, performing far better than any other team (1.8x above league average).

2. Different ways of visualizing and re-stating the metric yield the same conclusion that New England is the outlier.

3. There is a dome effect of about 10 plays per total fumbles, meaning that teams who play indoors ("dome") typically suffer 10 fewer fumbles than teams who play outdoors ("non-dome"). New England is an outdoor team that performs better than most dome teams on the plays per total fumble metric. If dome teams are removed from the analysis, New England is an outlier.

4. Assuming that the distribution of the metric by team is a bell curve, the chance that New England could have achieved such an extraordinary level of play per fumbles lost is extremely remote.

5. Therefore, it is "nearly impossible" for any team to have the New England type ability to prevent fumbles... unless the team is cheating.

Actually, Sharp to his credit did not argue Point 5. If you jump to the end of the article, all he said was the extreme value of New England's plays per fumble performance is not random fluctuation. And that there are many possible explanations, including both legitimate and illegitimate ones. So not a lot of bang for the buck of running the analysis.

However, any reader of the Slate article, especially an anti-Patriots fan, will be tempted to cite these statistics and actually conclude that the Patriots cheated. All he/she requires is some story about how deflating footballs reduce the chance of fumbling the football. Notice I say story because I don't think we have any scientific theories connecting the two, yet.

This is a good example of the limitation of data analysis, including Big Data analysis. However suggestive, the data could not prove guilt. In this case, it is hard to find data to deny that the Patriots couldn't achieve the result in a legal way.

In fact, the "dome" analysis for me weakens rather than strengthens the argument. Sharp switched from plays per fumbles lost to plays per total fumbles, the difference being fumbles that were recovered by the fumbling team. Since I can't understand how deflating the football could help the Patriots recover more fumbles after the ball hits the ground, I perfer plays per total fumbles as the measure. Given this measure, the Patriots is not an outlier at all, and is second to the Falcons--only when Sharp removed all dome teams (the Falcons being one) could he make the argument that the Patriots is the outlier. But this tells me there are legitimate ways to perform equally or slightly better than the Patriots did-just look at the Falcons.

The only true proof of cheating is if New England is caught red-handed. Show me the record of the weights of the footballs and you have a believer. Show me someone who saw the footballs being deflated. Similarly, drug testing (statistics) did not nail Lance Armstrong (I wrote about it a lot); what finally wrongfooted him were eye-witness evidence and law enforcement investigations.

Also, this is an example of what Andrew Gelman has been calling "reverse causation" problems (link). We learn that New England did spectacularly on a metric, and we want to know what caused it. This is the opposite structure from an A/B test where we vary some causes, and observe how the variations affect an outcome. The reverse causation problem is one of the big issues of the Big Data era that isn't getting enough attention.

***

As this post is dragging, I will leave other comments to Part 2. An issue I have with the outlined statistical argument is that Points 1, 2, 3 and 4 are essentially re-stating the same thing four times.

 

 

 

How Optimizely will kill your winning percentage, and why that is a great thing for you (Part 1)

In my HBR article about A/B testing (link), I described one of the key managerial problems related to A/B testing--the surplus of “positive” results that don’t quite seem to add up. In particular, I mentioned this issue:

When managers are reading hour-by-hour results, they will sometimes find large gaps between Groups A and B, and demand prompt reaction. Almost all such fluctuations result from temporary imbalance between the two groups, which gets corrected as new samples arrive.

Over the holidays, I paid a visit to the Optimizely team, and learned that they have been developing a solution to this problem. (Optimizely is one of the leading platforms for online A/B testing. They just made an announcement this week about a new feature they are calling “the New Stats Engine”.)

Optimizely also recognizes that their clients face a credibility crisis when the A/B testing tool returns too many “significant” results. Their new tool promises to reduce this false-positive problem. They tackle specifically two sources of the problem:

a) Many clients monitor A/B tests like horse races, and run tests to significance. This is sometimes known as “sampling to a foregone conclusion”.

b) Many clients run many (dozens to hundreds, I imagine) tests simultaneously; here, a test is any pairwise comparison of variations, comparison of variations within segments, or any comparison using multiple goals. This is the “multiple comparisons” problem.

***

Let me first explain why those are bad practices.

The classical hypothesis test is designed to work with fixed sample sizes, which should be determined prior to the start of the test. The testing protocol then allows up to a 5-percent probability of falsely concluding that there is an effect (That’s the same value as the significance level. This is not the same saying 5 percent of the positive results are false, but that’s a different article). However, if the analyst is peeking at the result multiple times during a test, then the analyst incurs a 5-percent false-positive chance, not once, but for every such peek. Thus, at the end of the test (when significance is reached), the probability of a false positive is much, much higher than five percent. It can be shown that every A/A test will reach significance eventually in this setting.

In a “multivariate” test, the analyst makes many pairwise comparisons, and each comparison is analogous to a peek of the data. Each comparison incurs a 5-percent false-positive chance so that across all of the comparisons within one test, the chance of seeing at least one false positive result is exponentially larger. There are many, many different ways to suffer a false positive (an error in comparison 1 only, in comparison 2 only, etc., in comparisons 1 and 2, in comparisons 1 and 3, etc.).

Now, if the multivariate test is being run to significance, you have a hydra of a head.

***

The Optimizely solution uses two key results from statistics:

a) A sequential testing framework is adopted, in which the analyst is presumed to be peeking at the results. The Bayesian analysis in most cases will not result in significance even if the sampling does not end--because of the skeptical prior. This line of research started in the 70s 40s with Wald.

b) All solutions to the multiple comparisons problem involves tightening the threshold of significance at the individual test level. Optimizely adopts the Benjamini-Hochberg approach to controlling the “false discovery rate,” (FDR) defined as the proportion of significant results that are in fact false. This line of research is from the 90s, and still very active. One advantage is that the FDR is an intuitive concept.

***

What this means for Optimizely clients is that your winning percentages (i.e., the proportion of tests returning significant results) will plunge! And before you despair, this is actually a great thing. Here’s why: In many testing programs, as I pointed out in the HBR article (link), there are too many “positive” findings, which means there are too many false positives. This is fine until the management starts asking you why those positive findings don’t show up in the corporate metrics.

If you currently rely on standard Optimizely reports to read test results, and run tests to significance, then the Stats Engine is surely a no-brainer.

In the next post, I have further thoughts for those customers who have more advanced protocols in place. 

 

PS. This is Optimizely's official explanation of their changes on YouTube.

Three reasons to doubt the GDP-gas price conjecture

Dragged by infectious incuriosity, the financial press ran with the story that falling gasoline prices (50% drop in 6 months) is "the best economic stimulus one can get". See former Deputy Treasury Secretary Robert Altman on CNBC, Business Insider's "cheap gas boost", Wall Street Journal citing the "low oil prices as an effective tax cut for consumers", New York Times quoting a Citigroup analyst claiming a global > $1 trillion stimulus, etc. etc.

This is the kind of story that one should believe only if half asleep. Here are three reasons why this conjecture is likely to be wrong:

1. Forgetting the big picture

There was a McDonalds next to a Burger King in a small town. The Burger King went out of business. The McDonalds suddenly did twice the usual business. Surely, McDonalds was the winner here but did the economy of the town expand? Unfortunately not. The consumers merely shifted their spend from Burger King to McDonalds.

Now, consider a household that spends $200 a month on gas before the oil price crash. Let's say the same amount of gas now costs $100. According to those rosy-cheeked economists and journalists, the household now has an extra $100 to spend on other things, and this "extra" spending stimulates the economy.

But the total amount of expenditure is still $200. The only thing that changes here is the mix of spending. GDP is based on total spend, not the mix of spend. Some sectors of the economy will benefit but at the expense of the oil and gas sector.

2. Imperfect substitution

Consider our household again. The total economy size remains the same only if the household spends every dollar of the $100. If the household saves even one of those dollars, the economy shrinks, compared to before. 

3. Making bad assumptions about the future

It's unclear from any of those articles how the analysts came up with the size of this oil-drop stimulus. Every one of them must make a forecast about future oil prices. I bet many of them take the current price as the new normal, and use that price as the future price. 

If I tell you, you should not take an extreme value and treat it as the average, you'd scold me for stating the obvious.

As with most economic arguments, one could posit a much more complex chain of relationships that would argue how one goes from 50% drop in oil prices to trillions of economic stimulus. It is the business journalist's job to explain that complicated chain. The connection is clearly not as simple as reported. If one establishes a chain, such as A up ->B down ->C down -> D up, etc., each one of those causal links should be supported with evidence.

 ***

The same type of fallacious thinking pervades the business sector. For example, we keep hearing about the growth in retail sales from mobile devices. We don't know if consumers are shifting from the Web channel to the mobile channel, and how much of the mobile sales are incremental.

 

Trifacta revisited: tackling a Big Data problem

During my vacation, I had a chance to visit Trifacta, the data-wrangling startup I blogged about last year (link). Wei Zheng, Tye Rattenbury, and Will Davis hosted me, and showed some of the new stuff they are working on. Trifacta is tackling a major Big Data problem, and I remain excited about the direction they are heading.

From the beginning, I am attracted by Trifacta’s user interface. The user in effect assembles the data-cleaning code through visual exploration, and suggestions based on past behavior.

Here are some improvements they have made since I last wrote about the tool:

Handling numeric data - Trifacta now generates some advanced statistics, e.g. percentiles, about the columns in the Visual Profiler whereas in the past, every column is summarized as a histogram. I believe there is also some binning functionality.

Moving beyond Top N - I ranted about Top N thinking in the past (link), and I wasn’t happy that the Trifacta demo seemed to encourage this bad practice. I’m happy that the team heard the complaint and now offer a Random N selection. Eventually, I think Random N should be the default; I don’t know why anyone would want to see Top N.

Interactive workflow - Random N is a big step forward but in the world of data cleaning, it’s not sufficient. The reason is that many data quality problems are rare cases that don’t show up in a random sample. To deal with this, Trifacta has created an interactive workflow. Through the visual exploration paradigm, the software prepares a set of code; when the user applies the code to the entire data, the tool automatically check for further anomalies, and reports those to the user. For instance, there may be a handful of email addresses with unusual structures not found in the random sample, and thus fall outside of any of the data-wrangling rules. These are flagged for further treatment.

Column metadata - Another exciting development is the expanded use of metadata associated with columns. Such metadata is a major difference between an Excel spreadsheet and any sophisticated data table. For instance, the user can now associate labels with values within a column.

New file formats - Trifacta handles many new data formats like JSON. It can, for example, accept a JSON file and parse the nested structure into columns. Very nice addition!

***

I think Trifacta can gain ground by pushing the envelope on two fronts: more and better visual cues to help users diagnose data-quality problems; and more sophisticated recipes for how to handle such problems, informed by a knowledge base of past user behavior.

 

 

Figuring out what data supports the argument, and what is just window-dressing

That is the question in my head when I read an article like USA Today's "Jobless Claims Fall, Suggests Strong Hiring". (link)

The headline makes the connection between newly-released jobless claims data and the conclusion of "strong hiring". But it turns out the new data is merely window-dressing, and the conclusion is based on longer-term trends.

Here is the new data, as reported by the USA Today reporter:

• applications for unemployment benefits fell 4,000 last week to a seasonally adjusted 294,000.
• The four-week average, a less volatile measure, slipped 250 to 290,500

Without even looking up the source, one should immediately see that a change of 250 on the four-week moving average is just statistical noise. The 4000 change for the last week is also statistically insignificant because the weekly series is highly volatile. The proper conclusion about this release of data is that the employment situation was stable from last week to the week before.

Now, one could go backwards in time and make an argument for "stronger hiring". This is exactly what the journalist did, by citing "total job growth in 2014 at just shy of 3 million, the best performance since 1999" and "[The 4-week moving] average [of jobless claims] has plunged 16 percent in the past 12 months, as averages have stayed at historically low sub-300,000 levels since September".

Take a look at this chart of the 4-week average in the last five years. The trend has been the same for five years (just draw a straight line through the series) and there is nothing at the right tail of the time series to indicate that the latest data release changed anything:

 

I'm also amazed that at this point, a journalist can write an article about employment without once mentioning the workforce participation rate. (Anyone who is excluded from the work force is not eligible to be "unemployed". The workforce participation rate has gone down without recovering.)

Notice that this time series was essentially flat until the recession.

I have a whole chapter on employment statistics in Numbersense (link).