Big Data, Plainly Spoken (aka Numbers Rule Your World)

Composite ranking and numbersense

Chapter 1 of Numbersense (link)uses the example of U.S. News ranking of law schools to explore the national pastime of ranking almost anything. Since there is no objective standard for the "correct" ranking, it is pointless to complain about "arbitrary" weighting and so on. Every replacement has its own assumptions.

A more productive path forward is to understand how the composite ranking is created, and shine a light on the underlying assumptions.

***

The New York Times recently published an article entitled "What's the Matter with Eastern Kentucky?" (link). The problem with Eastern Kentucky, as the reporter saw it, is that those counties rank at the bottom of their list. Here is their ranking methodology:

The team at The Upshot, a Times news and data-analysis venture, compiled six basic metrics to give a picture of the quality and longevity of life in each county of the nation: educational attainment, household income, jobless rate, disability rate, life expectancy and obesity rate. Weighting each equally, six counties in eastern Kentucky’s coal country (Breathitt, Clay, Jackson, Lee, Leslie and Magoffin) rank among the bottom 10.

There is a companion blog at The Upshot, giving more context, and a county-level map of the ranking (link). Here are the relevant sentences.

The Upshot came to this conclusion by looking at six data points for each county in the United States: education (percentage of residents with at least a bachelor’s degree), median household income, unemployment rate, disability rate, life expectancy and obesity. We then averaged each county’s relative rank in these categories to create an overall ranking.

(We tried to include other factors, including income mobility and measures of environmental quality, but we were not able to find data sets covering all counties in the United States.)

We used disability — the percentage of the population collecting federal disability benefits but not also collecting Social Security retirement benefits — as a proxy for the number of working-age people who don’t have jobs but are not counted as unemployed.

How should we read this article?

***

What is this a ranking of? What is the research question? The answer is "how hard it is to live in specific counties". Right away, we know any answer is subjective, even if data is proffered.

Look out for the relative weights. The authors tell us it's equally weighted. "Equal weighting" implies fairness but frequently hides the inequity. Are those six factors equally important? Are there strong correlations among some of those factors?

The blog post discloses that each of the six metrics is first converted to ranks before being averaged. This means we need to worry about how much each metric vary from county to county. Take obesity rate for example. Here is a map of obesity at the county-level published by the CDC, based on a model estimate (link).

The people who made this map placed the counties into five groups. The middle groups are narrowly defined, for example, 29.2% to 30.8%. Any analyst who converts the county-level obesity rates to ranks makes over 3000 gradations of obesity rate. Said differently, the worst county is rated as over 3000 times worse than the best county. In the case of obesity, the medical community would consider most of these counties unhealthy.

This is an example that shows too much granularity hurts you, a core insight of statistics that may seem counterintuitive.

***

Ultimately, it's for you to decide whether you believe this ranking makes sense or not. I'm not here to dismiss it because as I said in Numbersense (link), you can replace this methodology with something else, but the new method will also have its own assumptions.

Anyone still remember Groupon?

I just did a guest lecture at a New School journalism class. While preparing for the class, I pulled the sad stock chart for GRPN (Groupon):

If you bought the hype in 2011, you'd have lost 70% of your investment ($25 to $7).

Given what we know today, it's hard for people to feel the hype that the media helped fuel in those days. As a reminder, here is the New York Times's David Pogue gushing about Groupon, just before its IPO: link. Pogue was one of many such commentators. 

Around that time, I had this response to the Groupon boosters. There was a gaping hole in the win-win-win story from the start. Retailers are giving up sure profit for the probability that the coupon-users are not dealseekers and would come back for repeat business, at a higher price.

This is related to my current concern about the so-called "gas price stimulus". The hit to the oil and gas sector is immediate and certain. The shift of spending to other sectors, and the associated "multiplier effects", is a probability of multiple events occurring in the future.

 

Numbersense, in Chinese and Japanese

This is a cross-post on my two blogs.

The new year brings news that my second book, Numbersense: How to Use Big Data to Your Advantage has been translated into Chinese (simplified) and Japanese. Here are the book covers:

In Chinese, the title reads: "Say No to Fake Big Data". Captures the sentiment of the book pretty well, I must say.

I have no idea what the Japanese title means. Perhaps a reader can help me out here.

***

The Japanese version is available here or here.

The Chinese version is here.

The English version is here.

Optimizely Stats Engine 2: what about advanced users?

In Part 1, I covered the logic behind recent changes to the statistical analysis used in standard reports by Optimizely. 

In Part 2, I ponder what this change means for more sophisticated customers--those who are following the proper protocols for classical design of experiments, such as running tests of predetermined sample sizes, adjusting for multiple comparisons, and constructing and analyzing multivariate tests using regression with interactions.

For this segment, the choice of sticking with the existing protocol or not depends on many factors, such as the decision-making culture and corporate priorities. No matter what you do, it is important to realize that improved analysis tools do not obviate careful planning and execution.

***

Let me start with my advice. Initially, keep running your tests to the usual fixed sample sizes. In essence, you ignore the stopping rule suggested by the Stats Engine. Over a series of tests, including some A/A tests, you can measure how likely those stopping rules would have correctly ended the tests (relative to the fixed-size testing protocol). This allows you to estimate the “time saving” achieved from sequential testing.

***

As I pointed out in last year’s presentation at the Optimizely Experience, the testing team should be concerned about what proportion of significant findings are correctly called, and what proportion of non-significant findings are incorrectly called. The “false discovery rate” is the flip side of the first quantity.

A testing program using fixed samples may face one of several problems:

a) Too few tests are called significant.

b) Too many tests are called significant.

c) It takes too long to call a test.

You need to figure out what is your biggest problem.

Conceptually, relative to a fixed-size test, a sequential test saves time if the true response rate differs from the design assumption substantially. If you’re testing on a web page for which the response rate is well-known and relatively stable, then there should be hardly any time saving on average. This is why I don’t recommend watching tests like a horse race, minute by minute. (As I said in Part 1, if you are watching a horse race, the Stats Engine will provide some sanity.)

Assuming that you underestimated the true effect by say 20 percent. The following stylized chart is my expectation of how the new Stats Engine results compare to the classical results.

The horizontal axis shows the sample size (at which Optimizely calls an end to the sequential test) as a ratio of the fixed sample size (by design). When this is 100%, the sequential test has the same length as the fixed-sample test. Because the true effect is substantially larger than expected, for a large proportion of tests, the sequential procedure calls for an “early” exit. However, there will be a small number of tests for which the sequential test will end much later than a fixed-sample test.

 

 

 

 

 

On the other hand, if the design assumption is essentially correct, then I expect the behavior of the new Stats Engine will look something like this.

The line is mostly flat meaning there is equal probability of the test ending at any sample size, including sample sizes that are multiples of the fixed-sample requirement. This is the “price to pay” for doing sequential testing, i.e. multiple peeking. At the lower end of sample sizes, I expect a slight positive curve, because the Bayesian prior (assuming it is a skeptical prior) will prevent tests from being stopped “too early”.

[Thanks to Optimizely’s statistics team for entertaining my inquiries about this intuition.]

***

How important is saving time for your testing program? This depends on your readiness to move on. My experience is that unexpected time saving, say calling a winner one week before the test was supposed to end, frequently gets eaten up by the organization’s inability to move schedules around. Your IT or web developers may have other projects on their plates.

Further, if you tend to look at data by segments post-hoc, I don't think the current implementation supports that. If you know what segments you care about beforehand, then you can build those into the design.

Most importantly, please don’t fall into the trap of thinking that design and upfront planning become unimportant because of sequential testing and FDR. The design phase is very important in establishing expectations and facilitating communications within the organization.

I also recommend reading this post by Andrew Gelman on data-dependent stopping rules.

 

 

Deflate-gate, Part 2: not average != extreme, and Sunday talk shows

Last week, I pointed out the futility of using data as proof or disproof in Deflate-gate. Emphatically, a case of "N=All" does not make things better. I later edited the post for HBR (link).

In this post, I want to address a couple of more subtle technical issues related to the Sharp analysis, which can be summarized as follows:

1. New England is an outlier in the plays per fumbles lost metric, performing far better than any other team (1.8x above league average).

2. Different ways of visualizing and re-stating the metric yield the same conclusion that New England is the outlier.

3. There is a dome effect of about 10 plays per total fumbles, meaning that teams who play indoors ("dome") typically suffer 10 fewer fumbles than teams who play outdoors ("non-dome"). New England is an outdoor team that performs better than most dome teams on the plays per total fumble metric. If dome teams are removed from the analysis, New England is an outlier.

4. Assuming that the distribution of the metric by team is a bell curve, the chance that New England could have achieved such an extraordinary level of play per fumbles lost is extremely remote.

5. Therefore, it is "nearly impossible" for any team to have the New England type ability to prevent fumbles... unless the team is cheating.

***

Focus on Point 4 for the moment. This is a standard technique used by statisticians, and the basis of any analysis of "statistical significance". In statistical significance testing, we appeal to the normal distribution (bell curve) to estimate how close the observed sample is to the "average sample". The big question being addressed is: IS THIS AVERAGE?

Let's say we want to measure the effect of genetic modification on the size of fish. If the Fracken-fish sample is far from the average of natural fish samples, we conclude that Fracken-fish is statistically different (larger) from natural fish. A crucial requirement of this analysis is that the samples are randomly drawn.

But for Deflate-gate, the big question is: IS THIS EXTREME? The statistical significance tool is not designed to answer this question. The analysis tells us that the Patriots do not look like the average random sample from the NFL. Saying that something is not average is far from saying that it is an outlier! Indeed, statistical significance testing is frequently (and controversially) used to detect "small effects".

If the Patriot sample were randomly drawn out of the NFL, then Point 4 would have provided evidence of an extreme value but there is no random selection here. This takes us back to the point of my first post: the Patriots could belong to a group of elite NFL teams that have more "skill" in preventing fumbles, or there could be many other possibilities.

***

The other point of interest is that Points 1-4 say essentially the same thing: that the Patriots are far different from the rest of the NFL on the play per fumbles metric. Point 2 is the visual equivalent of the mathematics of Point 4.

Point 3 sounds different but it really isn't. Points 2 and 4 say the Patriots don't fumble much. But dome teams fumble less because they play indoors; thus, their presence in the analysis makes the Patriots advantage (a non-dome team) less pronounced. Thus, in constructing Point 3, Sharp removed dome teams. It's the same data, viewed from a different lens.

Repetitively stating the same statistic does not make an argument. I'm not saying Sharp should not have performed these steps. I'd have done many of these analyses myself. But they play the role of quality control. The reiterations don't strengthen the argument, and they sound a bit like Sunday morning talk shows.