Big Data, Plainly Spoken (aka Numbers Rule Your World)

Proxy unmasking

Consider this paragraph from a FiveThirtyEight article about the small-schools movement (my italics):

Hanushek calculated the economic value of good and bad teachers, combining the “quality” of a teacher — based on student achievement on tests — with the lifetime earnings of an average American entering the workforce. He found that a very high-performing teacher with a class of 20 students could raise her pupils’ average lifetime earnings by as much as $400,000 compared to an average teacher. A very low-performing teacher, by contrast, could have the opposite effect, reducing her students’ lifetime earnings by $400,000.

If I had told you that students who performed higher on achievement tests have higher average lifetime earnings by as much as $400,000 compared to students who performed average on achievement tests, you'd not be surprised--unless you are a skeptic of achievement tests. This is evidence that achievement test scores predict lifetime earnings.

Now, this is not what Hanushekthe journalist wants you to believe. HShe said that high-performing teachers "could raise" pupils' average lifetime earnings. Two logical jumps are made in one breath here: one is the use of student achievement scores as a proxy for "teacher quality"; the second is "causation creep," (i.e. allowing a causal interpretation to creep on correlational evidence), which is signaled by the use of weasel words like "could" and "may".

The use of proxy measures is the source for many "statistical lies". One tool I use is "proxy unmasking". Substitute the proxy metric with the actual metric. So in this example, when I see "high-performing teacher," I substitute "high-performing student" since the observed data measured students, not teachers. The sound you hear is air rushing out of the hyperventilated argument.

Another example of misleading time-based correlation

On my sister blog last week, I wrote about how to screw up a column chart. The chart designer apparently wanted to explore whether Rotten Tomato Scores are correlated with box office success, and whether the running time of a movie is correlated with box office success. In either case, the set of movies is a small one, those directed by Chris Nolan. Here is a better view of the data:

There were a few questions on Twitter, which I will address in this post. Someone complained about the horizontal axis thinking that year data is continuous and the axis should not be discrete. That would be true if the data is truly continuous. The way I interpret the year data here is ordinal: Chris Nolan only makes a movie once every 1 to 3 years; his career is also developing during this period of time; so I think of the horizontal axis as ordinal, that is, his first film, his second film, etc.

Another Twitter user tried a scatter plot. It's very satisfying to see the following chart. It shows a strong positive correlation between the running time of a movie and the box office receipts.

In fact, a hockey-stick line fits the data even better. This implies a multiplicative relationship between running time and box office receipts. We can fit this by first taking the logarithm of box office receipts. So, the following
chart showing how good this fit is:

 If you run a regression, the R-squared is 90 percent, and the effect of running time is extremely significant (p < 0.0002). So we have proved that Chris Nolan should make longer movies because the longer the running time of his movies, the bigger the box office.

 ***

Now, re-consider the last lines of last week's blog (link):

You might have noticed that both running time and box office numbers have gone up over time. (That is to say, running time and box office numbers are highly correlated.) Do you think that is because moviegoers are motivated to see longer films, or because movies are just getting longer?

And these items:

1) Chris Nolan's career experienced a hockey-stick growth during this time.

2) Movies have become longer and longer in general.

3) Rotten Tomatoes also experienced a hockey-stick growth in users during this time period. In January 2000, the site had 250K visitors, at which point the founders said they "started in earnest as a company". (link) Today, according to Wikipedia, they had almost 20 million monthly visitors. In other words, several of the early movies by Chris Nolan came out before RT came into its own.

 

 

 

 

How students can improve their final grade without studying even more

Cross-post on my two blogs

Here is something different: I wrote a piece on exam-taking tips. It's on a new website, Cafe, which has lots of good (non-quant) reads. The motivation for the piece is my observation that most American students are not taught how to take exams. As a professor, I notice that many students get lower scores than they deserve because of this. 

In this article, I describe five things that are often neglected here, but are common knowledge in exam-heavy cultures.

The link is here.

How to face the mid-life crisis in A/B Testing

There have been few updates as I was working on things for other people. One of these things showed up today. Here is an excerpt from the beginning of my new article on HBR:

For over 10 years and at three companies, I set up and ran A/B testing programs, in which we test a new offer with half a sample against a control group which doesn’t get a new offer. Executives quickly pick up on the headline benefit of testing: that A/B tests provide reliable answers to “why” questions. This comes as no surprise, as such testing has long been held up as the “gold standard” for learning cause-and-effect in scientific research, clinical studies and direct marketing. However, many executives eventually reach a mid-life crisis, developing doubts about the direction of the A/B testing program.

From my experience, here are three of the most common questions that arise from those doubts, and how managers should think about them.

 

Read the full piece here.