Big Data, Plainly Spoken (aka Numbers Rule Your World)

Statbusters are back, taking on robots that hire people

In our newest column, we take on the recent media obsession with companies who make robots that hire people. (link)

As with most articles about data science, the journalists failed to dig up any evidence that these robots work, other than glowing quotes from the people who are selling these robots. We point out a number of challenges that such algorithms must overcome in order to generate proper predictions. We also discuss why measuring the outcomes of these predictions is so hard: one problem is we have no objective standard for someone being the "correct" hire; another is the action we take based on the predictions affects the outcome that was predicted.

Read the entire article here.

Deja vu! Doping accusations at Tour de France

Gabe Murray wrote to Andrew Gelman, asking for comments about the accusations hurled at the current Tour de France front-runner Chris Froome. He said:

This post by VeloClinic has been getting a lot of media attention in the past few days, within the context of Chris Froome's dominant performance in the Tour de France:

 

http://veloclinic.com/estimating-the-probability-of-doping-as-a-function-of-power/

 

The assumptions seem very dubious to me, and I would love to see a critique of their methods.

 

Andrew forwarded the email to me. I wrote about steroids testing and Tour de France in my first book Numbers Rule Your World. The gist was that those tests fail to capture most dopers, which of course is not a controversial comment today. But pre the Armstrong confession, it used to be people believed in lots of things.

 

I clicked on the link, and read through the nicely-written summary of how they estimated the probability of doping, and wrote back to Gabe:

 

It looks like a reasonable way of proceeding to me. When you say "The assumptions seem very dubious to me, and I would love to see a critique of their methods.", what do you mean exactly? Are you unhappy with the method or are you unhappy with the assumptions (such as the prevalence of doping, the effect of doping, etc.)?

 

He replied:

 

The part of the analysis that seemed most dubious to me was the estimate of the prevalence of doping in the peloton today, estimated at 25%. The authors seemed to think that was a conservative estimate, but it sounds very high to me. If it was 10 or 15 years ago, sure, I think the number would be even higher than that. But given the monitoring that has been put into place by UCI since then, including the biological passport and year-round monitoring, if the number of riders doping was nearly 25% then I would expect that the number of positive doping tests would be much higher than it is. But of course it's hard to know, and perhaps the doping is just very sophisticated micro-dosing.

Also, for the distribution of W/kg, the parameters of the Gaussian distribution for clean riders (shown in the first plot) suggest that there is a very narrow range of W/kg values for clean riders in the peloton. I don't know enough about sports physiology to know whether this is true, which is part of why I was looking for critical analysis from others. I would have expected the standard deviation to be larger.

Finally, I wonder if the Gaussian distribution is even appropriate here when modelling the distribution of W/kg values for the most elite athletes in the world.

 

That was a nice deconstruction of the model used for the prediction. Two normal distributions are postulated, one for clean riders and one for dopers. Then you mix those distributions.

 

Now my response to Gabe's critique.

 

Generally speaking, this type of conversation falls into the same vein as the "deflategate" in NFL. Statistical modeling allows us to put a probability estimate on the chance that there is foul play. Note that it is an estimate, based on a model of the world. Further, note that the cheaters are adversaries, and they will hide their tracks, and most analyses just plain fail to take that into account.

All models can be challenged on the grounds that you don't like the structure of the model. I personally don't like that kind of critique unless someone can offer an alternative model that both sides agree to be better.

 

On the specifics about the prevalence of doping, I wrote:

 

This is the issue of the "prior" in Bayesian statistics. In theory, you can run sensitivity analysis to see how the 25% assumption changes things.

 

In reality, I don't think it matters. Their model is that the bell curve shifts upwards for dopers (which I think is reasonable). This means that if you focus on the extreme tail of the combined distribution, it is always going to be the case that the majority of those extreme people would come from the doping distribution.

 

Later, Gabe confirmed what I just said, as he played with the spreadsheet from VeloClinic:

 

Based on your comments, I should clarify that I am not opposed to attempts to estimate the probability of doping given power, and I accept that all models are approximations. I do think it's appropriate to question whether the parameters of a model are sensible, and I gave some specific examples that I thought could be improved. Also, I am not at all opposed to them using W/kg as a proxy measure.

 

Playing around with their provided spreadsheet ... does confirm that changing the prior does not have a big impact. However, changing the standard deviation of the clean rider distribution does.

 

To which I responded:

 

I think the question that gets to the core of this is: what should a reasonable model predict to be the chance of doping given extreme levels of performance? When you get into the extremes, it seems to me any reasonable model would say the same thing this model is saying - that the more extreme the performance, the more likely the athlete has doped. I don't think statistical modeling can give you the kind of answers you are looking for.

I'd add that given the revelations around Armstrong, we should be modeling: the more naturally gifted a performer, the more likely he/she dopes. People say the opposite before but they were wrong. Because of this, it is even more difficult to separate out doping from talent!

 

In the email exchange, I didn't comment on the standard deviation bit that Gabe raised. By increasing the standard deviation of the clean riders, you are explicitly allowing more extreme performers who are clean so what he said sounded right. In this scenario, I'd have shifted the dopers proportional to standard deviations.

 

I say there is great progress because reporters no longer believe the story that if you pass dozens of tests, you must be clean!

 

[I should add that Gabe later pointed out that it is no longer clear whether Froome was an outlier or not on that proxy metric for performance.]

 

Statistically significant. What does it mean?

Andrew Gelman has a great post about the concept of statistical significance, starting with a published definition by the Department of Health that is technically wrong on many levels. (link)

Statistical significance is one of the most important concepts in statistics. In recent years, there is a vocal group who claims this idea is misguided and/or useless. But what they are angry about is the use (and frequently, mis-use) of p-values, which is one way to measure statistical significance. In my view, this concept is never as important as it is today, in the world of Big Data.

Statistical significance codifies a core principle--that the observed dataset is not sufficient to answer your question (no matter how big the dataset is). It says that your observed sample is only one of many possibilities. If you could repeat your data collection, your dataset would look different. It might look just a slight bit different, or it might look vastly different. This "chance" that people keep talking about is merely the variations you get between one sample and another. By its very nature, this is a thought experiment -- you only have one observed sample.

But that is the point. Sound statistical reasoning requires you to think beyond your one observed sample.

 

It is possible to not learn real causes from some A/B tests

It is conventional wisdom that A/B testing (or in proper terms, randomized controlled experiments) is the gold standard for causal analysis, meaning if you run an A/B test, you know what caused an effect. In practice, this is not always true. Sometimes, the A/B test only provides a statistical understanding of causes but not an average Joe's understanding.

Let's start with a hypothetical example in which both definitions are aligned. The only difference between the test and control groups was (junk) mail sent to the test group that advertised Amazon's Prime Day. If a higher proportion of the customers who got the junk mail shopped at Amazon on Prime Day than the control group, then everyone agrees that the mailing caused an increase in shopping activities (assuming the difference was statistically significant). Of course, not everyone was influenced by the mail but on average, more people in the test group shopped.

This consensus falls apart in the canonical A/B testing example. Imagine that the only difference between the test and control groups is the background color of the home page. The test group saw a darker background than the control group. The test group had a lower bounce rate than the control group. Statistically, the background color is regarded as the cause of the drop in bounce rate.

But what does that last sentence mean to the average Joe? There seems to be a missing link. It really isn't the color that is influencing someone's browsing behavior. It may be that the darker color causes certain text or images on the home page to be more visible, which causes more visitors on average to browse more pages, sticking around for a longer period of time. It could be the opposite. The darker color may make certain text less visible, which causes visitors to stick around for a longer period of time because they were struggling to find what they were looking for!

In other words, the background color may only be an indicator of the cause, not the true cause itself.

Some people have argued that it doesn't matter. You don't need to know what the true cause is. All you care about is that this result can be replicated.

I disagree with that position because taking that stance frequently leads to misguided future actions. The belief that background color somehow caused lower bounce rates inevitably leads you to test a variety of other colors (red, green, yellow, orange, blue, purple, ...) Almost all of these tests would be a waste of time. If, on the other hand, you learn that making your text clearer to the visitors is the true cause, then you may instead test larger font, different font types, placement of the text, etc.

In short, you should have some mental model behind the cause and effect mechanism.

 

 

 

Flawed thinking about causes

One of the most misguided and dangerous ideas floated around by a group of Big Data enthusiasts is the notion that it is not important to understand why something happens, just because "we have a boatload of data". This is one of the central arguments in the bestseller Big Data, and it reached the mainstream much earlier when Chris Anderson, then chief editor of Wired, published his flamboyantly-titled op-ed proclaiming the "End of Theory."

In making the claim that causal analysis is hopeless and pointless, these proponents are disowning entire fields of study. While focusing their vitrol on the social sciences, they somehow miss the obvious: that causal thinking is and has always been the foundation of the physical sciences and engineering. Even business executives understand the primacy of "root-cause analysis."

Contrary to these folks, I believe social scientists will produce the most exciting research on causal analysis. Human behavior is generally much more variable than natural phenomena, complicating the search for causes. We need even smarter people to tackle these problems.

That said, a lot of published social science exhibit flawed thinking about causes. Andrew and I describe a few of the problems in the latest Statbusters (link). An assumption is often made that an observed effect is due to a single cause. Much effort is expensed on identifying this one cause from a slate of candidates. Further, it is just as important for us to know which studies failed but such failure is never reported in journals or the media. This publication bias results in researchers examining the same correlations over and over again, and eventually one research group will discover a "statistically significant" effect and get it published, even though in reality, the totality of the evidence would contradict the one published study.

What is an interaction effect?

One statistical concept that instructors frequently don't have time to cover in Stat 101 is the "interaction" effect. I will explain this concept using the fantastic interactive graphic by the visualization team at the German publication Zeit (please also read the corresponding post on Junk Charts here for some background.)

When we ignore interactions, we end up with overly simplistic statistical summaries. For example, some study might find that drinking six cups of coffee reduces the chance of prostate cancer by 60 percent. But is this effect the same for all age groups? Is it possible that the risk reduction is higher for older men and lower for younger men, for example? When we are asking questions like these, we are asking whether the effect of coffee consumption on health interacts with age.

In the Zeit visualization, the following graph illustrates that Germans living in the former East Germany (blue line, referred to as East Germans hereafter) are more likely to hold negative views about smoking cannabis, relative to Germans living in West Germany (yellow; West Germans). On average, about 70 to 80 percent of the people deem the activity "bad" or "very bad" and the East-West gap was roughly 8%.

A relevant question here is whether the size of this East-West gap varies by age.

In the interactive visualization, one can click on different age groups to observe how the lines shift. The left chart below shows the East-West gap for 45-64 year olds while the right shows the gap for people aged 65 and above.

 

The strongest signal to hit us is probably that the 65-and-above cohort has a much more negative opinion of smoking cannabis than the younger cohort. This is a statement about the effect of age on the attitude towards cannabis, regardless of where they live. It's about a single factor, and so it isn't an interaction effect.

The next observation is that in the 65-and-above cohort, the East-West gap is noticeably smaller than the average gap, and the gap has remained essentially unchanged over the last 12 years. However, for the 45 to 64 age group (left chart), the gap has markedly increased, meaning that in 2012, East Germans were about 15 percent more likely to dislike smoking than West Germans even though both groups started out with about the same attitude in 2000.

Statisticians call this a significant interaction effect between East-West and age group. When an interaction effect exists, the aggregate statistic is not very useful as it fails to acknowledge the variability by age. While the average gap was 8%, the gap for people above 65 was only half of that while the gap for the 45 to 64 cohort was almost twice that. (I am ignoring the other age groups; just keep clicking.)

It is important to distinguish between three effects: the "main" effect of East Germany vs West Germany, the "main" effect of age group, and the interaction effect between East/West and age group.

When you ignore interaction effects, you are assuming "additivity". Unfortunately, when it comes to statistics, one plus one usually do not add up to two! This point causes much confusion for non-statisticians. In statistics, "one" is not an exact quantity; there is a margin of error around it.

***

As a second example, consider the following graph which shows the East-West gap on the issue of whether working mothers are good.

The two bolded lines represent the average person in East and West Germany. We see that across time the East-West gap has marginally narrowed from 28% to 25% over 25 years.

Does this gap vary depending on whether the respondent is male or female? To see this, we split the male and female responses. Below, the males are shown in gray on the left chart and the females are in gray on the right side.

First thing to notice is that the two lines for men (East and West) are both below average, meaning that men are less likely to accept working mothers. Not surprisingly, the lines for women (on the right chart) show they are more likely to accept working mothers. But these comments relate to the effect of gender on the working mothers issue. What we are interested in is whether the East-West difference is affected by gender.

This means, we care about the gaps between the gray lines on both charts. With only a little effort, you can see that the gap is wider on the left chart than on the right. This means, the gap between West and East German men is larger than that between West and East German women when it comes to working mothers. Statisticians call this an interaction effect. When such an effect is significant, statisticians prefer to talk about the genders separately, rather than combining them into one average.

***

Next time you run a regression, add some interactions and see if it makes a difference. I address this issue in Chapter 3 of Numbers Rule Your World.

 

Is data privacy a fundamental right?

This piece is part of the StatBusters column written jointly with Andrew Gelman. Hope they fix the labeling soon. In it, we talk about two recent studies on data privacy, which leads to contradictory conclusions. How should the media report such surveys? Is the brand name of the organization enough? In addition, we debunk the notion that consumers will definitely get something valuable out of sharing their data.