Big Data, Plainly Spoken (aka Numbers Rule Your World)

The failure to replicate scientific findings

Andrew Gelman and I have published a piece in Slate, discussing the failure to replicate scientific findings, using the recent example of the so-called power pose. The idea of the "power pose" is that people develop psychological and hormonal changes by making this "power pose" before walking into business meetings, whereupon these changes make them more powerful.

As you often read here and at Gelman's blog, the fact that someone got a paper published in a scientific journal, based on a statistically significant result, doesn't automatically make it a believable result. Here, a different group of scientists tried to replicate the finding, with a sample size five times larger, and their replication did not come close to being statistically significant.

The original researchers wrote a response detailing differences between the two studies, which misses the point. While there are differences, as would be the case in any replication attempts, the key issue is whether readers should believe a study result that is so fragile. For example, what year the study was conducted is described as a difference, so is the proportion of females in the study (62% in the original study versus 49% in the replication). There are also differences of execution such as how many minutes the pose was held and what type of regression was used in the analysis. Even if these differences explain the inability to replicate the original finding, they would imply that the conclusion is dependent on those very conditions, and does not engender trust in its generalizability.

This situation is not unique to the "power pose" study. Over the years, Andrew and I have discussed many other studies with similar problems. This one is one of the few in which a replication has been attempted.

***

One of the points made in the Slate article is important to reiterate:

Through the mechanism called p-hacking or the garden of forking paths, any specific reported claim typically represents only one of many analyses that could have been performed on a dataset. A replication is cleaner: When an outside team is focusing on a particular comparison known ahead of time, there is less wiggle room, and results can be more clearly interpreted at face value.

This is a subtle point often missed by non-statisticians. There is a huge difference between a replication study for which researchers know in advance what is being analyzed, and a typical scientific study for which researchers may have measured an array of metrics, and then selectively reported ones that are "statistically significant."

Here is an analogy that may help understanding it:

You wanted to buy a woolen sweater during the after-Christmas sales at Macy’s Time Square. At the store, you discovered that woolen sweaters were hard to come by but cashmere scarves were the deal of the decade; in addition, Macy’s was running a buy-one, get-two free promotion on dress pants. So when you checked out, you purchased one cashmere scarf and three pairs of pants (none of which you had intended to buy). So we can ask whether the shopping trip was successful.

Imagine you have a tradition of going to Macy's every year to get a new sweater. If your metric of success is whether you purchased a sweater, your success rate would not be high. However, if your metric is whether you purchased something, your success rate would be much higher. A replication study has a fixed metric, fixed by the prior study: it is like measuring success based on whether you purchased a sweater.

It is much easier to prove that one of many things could happen than to prove that one specific thing would happen. The trouble is that in the reporting of scientific findings, one of many things is typically presented as one specific thing. This means replication is important: until we know that the one specific thing can be reliably replicated, we really don't have solid science.

Here is the link to our Slate article.

PS. This phenomenon is always hard to get across to students. I am not totally satisfied with this analogy. If you know of different ways to explain this, let me know.

PPS. This phenomenon is especially tricky with "big data" style studies. For example, a lot of people run "A/B tests" in which they simultaneously measure hundreds if not thousands of measures, and selectively report on the differences that are statistically significant.

 

 

Know your data 17: when other people can track your Uber location

The news is out that Uber got fined by the New York Attorney General's office for data breaches and privacy concerns. The headline writer for ZDNet nailed this one: "Uber fined peanuts in God View surveillance" (link). And the sub-lead has the kicker: "For a company with a valuation of over $50 billion, a $20,000 fine over user data protection is laughable."

This settlement tells us one of the following (or both) is not serious: the NYAG's attitude toward protecting consumer data privacy, or the valuation of the so-called unicorn.

But let's review the events that got Uber in trouble, and what it says about the state of ethics in data science.

As any Uber user knows, the service runs on an app, which means that the company has data tracking every Uber ride you took. Many users are fine with having such data collected and compiled by Uber, probably with the understanding that such data would not be used for purposes contrary to the consumer interest. Because the app runs on the smartphone, and payment is via credit cards, Uber certainly knows your identity, and also any additional information you provide on your profile. (Such data don't need to be merged with your travel logs but they almost surely will be merged by data scientists.)

At Uber offices, they are apparently very proud of a data visualization tool called God View. This is an aerial view of where all the Uber cars are, and where the Uber riders are.

There is a legitimate use of such a tool for managing the supply and demand of Uber cars, and for routing of cars.

Then the story gets muddier. Ethical issues often arise not because someone deliberately did something bad, but that something with a legitimate purpose is used, perhaps by other people, for more dubious purposes.

In November 2014, a Buzzfeed News reporter hired a Uber car to go meet with the GM of Uber NYC, Josh Mohrer. Upon arriving, Mohrer held out his iPhone, and remarked: "There you are, I was tracking you."

The underlying data processing technology is the same but two important distinctions must be made between God View and Mohrer's view.

First, for operations management, there is absolutely no need to use or even record any customer information--it doesn't matter whether it is John or Jane who is waiting for a Uber car at the corner of 14th and Madison; it only matters that John or Jane is one of 15 people all looking for rides within a two-block radius. But when Mohrer tracked the reporter, he was looking for a specific person, and this incident reveals that the travel log data are not anonymized.

Second, it is one thing for Uber to use the data internally for legitimate business reasons, such as managing supply and demand; it is a different thing to assess a customer's travel log, especially without first asking for that person's permission. The fact that a third party is able to do this without permission is quite disturbing. There are not many consumer-friendly use cases I can think of that requires looking up someone's past Uber rides.

Even before the Buzzfeed controversy, Peter Sims, a venture capitalist, already filed a similar complaint. Here's what he had to put up with. He was in a Uber car in New York City when an industry acquaintance, whom he "barely know," texted him from Chicago, and proceeded to trace his whereabouts during the trip.

It turned out that Uber was holding a gathering in Chicago in which someone was using Sims to demonstrate the power of Uber's data.

The informat remarkably showed no understanding of the privacy intrusion. She said the Uber Chicago event was "cool" and that Sims should be honored to have been selected for the demo.

Just think for a second, in his line of business, Sims could have been going to a meeting to strike an important business deal, which, for both financial and legal reasons, has to remain private until it is announced publicly.

***

One of the most concerning aspects of the massive data collection is the fact that once we allow data to be collected and stored in the "cloud" or corporate servers, we lose control of our own data. The idea that corporations are benevolent do not, and will not, stand the test of time.

And now I come to the juiciest part of this Uber story, what really got the NYAG involved. A Buzzfeed reporter was invited to a meeting between Uber SVP Emil Michael, and "an influential New York crowd."

During this meeting, Michael suggested that Uber would spend "a million dollars" to hire four top "opposition researchers" and four journalists in order to "help Uber fight against the press."

Specifically Michael was unhappy with Sarah Lacy, who was a website editor critical of Uber. Michael claimed that Uber's team "could prove a particular and very specific claim about her personal life."

People at the meeting suggested that such a move might present publicity problems, to which Michael said, "Nobody would know it was us."

After the Buzzfeed report, Michael and Uber's PR team now said that he made a mistake, and that the company "does not do oppo research of any sort on journalists" and "has never considered doing it".

Regardless of whether they have done it, it is clear that the data are available to dirt-diggers.

Uber also protested that the meeting was a "private dinner" and remarks were supposed to be "off the record." I am not sure how to interpret this. Are they saying it would be okay to carry out that plan so long as no one knows about it?

***
Look, these are not simple issues, and Uber is not an outlier. The answers to these questions go to the values of our society, and involve complicated trade-offs between conflicting goals. In particular, there are many things that corporations say they will not do which are well within their capability of doing. Unfortunately, in this case, the NYAG is setting a precedent that even when a company is found to have done something improper, it gets a light slap on the wrist.