Big Data, Plainly Spoken (aka Numbers Rule Your World)

If you are practicing your power pose, stop now

Andrew and I warned you about "power poses" in Slate some time ago (link).

Breaking news is that Dana Carney, a co-author of the paper that claimed the benefits of the power pose, has now confirmed that she no longer believes in the power pose. She is actively discouraging researchers from this "waste of time and resources."

Here is her statement (PDF link), which is well worth reading in full. This is a courageous statement.

The statement discloses a variety of tricks used to game p-values so that they meet the publishable 0.05 threshold. Everyone suspects someone else is playing such tricks but it's rare when someone actually confesses to them.

The highlights are:

Initially, the primary DV of interest was risk taking. We ran subjects in chunks and checked the effect along the way. It was something like 25 subjects run, then 10, then 7, then 5. Back then this did not seem like p-hacking. It seemed like saving money (assuming your effect size was big enough and p-value was the only issue)

 

Unfortunately, I have witnessed this type of p-hacking in industry all too often. In fact, many, many people run so-called A/B tests until they reach significance. There are many problems with what Carney described above. Imagine an effect size that is small (close to zero). As the samples accumulate, the measured effect will fluctuate around zero. If you wait long enough, the measure will hit p=0.05 by chance and then you stop. Further, they were reducing the sample size as the experiment continues - which means they are introducing more sampling variability, which means it is more likely that the measure will hit extreme values by chance!

 

It's tough for me to believe that she wasn't aware that stopping when you hit p=0.05 is p-hacking but that's what she is saying.

 

For the risk-taking DV: One p-value for a Pearson chi square was 0.052 and for the Likelihood ratio it was 0.05. The smaller of the two was reported... I had found evidence that it is more appropriate to use "Likelihood" when one has smaller samples and this was how I convinced myself it was OK.

 

She's focused here on the researcher degree of freedom issue. The larger problem is the magic dust that seems to sprinkle off p=0.05. If that is the chosen threshold for significance, and my result is right on the cusp, I would be very skeptical of this result. I don't think 0.052 is the better number - they are both bad.

The self-reported DV was p-hacked in that many different power questions were asked and those chosen were the ones that "worked".

Many A/B testing platforms come with a battery of hundreds of metrics automatically computed for each test. No further comment needed.

As of today, the TED talk on "power poses" is still going strong. It has accumulated 36 million "views" and the official description does not mention Dana Carney's retraction.

 

 

Inside joke for data analysts (via Gelman)

Andrew Gelman suggested this sign in a recent blog post:

This is patterned after a similar sign that used to show up at auto mechanics:

For those needing background, see this post.

Making sense of the most detailed map of gay marriages

Last week, my Columbia students discussed this nice article in the New York Times called "The Most Detailed Map of Gay Marriages in America". (link)

The center of the article is this map:

I asked the students to identify the problem that this dataviz is supposed to address. Someone responded that it tells us where gay married couples are found geographically. I asked her what is the answer to the question. She said they are mostly concentrated in the coasts and there are relatively few in middle America.

Then, I asked what assumption was needed in order for that comment to be valid, something that is not found on the map itself. If you look at the legend of the map, you'll see that the data being plotted are proportions of married couples that are same-sex, not counts of same-sex couples. Thus, when one makes that conclusion about coastal skew, one is using the knowledge that the population of the U.S. is concentrated on the coasts. (If the population density is higher in the middle than on the coasts by a certain degree, there could be more same-sex couples in the middle despite the lighter shades of orange.)

Next, I wanted an answer to a simple question: which (three-digit) area code (the unit of analysis on this map) has the highest density of same-sex married couples? Students scratched their heads. This "most detailed" map is not well equipped with the answer. The issue raised is whether the amount of details is an obvious virtue of a dataviz.

We then touched upon two other topics that are also very important:

1) How was this data obtained and computed? The author does a great job explaining two ways of arriving at these counts, a sample survey versus administrative records (tax filings).

2) How should the data be interpreted? The author walks through a number of unexpected comparisons, all of which point to the importance of statistical controls. For example, why do same-sex couples make more money than opposite-sex couples? Why do lesbian couples make less than gay-male couples?

I highly recommend reading this article closely.

PS. At the end of the class, one student approached me to suggest that I should not have wasted class time discussing this article.

 

Know your data 20: trust and distrust in our surveillance society

A very important article from the Times starts with the following sentence:

Want to invisibly spy on 10 iPhone owners without their knowledge? Gather their every keystroke, sound, message and location? That will cost you $650,000, plus a $500,000 setup fee with an Israeli outfit called the NSO Group.

In the U.S., there is a disconnect between a populace whose distrust of government is at an all-time high and the same people whose trust of the same government's handling of their private data has apparently been unshaken.

***

NSO, the company being profiled, claims to have an "ethics" committee to determine whom they will sell (out) and whom they won't. It's not clear whether this "ethics" committee vets the methods used by NSO to collect data, such as "baiting targets to click unwittingly on texts containing malicious links".

***

This bit is also interesting:

Pegasus [the surveillance system] can use the camera to take snapshots or screen grabs. It can deny the phone access to certain websites and applications, and it can grab search histories or anything viewed with the phone’s web browser. And all of the data can be sent back to the agency’s server in real time.

What might make it easier for others to remote control your gadget life? More connected devices (e.g. internet of things), putting stuff in the cloud, simple user interfaces that hide inner functions, always-on connections, auto-updating, real-time anything, never logging out of accounts, etc. etc. Basically, anything that allows software to establish connections to remote servers without identifying itself.