Big Data, Plainly Spoken (aka Numbers Rule Your World)

The data fix

Rachel Thomas's article came onto my twitter feed. It caught my attention because it has a click-baity title "How (and why) to create a good validation set."

Or, I thought it was click bait but she is really serious about this. (For those not familiar with the literature, we don't use all historical data to build machine learning models. The historical data are split, typically at random, into training and validation sets. The validation set is supposed to simulate new data the algorithms haven't seen before, a sort of honest check of the model.) She makes some alarmist claims here:

• there is such a thing as a "poorly chosen" validation set
• random selection is not a good way to make a validation set, a "poor choice for many real-world problems"
• the analyst should manufacture a validation set
• the validation set should be representative of future, currently-unseen, data

Even though I don't like any of her advice, I can't disagree with her diagnosis:

An all-too-common scenario: a seemingly impressive machine learning model is a complete failure when implemented in production. The fallout includes leaders who are now skeptical of machine learning and reluctant to try it again.

***

One of the examples given is a response function that has a time trend.

If this model does not detect the trend, indeed the prediction will have poor accuracy on  real-world data. She is making a claim that a validation set based on a pre-post time split is better than a random selection.

Since this is a simple linear trend, either way of making the validation set will capture the trend. So what makes the model fail in production is not the presence of this trend but a shift of the trajectory after the model is deployed. But the choice of validation set won't help prevent the problem.

The downside of the pre-post split is the presence of many time-varying predictors. A naive example: if an on-off switch just happens to be pressed at that time split point, then all your training examples have the "on" condition while your validation examples have "off".

Manufacturing the validation set to reflect some unknown future trends creates a conceptual difficulty. The training set is now materially different from the validation set, so why would we expect the trained model to perform well on the validation set? And how much degradation in validation set performance is considered the right price to pay for potentially better in-market performance? That question boils down to how much you want to generalize the data, and at the core of the statistical view of the modeling problem.

***

The subtext of the article is that if the model doesn't work, fix the data. I tend to want to fix the model. If it doesn't work in production because the nature of the time trend has shifted, then adjust the model to include the new time trend.

Diagnosing the difference between production data and historical data is part of good model hygiene. It's very hard to predict unexpected shifts in the data and even if you could, you wouldn't have any training data to support such shifts.

The "data fix" is not the solution. Refining one's model is.

 

PS. While I don't agree with designing your validation set, I do advise selecting your historical dataset carefully, and think about which units to include or exclude from the modeling process, which Rachel discusses at the end of her post.

 

A/B Testing Primer and the DEED framework

My short lecture on A/B testing went live yesterday, streamed on Facebook Live from Harvard Business Review's office in Boston. You can watch the recording here.

The outline of the talk is as follows:

• Expands upon the HBR article that Amy Gallo and I collaborated on titled "A refresher on A/B testing"
• Uses a real A/B test as a running example throughout: this test was run about five years ago in reaction to an alarming trend of website users migrating en masse to mobile devices. We tested a mobile-optimized conversion page, which is the webpage where our users sign up and pay for our service
• Explains the two distinguishing features of an A/B test: the comparison with a control group, and the random assignment of treatment
• Discusses why a pre-post arrangement in which version A is shut down and version B is turned on at a point in time is not an A/B test, and not as good as an A/B test
• Introduces the DEED framework - four key stages in a practical A/B testing process: Design, Execute, Examine, Deploy
• Most teams focus energy on Execute and Examine but the more sophisticated, analytically advanced organizations spend a lot of time on the Design and the Deployment phases. I reckon my teams spend at least 50% of our time on Design
• Design is much more than linear algebra. It's an organizational alignment exercise.
• Explains why many executives believe that most of their A/B tests "fail". I previously wrote about this here and here
• It is mistaken to think that a test fails when it fails to prove out a hypothesis
• Uses the DEED framework to describe real reasons for tests failing: bad Design, bad Execution, bad Examination, bad Deployment
• Gives an example of each type of failure in the context of the mobile optimization test
• In particular, you should not inspect a report with a hundred metrics and pick out the ones that pass the statistical significance filter. For more on researcher's degrees of freedom, see the materials about the replication crisis here or on Andrew Gelman's blog
• Describes why A/B testing is an organization-wide endeavor that requires bringing a broad set of players from the onset, and faciliated by a project manager
• Discusses what skills are required at the different stages of DEED, and how to build the project team

 Now watch the video here.

My letter to the New York Times is published

Some of you might be wondering why I haven’t commented on the front-page feature in New York Times Magazine two weeks ago, titled “When the Revolution Came for Amy Cuddy.” Susan Dominus, the author of the front-page feature, cited an article about Cuddy's "power pose" research that Andrew Gelman and I co-authored in 2016 as an example of a vicious personal criticism on Cuddy, in fact, one of the very few actual examples of vicious personal criticism that Dominus repeatedly told readers has destroyed civility in the field of social psychology.

You can now read my response to the NYT Magazine article in the now-published Letter to the Editor. (Scroll to the second entry.) I didn’t say much before because I didn’t want to front-run the letter but I do have lots to say. Since the NYT letter is limited to 200 words, it is a much abridged version of what I had in mind.

I find Susan’s interpretation of the events to be unbalanced. If you are a curious, or serious, reader, I urge you to follow the links below (in addition to reading Dominus's feature article), and form your independent judgment of the movement to reform the use of data in psychological experiments.

1. Susan Fiske, who is the academic advisor of Amy Cuddy at Princeton, is mentioned in the NYT Magazine. An important contribution of hers to this debate is a "Presidential Guest Column" in APS Observer, titled "Mob Rule or Wisdom of Crowds?," in which she coined the term "methodological terrorism." Prof. Fiske is a past president of Association for Psychological Science (APS). The entire column can be read here.
   
   
2. The NYTM article mentioned a “combative” response by John Bargh, Yale professor and father of the seminal priming study, to the failed replication study. He published his response on Psychology Today, a blogging platform. The blog post was subsequently deleted but you can find the post titled “Nothing in Their Heads” archived here.
   
   
3. Daniel Kahneman's letter to priming researchers is cited by Dominus, and deserves to be read in full. Dan Goldstein has a copy of it here. There are many good quotes, including "Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published."
   
   
4. Andrew Gelman, professor of statistics at Columbia, regularly blogs about the validity of social science research at his popular blog. As Dominus noted, the power pose research started by Dana Carney, Amy Cuddy and Andy Yap is one recurring topic on the blog.  Gelman has many other favorite, recurring targets, for example, Daryl Bem (ESP research, Cornell), Brian Wansink (Food and Brand Lab @ Cornell), Richard Tol (climate economics), Roy Baumeister (ego depletion), Satoshi Kanazawa (evolutionary psychology, LSE), etc. I can't help but notice that almost everyone on this list is male.
   
   
5. The Data Colada website, run by Uri Simonsohn and Joe Simmons, is only one click away. Given that a whole meal was made of the inner workings of a single blog post that criticized the Carney, et. al. study, the reader should absolutely check out this website where serious scientists are discussing how to reform science.
   
   
6. The original paper by Carney, Cuddy and Yap (2010) that launched this power pose research program is found here. If you don't have time to read the whole paper, at least read the abstract, for what their original scientific claim was:
   
   Humans and other animals express power through open, expansive postures, and they express powerlessness through closed, contractive postures. But can these postures actually cause power? The results of this study confirmed our prediction that posing in high-power nonverbal displays (as opposed to low-power nonverbal displays) would cause neuroendocrine and behavioral changes for both male and female participants: High-power posers experienced elevations in testosterone, decreases in cortisol, and increased feelings of power and tolerance for risk; low-power posers exhibited the opposite pattern. In short, posing in displays of power caused advantaged and adaptive psychological, physiological, and behavioral changes, and these findings suggest that embodiment extends beyond mere thinking and feeling, to physiology and subsequent behavioral choices. That a person can, by assuming two simple 1-min poses, embody power and instantly become more powerful has real-world, actionable implications.
   
   
7. Here is the Ranehill, et. al. replication study. Here is the full text of Dana Carney's renunciation of power pose research, in which she disclosed how the steps taken during the data analysis could have led to a false-positive finding. Here is the full press release by Amy Cuddy's publicist to New York Magazine, which is Cuddy's first public response to the Ranehill et. al. study.
   
   
8. Andrew Gelman has given a series of talks to academic conferences and academic departments on how to reform the use of data evidence in psychology and/or social science. For example, ESRC Research Methods Festival at University of Bath, Columbia Computational Social Science, Centre for Research in Statistical Methodology at Warwick, Department of Psychology at UT Austin, Psych Colloq at Harvard, Psychology/Statistics at Ohio State University, etc. etc. This is not a comprehensive list - I used Bing and Gelman's blog and all these showed up just in the first page of results.
   
   
9. Here is the article Andrew and I co-authored about our general concern about popular press and social media promulgating peer-reviewed studies that have weak statistical foundations, using the power pose research as an example.

****

One thing to which the NYTM article did not do justice is the underlying, deep issues related to how evidence from data is used in social science. In November 2016, I wrote a series of blog posts to explain the statistical issues being discussed. The agenda:

Key Idea 1: Peer Review, Manuscripts, Pop Science and TED Talks (link)

Key Idea 2: P < 0.05, P-hacking, Replication Studies, Pre-registration (link)

Key Idea 3: Negative Studies, and the File Drawer (link)

Key Idea 4: Degrees of Freedom, and the Garden of Forking Paths (link)

Key Idea 5: Sample Size (link)