« The plural of anecdote is not ... | Main | Reader's guide to the power pose controversy 2 »


Feed You can follow this conversation by subscribing to the comment feed for this post.


I enjoyed Freakonomics but also found some of the analyses a bit worrying. Given that I think economists don't understand some things about economies then it is unlikely that they have corrected properly in any longitudinal analysis. It is interesting that there is another theory about reduced levels of violent crime being due to changes to abortion laws and that is it is due to reduced lead. Both have their problems as there is so much confounding with socio-economic status.

21st century economics seems to be able to model everything except economies. I'm waiting for GFC 2 where economists will have to find another reason why they didn't predict it.



I like your post but think you have used "generalizable" where "replicable" would be better.

Correct me if I'm wrong. My sense of these things is that a replicable experiment is one where, taking then same population, the same effect will be found, with perhaps minor differences because the samples from the population will not be identical.

Generalizability raises the question of whether a same/similar effect will be found in another population. Case in point: effects observed in small samples of psychology students might not be generalizable to a wider, more diverse, perhaps less self-selected a population.

An experiment that is replicable in a not very interesting population is not very interesting. A generalizable one crosses age/gender/personality/geography etc to be a genuinely useful insight.


CfE: You raise a point that probably merits a different post. Because the current controversy isn't about sampling bias, I have stayed away from that subject. Replicability is necessary but not sufficient for generalizablity. In the framework for p-values, we assume that the sample is representative of the population to which one wants to generalize the result. In reality, you're opening a can of worms here: I think another huge concern with any of these social science experiments is the reliance on samples of students when conclusions are drawn on the population.

That said, I don't see why the replication movement can't extend beyond repeating the original experiment. Seems like applying the same experimental design on a different sample (such as not students) is a worthwhile undertaking as well.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)


Link to Principal Analytics Prep

See our curriculum, instructors. Apply.
Business analytics and data visualization expert. Author and Speaker. Founder of Principal Analytics Prep, MS Applied Analytics at Columbia. See my full bio.

Next Events

Oct: 31 Webinar on Data Visualization, online at JMP

Nov: 1 NYU unCOMMON Salon Public Lecture, New York, NY

Nov: 8 Tufts Gordon Institute: A Conversation with Kaiser Fung, Facebook Live

Nov: 8 Tufts TGI Careers & Networking Night panel, Somerville, MA

Nov: 26 Data Visualization New York Meetup, New York, NY

Nov: 27 NYPL Data Analytics Resume Workshop, New York, NY

Nov: 30 Purdue School of Engineering Seminar, West Lafayette, IN

Dec: 1 Purdue Mathematics, Data Science, and Industry Conference, West Lafayette, IN

Past Events

See here

Future Courses (New York)

Summer: Statistical Reasoning & Numbersense, Principal Analytics Prep (4 weeks)

Summer: Applied Analytics Frameworks & Methods, Columbia (6 weeks)

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee


  • only in Big Data