« Know your data 17: when other people can track your Uber location | Main | Know your data 18: your location is our product »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

John Butters

Thanks -- very interesting.

Is the following analogy any good?

Study: You go to Selfridges with the intention of looking for nice clothes. You hate the shirts, the hats, the shoes and the scarves, but you do find two nice things: a lovely jumper and some trousers. Success! You publish a study called, "Significant Evidence of Quality in Selfridges' Jumper and Trouser Buying".

Replication: Your wife loves how you look in the jumper. Sadly, one day, you catch it on some barbed wire while out shooting. Your wife goes so Selfridges with the intention of buying the same jumper, or one that is fairly similar. But the stock has changed and there is nothing that she likes. Your wife publishes a study called, "Jumper Buying at Selfridges: The Fallacies of Statistical Hope".

Yes, I have taken a childish delight in anglicising this. Sorry!

Chris

Randall's example always makes me smile...

https://xkcd.com/882/

Ken

The results have to be considered taking into account the huge number of studies that are conducted. Psychology departments often have a course that requires students to take part in a study so they have no trouble obtaining participants. In this case they obviously had some money so could also do some lab tests, although sometimes this is done by students. One aspect that makes the results look better is that all tests are significant but it is likely that the outcomes are correlated. So by chance they get a significant result.

I'm surprised that there isn't more replication, especially when the studies are not difficult. Great way to get citations. The only problem may be that often these studies results aren't replicated so it may be difficult to be published unless the replication study is better in some way, like larger sample size.

The comments to this entry are closed.

Get new posts by email:
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.

Search3

  • only in Big Data
Numbers Rule Your World:
Amazon - Barnes&Noble

Numbersense:
Amazon - Barnes&Noble

Junk Charts Blog



Link to junkcharts

Graphics design by Amanda Lee

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Principal Analytics Prep



Link to Principal Analytics Prep

Community