« Response to Kohavi's note on non-inferiority in A/B testing | Main | The absurdity of predictive policing »


Feed You can follow this conversation by subscribing to the comment feed for this post.


One problem with any type of experiment is that there has usually been considerable investment in the new treatment. It may be money, time, or prestige. Some told me about going to talk to a company about the results of a medical trial which showed no effect. The end result was going to be that the R&D division would be shut down, as they only had one project. It was fairly hostile, as they had gone through the report trying to find anything that might be an error.


Ken: Yes, that's exactly why Kohavi wrote this note. This kind of scenario is very common. Typically, it's not as extreme as lots of people will be fired because of the "flat" finding. I have had my fair share of such meetings. The fault-finding mission is pointless because (a) the nitpicking doesn't change the high-level result enough to matter, and (b) proclaiming the test is botched does not cause a positive finding to materialize. It seems to me the way the pharma industry is set up, with super-high stakes and allowing pharmas to run and analyze their own trials, provides perverse incentive to cheat. I wonder why we allow that.

The comments to this entry are closed.

Get new posts by email:
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.


  • only in Big Data
Numbers Rule Your World:
Amazon - Barnes&Noble

Amazon - Barnes&Noble

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Principal Analytics Prep

Link to Principal Analytics Prep