« Response to Kohavi's note on non-inferiority in A/B testing | Main | The absurdity of predictive policing »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Ken

One problem with any type of experiment is that there has usually been considerable investment in the new treatment. It may be money, time, or prestige. Some told me about going to talk to a company about the results of a medical trial which showed no effect. The end result was going to be that the R&D division would be shut down, as they only had one project. It was fairly hostile, as they had gone through the report trying to find anything that might be an error.

Kaiser

Ken: Yes, that's exactly why Kohavi wrote this note. This kind of scenario is very common. Typically, it's not as extreme as lots of people will be fired because of the "flat" finding. I have had my fair share of such meetings. The fault-finding mission is pointless because (a) the nitpicking doesn't change the high-level result enough to matter, and (b) proclaiming the test is botched does not cause a positive finding to materialize. It seems to me the way the pharma industry is set up, with super-high stakes and allowing pharmas to run and analyze their own trials, provides perverse incentive to cheat. I wonder why we allow that.

The comments to this entry are closed.

Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR.

See my Youtube and Flickr.
Numbers Rule Your World:
Amazon - Barnes&Noble

Numbersense:
Amazon - Barnes&Noble

Search3

  • only in Big Data

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Courses

R Fundamentals, Principal Analytics Prep

Numbersense: Statistical Reasoning in Practice, Principal Analytics Prep

Applied Analytics Frameworks & Methods, Columbia

The Art of Data Visualization, NYU

Signed copies at McNally-Jackson, NYC

Excerpts: Numbersense Ch. 1, 7, 8. NRYW

Junk Charts Blog



Link to junkcharts

Graphics design by Amanda Lee

Community