« What is an interaction effect? | Main | It is possible to not learn real causes from some A/B tests »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Ken

Looking at the brain imaging studies,there are a lot of 20 subject studies and it seems to be a standard that originates from when they were extremely expensive and even the best researchers could afford that number. Now it seems to be the number that someone with minimal funding can do. Then they apply a lot of different outcomes, looking at different parts of the brain, and different statistics. Amazingly, something always seems to be significant.

Unfortunately while the journals accept that then this poor research will be published. It would be better to run larger studies, but there is now an expectation of the number of papers that a researcher authors.

Adam Schwartz

Regarding publication bias, are you aware of any venues which allow researchers to share these "non events" to help prune the search space? As it exists, it seems like the process starts with an "interesting" finding which gets published and then (maybe) gets replicated to demonstrate the experiment can be reproduced or not.

Even assuming there was some outlet that allowed "failed" experiments to be shared, do you think there's a bias among researchers to p-hack or whatever in order to publish something more often than not - publish and perish and all that...

Ken

@Adam There is a discussion on outlets for studies at https://www.researchgate.net/post/Who_knows_about_journals_preferably_publishing_negative_results_from_pharmacological_clinical_trials

An obvious one is http://www.jnrbm.com

There has been research that shows that well-designed trials tend to be published irrespective of what the result is. Telling the world that a drug doesn't reduce blood pressure and giving a reasonably tight confidence interval is worthwhile. I suspect it is more the drug companies that don't want that published who are the problem.

Adam Schwartz

@Ken, thanks for the leads!

The comments to this entry are closed.

Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.
Numbers Rule Your World:
Amazon - Barnes&Noble

Numbersense:
Amazon - Barnes&Noble

Search3

  • only in Big Data

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Courses

R Fundamentals, Principal Analytics Prep

Numbersense: Statistical Reasoning in Practice, Principal Analytics Prep

Applied Analytics Frameworks & Methods, Columbia

The Art of Data Visualization, NYU

Signed copies at McNally-Jackson, NYC

Excerpts: Numbersense Ch. 1, 7, 8. NRYW

Junk Charts Blog



Link to junkcharts

Graphics design by Amanda Lee

Community