« The many laws of large numbers | Main | Agreeing to disagree »


Feed You can follow this conversation by subscribing to the comment feed for this post.


But are the climate scientists always correcting obvious errors in the data and reducing measurement error when they "clean" the data? In many--probably most--cases, they are, but the mismanagement of some of the data and the politicking in the climate research community does raise doubts--in my mind, at least. When you combine opacity of methods and lack of reproducibility in climate data management with sensitivity of climate models to inputs--to say nothing of the incentives resulting from politicization--you're leaning pretty hard on the integrity and the infallibility of the climate scientists.


Don Wheeler has made similar points many times in his books and articles. His advice, like yours above, is to first check the data for homogeneity (using process behavior charts). Coupled with this is his admonition that all outliers are evidence...though the evidence may be of problems in data collection rather than the process being studied.

I think that this first step in data analysis is under-appreciated, even among the scientific community. Perhaps the techniques for checking data are not taught in a rigorous way, and as a result, everyone cleans their data out of necessity, but no one is comfortable defending the process.

The comments to this entry are closed.

Get new posts by email:
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.


  • only in Big Data
Numbers Rule Your World:
Amazon - Barnes&Noble

Amazon - Barnes&Noble

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Principal Analytics Prep

Link to Principal Analytics Prep