« The many laws of large numbers | Main | Agreeing to disagree »


Feed You can follow this conversation by subscribing to the comment feed for this post.


But are the climate scientists always correcting obvious errors in the data and reducing measurement error when they "clean" the data? In many--probably most--cases, they are, but the mismanagement of some of the data and the politicking in the climate research community does raise doubts--in my mind, at least. When you combine opacity of methods and lack of reproducibility in climate data management with sensitivity of climate models to inputs--to say nothing of the incentives resulting from politicization--you're leaning pretty hard on the integrity and the infallibility of the climate scientists.


Don Wheeler has made similar points many times in his books and articles. His advice, like yours above, is to first check the data for homogeneity (using process behavior charts). Coupled with this is his admonition that all outliers are evidence...though the evidence may be of problems in data collection rather than the process being studied.

I think that this first step in data analysis is under-appreciated, even among the scientific community. Perhaps the techniques for checking data are not taught in a rigorous way, and as a result, everyone cleans their data out of necessity, but no one is comfortable defending the process.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Marketing and advertising analytics expert. Author and Speaker. Currently at Vimeo and NYU. See my full bio.

Next Events

Mar: 26 Agilone Webinar "How to Build Data Driven Marketing Teams"

Apr: 4 Analytically Speaking Webcast, by JMP, with Alberto Cairo

May: 19-21 Midwest Biopharmaceutical Statistics Workshop, Muncie, IN

May: 25-28 Statistical Society of Canada Conference, Toronto

June: 16-19 Predictive Analytics World (Keynote), Chicago

Past Events

Feb: 27 Data-Driven Marketing Summit by Agilone, San Francisco

Dec: 12 Brand Innovators Big Data Event

Nov: 20 NC State Invited Big Data Seminar

Nov 5: Social Media Today Webinar

Nov: 1 LISA Conference

Oct: 29 NYU Coles Science Center

Oct: 9 Princeton Tech Meetup

Oct: 8 NYU Bookstore, NYC


Jul: 30 BIG Frontier, Chicago

May: 30 Book Expo, NYC

Apr: 4 New York Public Library Labs and Leaders in Software and Art Data Viz Panel, NYC

Mar: 22 INFORMS NY Student-Practitioner Forum on Analytics, NYC

Oct: 19 Predictive Analytics World, NYC

Jul: 30 JSM, Miami

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee


  • only in Big Data