« How to act like a data scientist 10: digging for information in unlikely places | Main | It begins with the data »


Feed You can follow this conversation by subscribing to the comment feed for this post.


Do you think that there may be an inherent bias in "Big Data" practise toward data set collection and post processing at the cost of preprocessing and methodological rigour? Or is this an isolated case?


GIGO: This is certainly not an isolated case. Most big data type analyses are even simpler. Most analysts assume the data they have are the best they've got. Another example is the restaurant analysis using Opentable data. Is Opentable a global monopoly such that its data represent not just the entire global population of restaurants but also the population of restaurants at each local level of analysis? (rhetorical)

I may have written about this somewhere else but my fear is that big data analyses adopt the "complete information" assumption. This comes from two axioms: one, that we have data on the entire population (the "seemingly all" in the OCCAM defintion); two, because of Axiom #1, plus the oft-mistaken belief that unknown selection equals random selection, that we have unbiased data. From the "complete information" assumption comes the idea that all variation is random variation. But random variation is minmized by Axiom #1. Thus, you can take whatever you see in the big data and generalize it.

There are pockets of researchers who are treating these problems seriously but their influence is restricted to publishing papers. Dealing with these issues head-on almost surely requires post-processing and adjustments, which I think is not popular yet in this field.

Thanks for the comment. I think I will turn this into a post.


Look forward to it. I admit even I was impressed at the the size of the response they got from this app. I have a feeling their symptoms response section not v well thought through. Not enough external input,?

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name is required. Email address will not be displayed with the comment.)

Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR.

See my Youtube and Flickr.
Numbers Rule Your World:
Amazon - Barnes&Noble

Amazon - Barnes&Noble


  • only in Big Data

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here


R Fundamentals, Principal Analytics Prep

Numbersense: Statistical Reasoning in Practice, Principal Analytics Prep

Applied Analytics Frameworks & Methods, Columbia

The Art of Data Visualization, NYU

Signed copies at McNally-Jackson, NYC

Excerpts: Numbersense Ch. 1, 7, 8. NRYW

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee