« Experts clinging on to their predictions | Main | Count data are less useful than you think »


Feed You can follow this conversation by subscribing to the comment feed for this post.


What is OCCAM data? I did Google it, didn't turn up anything in the first couple of pages. And a scan of the NYT article didn't help either.


Andy: see this link.


Kaiser, I think OCCAM is a great framework for thinking about big data issues, but I would recommend you provide that link every time you mention OCCAM. Not all of us are regular readers.


Many of the problems seem to be missing data problems, and that is something that statisticians should be expert in.


zbicyclist: Point taken. I was in a bit of a hurry when I wrote that post.

Ken: yes, many issues like biased samples and censoring are nicely modeled as missing data problems. Another big issue I'm concerned about is having nuisance predictors while also missing relevant predictors. Because of observational data, we frequently don't have the right set of predictors. A third issue would be a network of direct and indirect effects. The general point is that the complexity of the analysis is increased, not decreased, by having "big data".


To add, the problem with the clotting agent is similar to one that occurs in diagnostic testing. Only those that have a positive result on a screening test actually have a better test. This effectively results in lots of missing data as most subjects don't have the better test, which results in severe biases if the data is not treated correctly.

When looking at big data for medicine we are going to have a mess of different tests and possibly treatments that will obscure any possible diagnosis, as you will never know if the disease just went away or was fixed if you don't know if there was a disease in the first place. I expect there will be some very nice Bayesian analyses that will demonstrate that standard data-mining techniques just aren't enough.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)


Link to Principal Analytics Prep

See our curriculum, instructors. Apply.
Business analytics and data visualization expert. Author and Speaker. Founder of Principal Analytics Prep, MS Applied Analytics at Columbia. See my full bio.

Next Events

July: 24 Data Analytics Resume Workshop, NYC

July: 30 Joint Statistical Meetings, Vancouver

Aug: 28 Swiss Statistics Meeting, Zurich

Sep: 6 Data Visualization Seminar, San Diego, CA

Sep: 12 NYPL Analytics Careers Talk, NYC

Past Events

See here

Future Courses (New York)

Summer: Statistical Reasoning & Numbersense, Principal Analytics Prep (4 weeks)

Summer: Applied Analytics Frameworks & Methods, Columbia (6 weeks)

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee


  • only in Big Data