« Story time, known unknowns and the endowment effect in an HBR article on customer data | Main | Talking past the consumer on customer data »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

ceolaf

I’ve a HUGE problem with the “more probably than not” standard used in this report.

We have a tradition in this countries of going wiht a “preponderance of the evidence” standard in civil court cases. I think this standard is a bit problematic, because it does not allow for any sort of uncertainty. The court must find for one side or the other, as though the jury (or judge) can meaningfully tell the difference between 49-51 and 51-49. My guess is that what acutally happens is that the default is really not ot award damanges (i.e., to side with with the defendent) unless the evidence is sufficently beyond a prepdondance + allowed margin of error.

So, I’ve a problem with that standard for civil trials. But at least in those situations, but sides gets to present their best case and to provide evidence ot undermine the other side’s case. That’s an adversarial system.

But this kind of report does NOT make sure of an adversarial system. Rather, it is an inquisitorial system. There is just one “side,” subject to whatever biases s/he/it might have, search for whatever evidence s/he/it would like to examine. There is no mechanism to ensure that alternative or exculpatory evidence/reasoning is factored in.

In other words, there is room for a HUGE selection bias in the evidence examined, and therefore this idea of a preponderance of the evidence is just crazy. Which bring us to this “more probable than not” idea. Of course, that would depend on your probability model. But there is not probability model here. Rather, there is a story and some (biased) collection of investigation of the story.

Yes, I believe that virtually all investigations are biased. That is why we have an adversarial system in our courts and that is why we have peer review and replication in science.

The offensive thing here is that the NFL seems to be saying, “Well, we are convinced.” But the NFL convinced itself.

And that’s a lousy way to use data.

The comments to this entry are closed.

Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.
Numbers Rule Your World:
Amazon - Barnes&Noble

Numbersense:
Amazon - Barnes&Noble

Search3

  • only in Big Data

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Courses

R Fundamentals, Principal Analytics Prep

Numbersense: Statistical Reasoning in Practice, Principal Analytics Prep

Applied Analytics Frameworks & Methods, Columbia

The Art of Data Visualization, NYU

Signed copies at McNally-Jackson, NYC

Excerpts: Numbersense Ch. 1, 7, 8. NRYW

Junk Charts Blog



Link to junkcharts

Graphics design by Amanda Lee

Community