Big Data, Plainly Spoken (aka Numbers Rule Your World)

Statistical thinking

For a detailed introduction to the book, you can now read the sample chapter, which the publisher kindly made available here.

***

Statisticians will be the sexy job in the next 10 years, so said Google's chief economist, Hal Varian. In many ways, 2009 was a banner year for business analytics. IBM, a giant of corporate IT and consulting, shelled out $1.2 billion to buy SPSS, one of the most well-known vendors of analytics software. The #1 analytics software provider, SAS, still privately-owned, enjoyed record earnings (see co-founder John Sall's take here, in which he mentions my book). The Netflix contest -- a showcase for how statistics influence our entertainment choices -- came to a thrilling conclusion. Amazingly, the R software, long a favorite of statisticians, got some exposure in the New York Times.

But -- if you ask anyone who works in a statistics or analytics function in industry, you'll find that there is a severe shortage of properly-trained workers. It turns out statistical thinking is a specialized skill that does not come naturally to us. This was first observed by psychologist Daniel Kahneman: yes, the same Kahneman who won the Nobel Prize for his foundational work on behavioral economics (both lines of research done in collaboration with Amos Tversky).

***

Based on the results of a series of experiments, Kahneman and Tversky hypothesized that our natural way of thinking is non-statistical. Put differently, they found that statistical thinking is often non-intuitive. Consider the following question:

Consider two hospitals, one with 15 child births per day, the other with 45. Which hospital will have more days with 60% or more male births? Or will both experience the same frequency? 

Most people believed that the frequency would be the same, since each birth should be roughly equally likely to be male as female, regardless of day or hospital.

Statistical thinking leads to the realization that the smaller hospital will have a greater number of days with large excess male births because the variability is much higher (this also means the smaller hospital will have more days with large excess female births).  Variability is higher when the sample size is smaller.

This is no mere toy problem: Howard Wainer's evisceration of the rationale behind the small schools movement relied on the same idea. (See my discussion here.)  In Chapter 1 of Numbers Rule Your World, I look at why statisticians obsess about the notion of "variability".

***

Not long ago, statisticians could hardly dream of an article in the New York Times titled "For Today's Graduate, Just One Word: Statistics". For graduates, and anyone else interested in number-crunching, recognize that statistical thinking is learnt and developed over time.  A good degree is nice but practical experience is crucial.  

Four challenges, five principles

The following first appeared on Junk Charts on Feb 12:

At one level, Numbers Rule Your World (see here) is a primer on statistical thinking. If you are reading Junk Charts, you already know its importance.

In putting together the book, I gave myself these four challenges:

• No equations

In order to make the book accessible to as many as possible, I borrowed the story-telling style of Freakonomics and The Tipping Point. This, the need to transform numbers into words, comes naturally having worked in business analytics for a long time; readers of Junk Charts will recognize how I always look for the message behind the numbers. And no equations means no equations.

• No toy problems

No Monty Hall problem, no birthday problem, no urn problems, no St. Petersburg's paradox.  Not only have these topics been covered well by others, they are good for teaching but ultimately unrealistic.  I want to cover statisticians who have harnessed real data to make socially important decisions, such as telling us what makes us sick, setting insurance rates, evaluating SAT questions, catching thiefs.

• Long-form stories

The book is organized around five statistical principles, with a pair of stories illuminating broad aspects of each principle. Each story is developed in rich detail, bringing out the players, the background, the numbers, the conflicts that form the process of applying science. In this way, the book's structure is different from The Tipping Point and Freakonomics, which can sometimes feel episodic.

• Contemporary examples

Nearly everything in the book occurred in the last two or so decades.  Statistical thinking is evergreen.  I left the standard historical examples on the cutting floor -- Fisher's tea-tasting ladies, Student's brewery experiments, Galton's regression studies, etc. (For those who enjoy history, and can read math, Stigler's two books on the history of statistics are not to be missed.)

***

Here are the five key principles:

1. The discontent of being averaged: Always ask about variability

2. The virtue of being wrong: Pick useful over true

3. The dilemma of being together: Compare like with like

4. The sway of being asymmetric: Heed the give-and-take of two errors

5. The power of being impossible: Don't believe what is too rare to be true

Come back for more on these, or get the book.

***

Book available from Amazon, B&N, Borders, and also worldwide

The Beginning

The following first appeared on Junk Charts on Feb 11:

One of the reasons I started Junk Charts five years ago was to create an outlet for my writing.  To manufacture a reason to write regularly.  Writing has been important to me since young, but for a long time, I did not have the discipline to sit down and produce.  Once the blog found its legs, the need to satisfy a responsive, attentive, demanding audience dissolves the problem. 

Then I got more ambitious: the blog experiment soon parlayed into a book project.  The book project started close to Junk Charts but eventually took its own path.  While Junk Charts focuses on how data is, and should be, presented in the mass media, and how the media influences public opinion through such presentation, the book takes a more expansive look at how data-driven decision-making occurs in our world: how statistical thinking shapes our work life, our health, our economy, our education, etc., often in hidden and surprising ways.

The title is "Numbers Rule Your World: The hidden influence of probability and statistics on everything you do."  It is published by McGraw-Hill; the official publication date is February 19.  It is already available from various online retailers. (Amazon, B&N, Borders)  My neighborhood bookstore has it out already.

***

This is a timely topic.  So many current events are influenced strongly by statistical thinking:

• Counter-terrorism: how do statistical technologies, such as profiling and prediction via data mining, work? what are the benefits and costs of such methods? what is the risk of dying in a terrorist attack relative to other risks?
• Toyota recall: do recalls work: how many deaths do they prevent, how much economic losses do they cause? what is the risk of dying due to the brake defect relative to other risks? how can one establish the brake defect as the cause of prior accidents?
• Mammogram guidelines: what is the statistical science behind the revised guidelines? why is this not a black and white issue? how should we evaluate research studies of this kind?
• Economic statistics: what is the "seasonal adjustment" used to modify employment data? what is the birth death model? should one look at quarter to quarter comparisons or year to year? why is it useless to compare same-store sales statistics for retailers in the past few years? 
• Efficacy of anti-depressants: what is the placebo effect? do anti-depressants work for some but not for others? how are clinical trials conducted, funded and reported? 
• Swine flu response: what is the risk of dying from swine flu? how should one evaluate the level of risk? How do vaccines work? what is "herd immunity"? what is the "precautionary principle"? why do responses in different countries differ?
• Autism clusters: when do statisticians tell apart "something small" from "nothing at all"? how can one draw confident conclusions with only a few samples? do they know the cause of such clusters?

Numbers Rule Your World explains, using stories (no formulas), how we can use statistical thinking to think smartly about these issues and more.

In the acknowledgement, I wrote:

throughout this project, I was inspired by fans of my Junk Charts blog 

So you are the first to learn about the book. 

In the next few days, I will offer perspectives on how the book came into being, why I wrote it, and who I wrote it for.  I hope you'll be as excited about the book as I am!