Big Data, Plainly Spoken (aka Numbers Rule Your World)

The mirage of large numbers

The first thing one (should) learn about statistics is "all that data is not information." That's the very first thing I tell my class each semester. This message is doubly resonant in this age of "Big Data".

***

I was reading a post on Dell at Felix Salmon's blog, a post written by Ryan McCarthy or Ben Walsh. It cited BusinessWeek's Roben Farzad: "When it comes to putting a price on Dell, [he] points out one wild card: the company’s 3,449 patents (with another 1,660 pending)."

This is not the first time I read about the potential value of the massive store of patents owned by some tech company. Large numbers are used to reinforce the point. This type of argument preys on people's tendency to extrapolate linearly. (The original BusinessWeek piece goes into describing how a consultancy tried to evaluate Dell's patents. Amusingly, the article contains no dollar amounts related to Dell, only the large number of patents.)

The value of patents might be modeled as an exponential distribution with a very short half life. That is to say, the vast majority of patents will have value zero but there is a long tail consisting of things like Google's algorithm, the CAPTCHA idea, the coffee cup sleeve and so on, which have generated huge profits for the inventors.

So, even if you own only one patent, the value could dwarf the total of 3,500 patents. If we look back at Dell's portfolio ten years from now, it is guaranteed that only a few of the 3,500 would provide a windfall while all others would prove to be duds. It's about the quality of the patents, and not just the quantity.

***

I'm sure you have come across these moments when you want to scream: your large numbers don't impress me. Feel free to share your stories.

Getting inside my head

Some interviews with me or snippets of such have surfaced recently. Here is a list:

Kate Meersschaert interviewed me for New Learning Times (link; registration required). I talked about my teaching philosophy, and why I write books.

Jay Ulfelder, a political scientist who keeps an interesting blog, recommends Numbers Rule Your World, and a few other books for political scientists (link).

If you haven't heard already, 2013 is the International Year of Statistics. I was one of the talking heads here.

Here, I talk about the history of Junk Charts, and the new paradigm of interactively building graphs, as opposed to the template paradigm popularized by Excel.

 

Free magazine, negative heights, and statistical methods

There is a magazine that you should know about. It’s called Significance (link). With that name, it's hardly surprising that it has statistical content. Mathematical equations are kept to a minimum, and typically set off in boxes. I highly recommend it. The reason why I’m writing about it today is that for 2013, you can read it for free via their iPhone or Android app (information here). The magazine is normally for subscribers only, but this offer is sponsored by the International Year of Statistics in 2013.

Significance used to be published solely by the Royal Statistical Society, but recently it’s a joint effort between them and the American Statistical Association.

***

In the most recent issue, there is an intriguing article on the distribution of heights of adults. Of course, we know it's a Bell Curve. That's what statistics teachers tell us anyway. These authors ask if the distribution is indeed normal, then where are the people with negative heights? I am one of those statistics teachers so I was happy enough to spend 10 minutes or so to indulge the authors in this flight of fancy. (The question has no practical value but it's interesting nevertheless in the context of modeling.)

I won't repeat the calculations from that article. The general lesson is that statistical—in this case, probability—models are not theorems, and they don’t usually work for every conceivable case. In fact, the normal distribution model works exceedingly well for all but an inconceivably small number of cases, so it is an exceedingly useful model for all but a few cases.

This leads me to one of my pet peeves.  I often encounter criticism of statistical models of the form “this model assumes X, and X is violated by this counter-example, and thus the model is falsified.” In this example, someone might say: this model assumes the existence of negative heights, which is both absurd and obviously wrong, therefore the model is falsified. The bottom line is every model contains assumptions that are not universal, and this example shows that such models can still be useful.

Being locked up in a room with nothing but a bed and food

Imagine you are locked up in a hospital room with a bed and plenty of food. What do you think you'd be doing when you aren't sleeping?

***

Well, some researchers (link) discovered that you'd be eating the food. What a surprise.

The New York Times then saw it fit to report this result as "Lost Sleep Can Lead to Weight Gain", which isn't as bad as the lead given on the front page of NYTimes.com "To Put on Pounds, Just Sleep Less".

Of course, the researchers never proved that sleep deprivation is a cause of obesity. In the original press release, the lead scientist explicitly noted: "Just getting less sleep, by itself, is not going to lead to weight gain. But when people get insufficient sleep, it leads them to eat more than they actually need."

It's a scandal that the prestigious paper would write a headline that directly contradicts what the researcher said.

***

I also have a problem with the link between insufficient sleep and eating more. Here is their description of the experiment:

For the study, researchers monitored 16 young, lean, healthy adults who lived for about two weeks at the University of Colorado Hospital, which is equipped with a “sleep suite” for controlling sleep opportunities — by providing a quiet environment and by regulating when the lights are on and off — and a sealed room that allows researchers to measure how much energy participants are using based on the amount of oxygen they breathe in and the amount of carbon dioxide they breathe out.

All participants spent the first three days with the opportunity to sleep nine hours a night and eating meals that were controlled to give participants only the calories they needed to maintain their weight in order to establish baseline measurements. But after the first few days, the participants were split into two groups: one that spent five days with only five hours to sleep in and one that spent five days with nine hours of sleep opportunity. In both groups, participants were offered larger meals and had access to snack options throughout the day ranging from fruit and yogurt to ice cream and potato chips. After the five-day period, the groups switched.

The participants who were forced to sleep less spent more time eating.

***

The NYT column actually described another study which I think holds more promise. It tries to connect sleep deprivation with fat cell biology. The sample size is really tiny, and a lot more work is needed.

In search of the honest credit repair shop

FTC made the headlines recently complaining about inaccurate consumer credit reports. The Wall Street Journal (link) has a typical report on this research. Here's their summary:

In the FTC study, 262 of the 1,001 people who reviewed their credit reports spotted at least one potential "material" mistake, such as a credit-card account that wasn't theirs or a late payment that they didn't believe was late.

That sounds like a worrisome 25% error rate.

***

In Chapter 2 of Numbers Rule Your World, I explain why focusing on reporting inaccuracies is counterproductive, particularly and ironically for those who have good credit. The reporters clearly haven't read the 370-page report the FTC has put on line.

If they did, they may have noticed the chart shown on the right (in the main document). The chart shows the proportion of consumers in their sample who have all the disputes denied (green), who have all the disputes deniedaccepted (blue), and those who have some of their disputes accepted (yellow).

One feature of this chart that can easily get lost is the uniformly short columns corresponding to those with high credit scores and the much taller columns for those who bad credit scores.

Those with low scores are much more likely to find mistakes in their reports than those with higher scores.

If the reporters flipped to Appendix D, they would have seen the following table:

Forty-five percent of those with credit scores below 590 claimed at least one potential mistake while only five percent of those with 790 or higher credit scores did.

What kinds of mistakes are being reported? Refer back to the WSJ quote above. A late payment that may not have happened will be disputed - what about a late payment that the credit bureau didn't know about? Would the consumer volunteer that information? Similarly, if the account not belonging to the consumer has great credit, thus pulling up one's credit score, would the consumer ask the bureau to take it out?

If the FTC encourages consumers to dispute credit reports, the end result is that the credit scores become less accurate, not more accurate. The dispute process introduces a bias so that the reports tend to contain mistakes that cause overestimation but not those that cause underestimation.

Worse still, this policy penalizes the good people, as it allows less creditworthy consumers to unfairly inflate their scores. In general, it hurts the predictive power of credit-scoring algorithms.

***

It's ok for the credit reports to have errors so long as they are unpredictable. This means that positive errors are as likely as negative errors.The following table is overwhelming proof that the credit dispute process introduces bias:

Pretty much all disputes are resolved to the favor of the consumer. I don't think the credit reports only contain errors that penalizes (low-scoring) consumers.

***

Final question: we know there are credit repair agencies who can help you correct your credit scores. Have you seen any of these agencies advertise as the honest agency who will help lower your credit score to make it more accurate?

 

Statistically improbable words 5

What I read this week:

Traditional Coast Guard boot camp when I was a grunt in the 1960s was twelve weeks, versus the Army's nine-week stint. This was because people who came into the Guard were usually in suboptimal physical condition and because we had more instructional classes, such as semaphore and maritime law.

(From Lee Gutkind's book on creative non-fiction, You Can't Make This Stuff Up)

If some study found that Coast Guard graduates are in worse physical shape than Army graduates, what is the cause?

Man versus man-plus-machine

As you read Steve Lohr's article "Algorithms Get a Human Hand in Steering Web" (link), recall what I wrote when discussing David Brooks's piece on Big Data (link):

The biggest issue with Brooks's column is the incessant use of the flawed man versus machine dichotomy. He warns: "It's foolish to swap the amazing machine in your skull for the crude machine on your desk." The machine he has in his mind is the science-fictional, self-sufficient, intelligent computer, as opposed to the algorithmic, dumb-and-dumber computer as it exists today and for the last many decades. A more appropriate analogy of today's computer (and of the foreseeable future) is a machine that the human brain creates to automate mechanical, repetitious tasks at scale. This machine cannot function without human piloting so it's man versus man-plus-machine, not man versus machine.

At the Purdue talk, the students asked if I'm a disbeliever in machine learning. I believe in the mixed approach which leverage human creativity together with machine efficiency.

Talk at Purdue

I will be giving a talk at Purdue University on Wednesday afternoon, 4:30 pm. The details can be found here. I am hosted by the Math Department in Lawson, Room 1142.

The general theme is the real-world challenges of "Big Data". After the talk, I'll discuss the business analytics career.

Please drop by if you are in the area.

 

Revisiting Grouponomics

Two items of news came out this week regarding Groupon, the daily-deals company. I first wrote about them here in response to Felix Salmon's thoughtful piece.

Restaurants in Westchester are unhappy about Groupon-type deals "sucking $280 million from independent restaurants". (link) Wasn't it not long ago that we were supposed to believe that Groupon is "free advertising" for restaurant owners, and brings a lot of new customers to fill the empty seats? Time to re-read my earlier post to see why this scheme doesn't work for most.

Meanwhile, Groupon stock is in the cellar, and its controversial CEO has been kicked out. (link) He did make himself and a number of other people extremely rich by pushing the IPO out with not even two years of operational history. Well-timed.