Big Data, Plainly Spoken (aka Numbers Rule Your World)

A short statistics course

For years, I have wanted to see a statistics course that is not a math class. So I made one myself. The title of the course is "How to do statistics without really doing statistics?". It's on a new online learning platform called Three Nights and Done. There are three hours worth of materials divided into three or four chunks each hour.

Here is the link.

I'd love to hear some feedback. Please leave a comment here.

 

PS. Errata. I have requested these be fixed. But until that time, please note:

Night 2 - Part 3: at the start of the discussion of the Monty Hall problem (roughly 6:56), there is a duplicated segment which you will hear again later in the same recording. This can be very confusing to those who don't know what the Monty Hall problem is. You should skip from 6:56 to 8:53 (redundant), then the flow is maintained.

Night 3 - Part 1: the introductory comments to Night 3 went missing, and will be added back. Here is the transcript: Class 3 is based on my new book Numbersense. The premise of the book is that the world of Big Data is very confusing, and filled with claims and counterclaims. A key skill is how to analyze and interpret other people's data analyses. The book gets at different aspects of Numbersense. In Class 3, I focus on two fundamental ones: knowing how data is collected, and knowing how data is processed. These two sound simple; they are anything but. Data collection and processing is a very messy world, and very arbitrary as well. However, these are two really important things to know. If you come across a study that does not disclose details of how data was collected and processed, you should be highly skeptical about the results.

Night 3 - Part 3: start listening at 12:53. the segment before 12:53 is an exact copy of the end of Part 2.

 

Reading Predictive Analytics

Predictive Analytics by Eric Siegel (link) was published earlier this year. Siegel is a consultant and organizer of a series of popular industry conferences, which I attend with some regularity. I  recommend this book for readers who want to understand the current state of “data science” at a deeper level than the New York Times’s but still nonmathematical. If you want to measure against my own writing, then Siegel spends more time addressing “how” than I typically do on this blog. He also has a fondness for lists, tables, quotations, pictures, and turns of phrases—I did mention lists, and lists within sentences.

Reading this book is like worming through the crowd at a conference center in New York, or Boston, or San Francisco, where Siegel’s meetings are usually held. Siegel is like your Kevin Bacon, your link to practitioners of the art/science of data-driven business decision-making. (I just gave the definition, rather than the name, of the field, which has as many names as cheongsams worn by Maggie Cheung in In the Mood for Love. Siegel selected “predictive analytics,” while you may also see “data science,” “data mining,” “machine learning,” “statistical learning,” “knowledge discovery,” “statistical modeling,” “business analytics,” etc.)

***

Predictive Analytics paints an accurate picture of the applications and discourse circa 2000s. Chapter 2 describes a model used by the retailer Target to find potential customers who are pregnant women—this application was later picked up by Charles Duhigg in The Power of Habit (link), and a New York Times Magazine article, which I examined in a previous post. (Disclosure: I have further comments on pregnancy targeting in my forthcoming book.) Also in Chapter 2 is a description of how Hewlett Packard analyzes data on its employees, a nice companion piece to the recent New York Times profile of Google’s SVP of “people operations.” See my post here. The third example in Chapter 2 concerns local police using data to find criminals.

Social media analytics, possibly the hippest corner of the industry, gets star billing in Chapter 3. Two researchers summarized Livejournal blog posts into an “Anxiety Index” which they claimed predicted S&P 500. Chapter 4 contains an extensive description of a popular technique known as “decision trees” applied to financial risk management. For those who read Chapter 2 of Numbers Rule Your World, this section provides more technical details on risk scoring. Some keywords to look out for are overfitting (called overlearning), using test datasets to evaluate accuracy, and Occam’s razor.

Chapter 5 covers ensemble models, a relatively new technique with broad applications. What this means is instead of the traditional route of developing one “best” predictor, conduct a poll of a set of predictors. Sort of a wisdom of crowds approach. The winning team in the Netflix Prize—in which teams competed to improve the “accuracy” of Netflix queue recommendations—used an ensemble.

Chapter 7 introduces net lift models, which is an unresolved but important area of business analytics. Take an example of Time Warner Cable wanting to send special offers to “vulnerable” customers hoping to retain them. Traditional predictive models find those customers who are most likely to cancel their service. The trouble is that special offers are very expensive, and some of those customers would not require this incentive in order to renew. A “net lift” model is more accurate in only targeting those customers who are likely to cancel plus likely to renew only if Time Warner makes a special offer. Technically, the latter problem is much harder to solve.

***

Needless to say, the price of the book is a fraction of the conference fee. While the overall tone is optimistic, Siegel does not shy away from discussing the limitations of data analytics. This I find to be a virtue, a relief from the relentless hype that has enveloped this field of work. I’d like to end this review with a quote (p. 201):

Commanding a computer to learn is like teaching a blindfolded monkey to design a fashion diva’s gown. The computer knows nothing. It has no notion of the meaning behind the data, the concept of what a mortgage, salary or even a house is. The numbers are just numbers. Even clues like “$” and “%” don’t mean anything to the machine. It’s a blind, mindless automation stuck in a box during its first day on the job.

Management as popularity contest

One of the unlikely areas in which data science has played a role is employee management. New York Times has an informative interview with Laszlo Bock, an SVP at Google. (link).

Here's Bock on the famous brainteasers used in Google interviews:

On the hiring side, we found that brainteasers are a complete waste of time. How many golf balls can you fit into an airplane? How many gas stations in Manhattan? A complete waste of time. They don’t predict anything. They serve primarily to make the interviewer feel smart.

Many years ago, I joined a strategy consulting company after graduating from college. Consulting companies are also famous for using brainteasers in interviews. Microsoft, in those years, was the Big Daddy in tech, and they too were  known for using brainteasers.

I always hated brainteasers and I swore never to use them. In addition to the reason given by Bock, the obvious problem is that the actual skills needed for most jobs do not correlate with solving brainteasers. Also, using brainteasers turns the interviews into a game of chance; you're differentiating between people who have heard of the question before and people who haven't.

And finally, some brainteasers are just plain devious. There was one that irked me and I can't recall the details now. The question was designed in such a way that if one systematically tackles the question, setting up equations with x, y, z, etc., one would miss the answer. The data were arranged in such a way that one would need to do tedious brute-force calculations to find the solution. If I am hiring a data analyst, I'd prefer the one who set up the equations, who demonstrate the ability to systematically break down a problem.

***

Another item of note from that interview is the following:

Twice a year, anybody who has a manager is surveyed on the manager’s qualities. We call it an upward feedback survey...  Over the last three years, we’ve significantly improved the quality of people management at Google, measured by how happy people are with their managers.

This use of analytics is quite different from the above. This use is at the individual level, while the brainteasers analysis operate at the aggregate, corporate level. Also, the former looks backwards, and the latter looks forwards.

This upward feedback mechanism turns management into a popularity contest. My own experience is that many of the most productive managers are not well liked. This is similar to having students rate their professors or courses. Easy graders and easy courses often receive high grades, and boast happy students.

I'd hope managers at Google are evaluated based on other criteria also.

 

 

 

Surveys on sensitive topics

Andrew Sullivan (link) has a few questions about a new Pew survey focusing on the LGBT subpopulation. He wonders, for instance, about the high proportion of self-identified bisexuals in this poll.

What interests a statistician here is the fact that the poll deals with sensitive matters, which typically present a challenge in terms of survey response, and nonresponse bias. The main accommodation that the pollster (GfK) made here was to conduct an online survey because people apparently let their guards down in that medium.

Andrew also wondered about the fact that "only 28% of bisexuals have come out, compared to 77% of gay men, and 71% of lesbians," among the respondents. I find the concept that a closeted person would identify himself/herself to a pollster as LGBT to be inconsonant. But then, the pollster appeared to re-confirmed eligibility carefully. (After the recent revelation about online privacy, one thinks this might change.)

In any case, I am glad to see a very detailed description of the methedology. See here.

Willing the data to fit your model

It strikes me that in medicine, we are stuck with simplistic models - models that use one variable only, and are linear in the response. In short, we are told X results in Y, and the more X, the more Y. Real life often does not cooperate, but many people in medical research hold on to their models for dear life.

Exhibit 1 is the disappearing of unhelpful data used to reject the non-linear relationship between BMI and mortality, which I discussed here. There are many myths associated with the obesity epidemic. It is a surprise for me to learn that how inefficient physical exercise is for weight loss.

Exhibit 2 is the case of vitamin overload, as described in this NYTimes column. Here are the juicy bits:

In December 1972, concerned that people were consuming larger and larger quantities of vitamins, the F.D.A. announced a plan to regulate vitamin supplements containing more than 150 percent of the recommended daily allowance. Vitamin makers would now have to prove that these “megavitamins” were safe before selling them. Not surprisingly, the vitamin industry saw this as a threat...

Industry executives recruited William Proxmire, a Democratic senator from Wisconsin, to introduce a bill preventing the F.D.A. from regulating megavitamins.

A little more than a month later, Mr. Proxmire’s bill passed by a vote of 81 to 10. In 1976, it became law. Decades later, Peter Barton Hutt, chief counsel to the F.D.A., wrote that “it was the most humiliating defeat” in the agency’s history.

The studies cited in the article show that too much of a good thing kills.

Exhibit 3 is the demonization of salt. It's completely ingrained in our brains that salt is bad for us. However, the evidence is decidedly mixed. New York Times has a summary recently (link). Years ago, I remember reading the late David Freedman's article coming to the same conclusion. (D.A. Freedman and D.B. Petitti. “Salt and blood pressure: Conventional wisdom reconsidered.” 2001.)

A key sentence in the NYT's article is this:

Until about 2006, almost all studies on salt and health outcomes relied on the well-known fact that blood pressure can drop slightly when people eat less salt. From that, and from other studies linking blood pressure to risks of heart attacks and strokes, researchers created models showing how many lives could be saved if people ate less salt.

This is also typical of medical studies, the assumption of transitivity between studies. Anyone trained in statistics would have left transitivity at the door. When each number is not precisely estimated but has an error bar around it, transitivity is not automatic. Notice that the data did not prove the health benefit of less salt; it's the models that assume such benefits.

 

 

 

Cutting data won't make cutting weight easier

In my new book, I have a chapter on interpreting the statistics of obesity. Andrew Sullivan (link) recently pointed to a Nature article discussing an aspect of the controversy around these numbers.

The bone of contention is the shape of the mortality curve. It has been thought that the curve is monotonic increasing, meaning that the higher your BMI, the higher the mortality rate. But survey data in the U.S. now show that the curve is probably U-shaped: mortality rates are high for both obese and thin people. Overweight (less than obese) people paradoxically seemed to live longer than those with "normal" weight. This last observation has driven some people nuts.

The article focused on two Harvard researchers who organized a conference specifically to attack a CDC paper demonstrating the U-shaped curve. This is the crux of their argument:

When the researchers excluded women who had ever smoked and those who died during the first four years of the study (reasoning that these women may have had disease-related weight loss), they found a direct linear relationship between BMI and death, with the lowest mortality at BMIs below 19.

Excluding portions of a sample from analysis is a dangerous game, and should be heavily discouraged. It's one thing to adjust the data; it's another thing to remove data completely. Notice that what was removed weren't outliers, that is, data that might be incorrect and so extreme as to dominate the outcomes. They removed data specifically to conform to their model of the world.

First, they removed smokers because "smokers tend to be leaner and die earlier than non-smokers". This sounded like smokers who die earlier are on the thin side of the curve; removing them has th effect of straightening the curve.

The second cut is even more egregious. How can there be any justification for removing people who died during the first four years when the study's primary metric is death rate? They claimed reverse causality.

The most important reason why you should never drop large chunks of data in a systematic way is that your conclusions are now limited to the group that hasn't been dropped. Since there are no smokers in your sample, you cannot make a statement that applies to the general population. And yet, these researchers seem to have done so.

***

Later on in the article, the journalist repeats the nonsense about how using BMI is a problem. I have previously written about this topic here.

On a related note, a visiting professor at NYU has been making the news, having made insulting comments about "fat PhD applicants". Somehow, the field of evolutionary psychology has attracted many crazies.