Big Data, Plainly Spoken (aka Numbers Rule Your World)

Data mining in the news

The Wall Street Journal has been doing an excellent job describing the various data mining activities undertaken by a variety of companies out there in a series of articles. Readers can judge for themselves whether these activities are a breach of privacy, or a reasonable trade-off that provides more benefits than harms.

The latest article focuses on an experiment in which a life insurer attempts to use consumer marketing data instead of direct medical test results to assess the risk of applicants. This type of model is essentially the same as the credit-scoring models described in Chapter 2 of Numbers Rule Your World; and in Chapter 3, I discussed the basic problem facing insurers of separating the high-risk and the low-risk subgroups of customers.

One particular sentence in the article requires some comments:

The goal of Aviva's test: With 60,000 actual insurance applicants, figure out how to use the marketing databases and other information to reach the same underwriting conclusions that Aviva reached using traditional methods such as blood work. The 60,000 people were applicants Aviva had already judged.

The stated objective is by no means the only natural objective for such a pilot test. The end goal of any predictive model is to accept a subgroup of applicants who on average will generate enough revenues to cover the payouts plus the insurer's desired profit margin. Two types of models are being compared here, one using "traditional" medical test data, the other using marketing data (collected both online and offline). Instead of directly testing whether the marketing-data-driven model can predict accurately whether the approved applicants are profitable, the stated goal only compares how well the marketing-data-driven model mimics the predictions of the medical-data model.

There is an underlying assumption that the medical-data model is accurate enough. In other words, the objective of the insurer is purely to reduce the cost of predicting because the cost of buying marketing data is much lower than the cost of conducting medical tests. This test will not improve the accuracy of the prediction while it is possible that the marketing-data-driven model may in fact reduce the accuracy. The insurer might very well accept a less accurate model because according to the Journal, the cost differential is very large: $125 per head (I think this is the right number; the article is a bit confusing on this point) which might cover the incremental cost of lesser accuracy.

***

The purported benefit to potential customers is less hassle, and lower cost (assuming that the insurer in fact passes on the cost savings). However, the insurers are understandably cautious about this because of something called adverse selection -- the type of customers most likely to benefit from no medical tests is the subgroup of customers who have private knowledge about their own longevity, and so removing the requirement for testing will likely attract this type of applicants disproportionately.

This explains why the insurers say:

The process would simply speed up applications from people who look like good risks. Other people would go through the traditional assessment process.

***

In Chapter 2, I also described various credit-repair scams, including piggybacking on other people's good credit history. A fundamental flaw of using online data, especially using social-media data, is the ease with which any customer can create fake personas. If you know that insurers are tracking you on Facebook, you can easily paint yourself as their ideal customer, making yourself look like a "good risk" when you aren't. I'd love to hear what the data miners have to say about this.

Tom Friedman's Law of Statistics refutes Chapter 1

Reading this completely incoherent, frustratingly unfathomable column by Tom Friedman, the star columnist at the New York Times, I wanted to pull my hair out. From first word to last, he confuses averages and extremes -- the stuff in my Chapter 1, which is the very first concept in statistics.

Friedman has discovered a new law of statistics: to understand averages, look at extremes.

Starting from the end. He claims that the average American should learn from the average immigrant parents who value education more. His evidence: half of the 32 Rhodes scholar winners have foreign-sounding names. Where to begin?

1. Rhodes scholars are outliers. The existence of outliers does not imply that the median immigrant kid has better educational outcomes. It appears that Friedman has never heard of the "achievement gap", that is, the average test score of most minority groups are far below that of white students. Any teacher can tell you trying to raise the average performance in a class and trying to nurture the top student requires completely different approaches, probably mutually exclusive. Setting policies for everyone based on outliers is not very smart.

2. He equates Rhodes scholars with "America's top college grads". Maybe recognized by some as such but certainly not by all.

3. If he knows any immigrant kids, he'll learn that it makes a big difference whether you are first generation or latter generations, and looking at names does not give you a hint of this key distinction.

Friedman starts the piece trapped in an impossible unpainted corner. Using the story of a girl who texts 27,000 times a month on average, he claims that today's unemployment problem is partly a function of kids not going to college, and they don't go to college because they waste their time texting. Where to begin?

4. He goes on to say, and I quote: "I don’t want to pick on Miller [the girl]. I highlight her words only because they’re integral to a much larger point..." Miller is clearly an outlier (see this recent Nielsen data for evidence). How an extreme case can be "integral" to making a "much larger" point, presumably about the average American is beyond me.  We are not even complaining about a sample size of 1 but a sample size of 1 extreme case. Beyond frustrating.

5. The unemployment report that tells us only half of B.A. graduates found jobs requiring a college degree apparently has not yet reached his desk, after over two years. (See for instance this blog post by two seniors.) Among 18-24 year olds, the unemployment rate is 25% (!!!), not counting the underemployed and "marginally attached" and "discouraged".

6. Texting is not the only distraction, and is probably not the most important reason why some kids are not going to college, or fail to graduate.

7. In Friedman's world view (shared and pushed by a bunch of economists who call this "structural unemployment"), it is not a problem that we have eliminated most jobs suitable for people with high school diplomas. When these people can't find jobs, it is their own fault, and their parents' fault. I am not sure how his algebra works: he mentions the case of the micro-chip replacing his receptionist; you know what, one micro-chip will replace an entire team of receptionists - that's the nature of technology replacing labor; and if, as Friedman seems to agree, the "low-end" (read: labor intensive) jobs producing the technology will be exported due to globalization, I am not understanding how the numbers add up - don't we end up with fewer jobs? There is a shift to higher-end jobs but aren't there fewer total jobs?

Now, let's venture into the middle of the column. Friedman claims that Americans have to "step up their game" because of competition from foreign students. How does he know that foreign students are providing tough competition? Look at "two of our most elite colleges" (those in his world are Harvard and Yale); look at foreigners applying to "Ivy League schools". Say what? He thinks these extreme outliers will tell us something about "every school". So there it is again, Tom's Friedman Law of Statistics: to understand the average, look at the outliers.

What Friedman's column proves is that having gone to college and grad school (Brandeis and Oxford) is no guarantee that you can make a cogent argument based on evidence.

 

Two wrongs by Stanley Fish in one sentence

This one sentence sums up the source of trouble in Stanley Fish's widely-panned blog post:

My [Fish's] criticism was a proto version of theirs [Archibald and Feldman whose book Fish was pitching], absent the massive data and the economic sophistication.

Here, he admits two reasons for trusting Archibald and Feldman: their position agrees with his own point of view; and economists wielding data.

Under reason one, it wouldn't matter if the analysis makes sense or not; his support is based on their agreeable conclusion. It is shocking that he so blatantly and openly says this. Making the numbers fit the story, rather than making the story fit the numbers, is a cardinal sin.

Reason two reveals his laziness; if he had called up any economist or statistician, chances are he or she will have major problems with Archibald and Feldman's analysis. The quantity of data has nothing to do with the quality of the analysis. That's the first thing your statistics teacher should be telling students. Higher "sophistication" (or "complexity") of analysis also does not guarantee better quality -- but it does guarantee slower understanding. That is not the first lesson but still an important lesson in statistics.

Neat reads 5

Digital keys for unlocking the humanities' riches, New York Times, Nov 19, 2010

This story claims the "next big idea in language, history and the arts" is "data". It's an "alliance of geeks and poets". The NEH and NSF band together to create the "Digging into Data Challenge". It's an interesting concept but way overhyped here. Where there exist data, one can do some innovative work; but what about the time before computers?

Statistics and storytelling: the grudge match, Nielman Storyboard (Harvard), Nov 19, 2010

They picked up my "story time" complaint, and wrote a thoughtful article about the uses and limits of stories. To clarify my own position, I am not against narratives in general. Narratives are necessary to explain statistical correlations; I'm against narratives not substantiated by evidence. Narratives can and should be subject to reality checks before they get published.

There is no college cost crisis, New York Times, Nov 15, 2010

Professor Stanley Fish really should stick to subjects he know about, such as what books first-year college students should be taught. He is completely out of his depth here, putting his weight behind a new book, in which the authors claim that college is becoming "more affordable". There is no indication he understands any of the numbers; he approvingly cites "massive data", "economic sophistication", and "buttressed by statistical tables" as if having lots of data is sufficient as an argument. Here's Felix Salmon mincing Fish's blog post. Here's Brad Delong smelling stinking fish.

 

 

Statistically improbable words 4

Historian Anthony Grafton, speaking about the use of data analysis in the humanities:

 

I'm a believer in quantification. But I don't believe quantification can do everything. So much of humanistic scholarship is about interpretation.

 

Grafton's words could have come out of my mouth, or that of any statistician! It applies to any data analysis.

Discussing Numbers Rule Your World

A big thank you for all of you who showed up to meet me at the Princeton Library. I was very happy to see so many people interested in statistics. There were many terrific questions, and here are the first five topics that come to mind as I write this post:

• Are there cultural differences in the level of statistical thinking? Are people in certain countries more likely to do better at statistical thinking than others? I'd surely love to see some studies in this area. I suspect that there are differences. Climate change is one area, for example, where clearly the level of understanding differs by country.
• Have our politicians learn to lie less using statistics over time? Unfortunately, not. I think, or hope, the people can pick up more statistical thinking, and see through lies better. But politicians choosing to not lie, I don't think so.
• How much are quants/data people to blame for the recession? Not very much. Statistics provide tools for making decisions but they do not provide decisions. One of the key messages of Numbers Rule Your World is that people make decisions based on incentives and compromise; the data serve as a guide only. One can't blame credit scoring systems for getting it wrong when banks accept "liar loans", "no doc" loans, "ninja" loans (no income, no job, no assets).
• Explain the Monty Hall problem. (I mentioned this as the kind of toy problem that I do not use in my book.) Here's my quickest explanation: your first guess is either right or wrong; if the first guess is right, switching will make you a loser for sure; if the first guess is wrong, switching will make you a winner for sure (verify this); since there is a 1/3 chance your first guess is right and 2/3 chance your first guess is wrong, switching gives you a better chance of winning.
• Are there online resources that helps us understand economic numbers? I highly recommend Dean Baker's blog. Almost every day, he makes me laugh out loud.

Checklist for reporters re: full body scan and enhanced patdown

The author of this Christian Science Monitor article on the "full body scan" and "enhanced patdown" procedures makes us click through 8 pages of ads but manages to miss all of the important questions for which a reader would like to get answers.

Not picking on CSMonitor, I have yet to see a reporter from any mainstream media outlet bothering to ask any of these questions:

• How effective are these TSA screening procedures at preventing terrorists? This must be considered relative to the traditional metal detector and patdown.
• What proportion of these "scans" and "patdowns" results in "positive" findings?
• What proportion of these "positive" findings are "false positives"?
• What's the likelihood that a real terrorist will get a "positive" finding? Or, how many terrorists will pass this scan/patdown for every terrorist caught by it?
• What's the likelihood that a non-terrorist will get a "positive" finding? Or, how many innocent passengers will fail this scan/patdown for every terrorist caught by it?
• What's the cost of each scan/patdown and what is the total cost of all the scans/patdowns needed to catch one terrorist?

 

Update (Nov 22): In the latest article, the New York Times wasted more space re-printing the same old story with no new information. In an article auspiciously titled "Administration to seek balance in airport screening", one is left to wonder if the NYT's dictionary has the word "balance" in it. Their dictionary also does not appear to contain the word "irony". The reporter alleges that the enemy is trying to "impose huge costs on the United States" by a strategy of "a thousand cuts", and then tell us that the U.S. will spend huge amounts of money and resources to adopt these scanners for which no data is available on their effectiveness. Since I'm assuming that our journalists have the minimum competence to ask about effectiveness, the awkward silence is to be interpreted as no data.

Updates

I'm late in linking to this positive review of the book from Robert Kosara, who writes the EagerEyes blog. I love this review because it's one that truly gets the concept of the book. Kosara's blog is a great read too.

***

On Wednesday, I'll be at the Princeton Library at 7:30 pm to meet readers. Note that this is the township library, not the university library. Hope to see some of you there.

Book review: Proofiness by Charles Seife

Proofiness is what Numbers Rule Your World isn't, and because of this, it serves as a perfect foil. Seife documents a lengthy list of examples in which numbers are manipulated, bent out of shape, invented, thrown by the wayside by journalists, politicians, Supreme Court justices, the Pentagon, scientists, non-scientists, consciously or not, maliciously or not.

Seife is a science journalist for the New York Times, and teaches journalism at NYU. Like similar books in this genre, Seife writes well and in an accessible manner; unlike those other books, there aren't glaring errors (I will put up a few more posts doing a close reading a la SuperFreakonomics later.)

There are really two books in one: the first three chapters are, in substance, comparable to Huff's How to Lie with Statistics, one of the truly invaluable pop-statistics books; in the remaining chapters, Seife aims his arrow at different groups of people who engage in what, in the subtitle of the book, he calls "the dark arts of mathematical deception", namely, pollsters, politicians, justices and lawyers, the military and other propagandists. Even those familiar with this subject will learn a few new examples.

***

Since I will be writing about some of the details in future posts, I'll restrict my comments here to some of Seife's big ideas.

He's very concerned about politicians and justices destroying our democracy through the rejection of proper statistical methodologies, such as the rejection of sampling in the Census. It is a valid point. I think he is too soft on his fellow journalists because they are the messengers and enablers. To Seife's credit, he does mention the problem of "balanced coverage" when one side is clearly 90% right; and he does stress the importance of fact-checking. He talks about how journalists laughed off the Pentagon's propanganda during the Vietnam War (Chapter 8) and looked for independent evidence in the war zone but dodges the obvious questions: why the journalists relayed all of the false stories that came out prior to the current Iraq War? and whether the practice of "embedded reporting" hampers the ability to find field evidence?

Seife does an admirable job telling us to expect uncertainty, particularly why it is always there. He makes an excellent point about polls being different from election counts: they both suffer from uncertainty but the source is different, the former from random sampling, the latter from measurement error. The very important message being: a count is a type of estimate. There is no such thing as a perfect count, especially when it comes to counting millions of votes. So, I am surprised that Seife goes off message in a section of Chapter 8 when he posits the availability of "truths". He says this (p.226):

Journalists are supposed to... admit nothing but the information that we get from our own senses and the verifiable truths that statistics seem to represent. This sort of information is the raw fuel of journalism...without hard facts to pin our words to, we are powerless to express ourselves.

Even if one ignores the nostalgic reverence of old journalism (for journalism in the Internet age is anything but objective and careful with facts), it is surprising to hear Seife talking about uncovering "truths" through diligence after reading so much about the impossibility of perfect measurement.

I'd like to see a sharper distinction made between malice and incompetence. Seife lumps the two together. People who are fooled by the data (say, NASA engineers) are criticized next to people who willfully deceive with numbers. (See my related post here.)

***

Personally, I like the second part of Proofiness better as it is more focused, and not much done before. The first part is very readable (verbose at times), and still interesting for those who haven't read similar works before. As I said, it's a good foil for my book: Seife warns you how numbers might deceive you while I tell you how numbers can benefit your lives.

 

 

Meaningless data make words meaningless

The overly excitable people come out of the woodwork again, following this morning's information-less report on weekly jobless claims. Here are two of the breathless commentary:

Business Insider: Weekly jobless claims of 435K are way better than expectations

CNN Money: Initial jobless claims tumble to 4-month low

By "way better", BI is talking about a difference between 435K and 450K. By "tumbling", CNN is talking about a drop of 24K or 5%.

***
Here is why the announcement this morning contains no useful "information," as in anything that allows us to conclude that the job market is any better.

The two journalists both note that the 4-week moving average (average of past 4 weekly figures) dropped by 10K. This means on average over the past 4 weeks, job losses decreased by 10K per week. So the current estimate (almost always revised with more losses) of 24K last week is in the same ballpark as the previous 4 weeks. And note that the job loss number has to decrease at this rate for the next few weeks in order for the moving average to show a decrease at the 24K level. So there is no news here at all.

The graph below clearly shows that nothing remarkable is happening in the context of the recent history of this data series; we are bouncing around the plateau of about 450K losses per week, which as economists keep telling us, is not even sufficient to deal with the influx of working-age people.

(Image from Calculated Risk).

In particular, focusing on the tail of this data series, we see that the 24K change is not "way better," nor is anything "tumbling". The reporters rob these words of their meaning when they are used inappropriately.

***

For those who want to explore more, the government's press release is here. You will find that the entire drop in job losses comes from their "seasonal adjustment". The raw numbers show a slight uptick in job losses from the previous week of 30K. The adjustment was about 54K fewer jobs lost. I am not saying seasonal adjustment is a bad thing but it is important for us to differentiate between the "run rate" of job losses (which is post-adjustment) and the reality of job losses (the raw count).