Big Data, Plainly Spoken (aka Numbers Rule Your World)

Paulos on statisticians

John Allen Paulos has a new piece at NYTimes.com called "Stories vs. Statistics". 

It's thought-provoking, somewhat discursive, and I don't agree with everything there.

He argues that we subconsciously use statistical concepts in our everyday speech:

With regard to informal statistics we’re a bit like Moliere’s character who was shocked to find that he’d been speaking prose his whole life.

I don't buy this. For instance, Paulos thinks: 

the idea of sampling is implicit in words like “instance,” “case,” “example,” “cross-section,” “specimen” and “swatch”

But sampling is a complicated concept. I think we can use words like example, instance, case and so on without ever understanding anything about the science of sampling, like how to construct samples, random v. nonrandom samples, how to generalize information contained in samples, bootstrapping, etc. Just ponder this question for a moment: why is random sampling a good idea if the act of introducing randomness into the proceeding introduces errors? That's a question my students get to think about in class.

I recommend Steven Stigler's books on the history of statistics, in which he convincingly argues how the most important statistical concepts took a long time to develop, after many false starts. Warning: some of Stigler is hard to read without a mathematical background.

***

I love the part where Paulos points out the inevitable tradeoff between false positive and false negative errors. That's one of the key points in Chapter 4 of Numbers Rule Your World. Beware of those people who sell you technology that minimizes just one of the two types of errors.

Then, Paulos walks through several important examples of non-intuitive statistical thinking, first studied by Kahneman and Tversky. This part is fun, and makes you realize yet again how brilliant and important were their contributions.

***

This is a great sentence:

In listening to stories we tend to suspend disbelief in order to be entertained, whereas in evaluating statistics we generally have an opposite inclination to suspend belief in order not to be beguiled. 

He equates statisticians to people who can't stand Type I errors (false positives), seeing things that are not there. He equates storytellers to people who can't stand Type II errors (false negatives), not seeing things that are there. He thinks there are two types of people.

Another great sentence: 

The focus of stories is on individual people rather than averages, on motives rather than movements, on point of view rather than the view from nowhere, context rather than raw data. 

I bolded the part that resonates with me. Much of the abuse of statistics stem from the fallacy that everyone is like the average. Statistics can predict/explain the average very well but it cannot predict/explain individuals. I'll leave it at that for now, and maybe I'll come back to this in the future.

How lucky numbers betray you

This post is inspired by today's New York Times mush piece about people's obsession with lucky numbers when looking to buy apartments.

First thought: the behavior of these real estate agents - extracting small extra fees like $500 to arrange a particular closing date - seems to contradict the Freakonomics's claim that agents have no incentive to push for small changes that affect their fee by small amounts. I don't think Levitt and Dubner are wrong--rather, that they ignored the impact of the general economic conditions; it might well be true that a few years ago, when apartments were selling like hotcakes, small amounts meant nothing but now, when apartments are not moving, the agents are more fee-hungry.

In general, that's one of the things we have to keep in mind when reading Freakonomics and similar books. At the micro level, they are telling us about unexpected effects but very often, these effects have marginal impact on the outcome--and there are other factors that have much stronger effects. Take for instance the advice to time your son's birth to make a major leaguer... if you compare two parents that did everything else the same except when they conceived their sons, then yes, the one who timed the birth might have a little advantage; but in the real world, you can't find two parents that did everything else the same, but you'll find the parents who, say, hire the better coach or send the kid to the school with the better baseball program, would stand a much better chance of producing a major leaguer.

***

Second thought: lots of people who play lotteries pick lucky numbers. (The retiree who got scammed in Chapter 5 of Numbers Rule Your World bought the same set of numbers every time--a combination of the days of birth of his family.) The problem is that these numbers are not unique to you. If you're Chinese and like 8s, lots of other Chinese also like 8s. If you are born on the 10th and your wife on the 20th, lots of other couples also share the same birthdays; in fact, there are still others who'd pick 10 and 20 for other reasons unbeknownst to you.

If the numbers don't win, it's not a big deal. If they win, you find lots of company.

John Haigh's book, Taking Chances, has a lot of great insights about lotteries. For instance, why is the sequence {40 39 48 41 20 45} popular in Canada's 6/49 lotteries? It turns out that these are the 6 least popular numbers in the lottery's history, and this information is published so many players use it. (Notice the paradox: while each number individually is unpopular, the set of unpopular numbers itself is quite popular.)

Sports scams

These articles are highly recommended. Great use of data to unearth an unexpected insight.

The MLB scam

The NFL scam

 

Upcoming lunch talk

On the right panel of this blog, I have been putting information about my upcoming book-related events.

The next one is at the NYU Bobst Library on Tuesday 6-7 pm. (It is a repeat of a talk I gave last week at the same venue, which was very well received.) I'll be signing books.

***

On Wednesday at noon, I'll be the presenter at the INFORMS NYC chapter luncheon. The announcement is found here. The talk will cover what I consider to be the next key challenge in business analytics. It is similar to the plenary talk I gave at the JMP Discovery Summit last month.

I have been involved in the INFORMS NYC chapter for a few years, and we have a solid group of practitioners who meet monthly, and always have great discussion, so I'm looking forward to this talk. Take a look at our past luncheon topics.

Calling something a spade doesn't make it a spade

Andrew Gelman pointed readers of his blog to a really great press release / report for those interested in how statisticians look at real-world data. This is work done by the Highway Loss Data Institute (HLDI) which made the Freakonomics-style claim "Texting bans don't reduce crashes; effects are slight crash increases."

This is a long post as I'm going to walk through the analysis in detail. In short, great methodology, bad conclusion. (Bear with me, I'll get to the spade part.)

***

What's good about the piece is how they found a creative and yet credible way to approach a seemingly impossible problem. Ideally, to answer the question "Do texting bans reduce crashes?", you would ban texting for a random group of drivers, and allow texting for another random group of drivers, and then compare the accident rates for the two groups. Of course, we can't do that. Like most public policies, once you make the policy, you don't have test and control groups to do direct comparisons.

What helped is that not all states have banned texting (yet). So they came up with the concept of comparing a text-ban state with its surrounding states that have not banned texting. The chart on the right showed the accident rates in California (text-ban state) compared to Arizona, Nevada and Oregon, before and after the ban went into effect in 2009.

The two lines are essentially the same, implying that the texting ban did not change the accident rates.

What's impressive about this analysis is that they did not make the classical mistake of looking only at the red line. (If the red line shifted a lot after the text ban but so did the blue line, then something else other than the text ban was probably causing the accident rates to change.)

***

Now we get to the spade part. If you persisted and read to the last 25% of the press release, you learned something super-important. There, they tell us:

Noncompliance is a likely reason texting bans aren't reducing crashes. ... Many respondents who knew it was illegal to text said they didn't think police were strongly enforcing the bans. (my italics)

So, here are the three super-important words: nonawareness, noncompliance, nonenforcement. Sounds like the "texting ban" has not stopped anyone from texting while driving, and this can explain the unchanged accident rates. When they state the claim "texting bans don't reduce crashes", it sends the message that despite drivers not texting while driving, accidents remain as frequent as before. That's absolutely not what the research is saying.

In other words, the test group is called "texting ban" but in reality, the behavior of these drivers was no different before or after the ban. (According to surveys, of the 48% drivers who admit they text, 45% said they ignored the ban.) But naming something is very powerful. We think text-ban states have drivers who don't text but with compliance close to zero, text-ban states are essentially the same as non-text-ban states.

Imagine in a clinical trial: drug A is a pill, drug B is administered by a painful injection. Drug B is in fact extremely effective. However, in this trial, drug A beats drug B handily. But looking more deeply, you  discover that nobody in the test group B actually took drug B because the pain was too much. So in reality, the trial compared drug A to no drug at all. A good conclusion for this study is: taking drug A is better than nothing, and no one wants to take drug B. A poor conclusion is: taking drug A is better than taking drug B.

***

Beyond this, savvy readers should be asking these questions:

• How is accident rate measured? The collision claims per 100 insured vehicles, how is it correlated with the accident rate due to texting?
• Why are vehicles older than 9 years old excluded? What impact might this exclusion have?
• What do they mean by the nearby states where texting laws "weren't substantially changed"? Substantially and substantively are very different; substantially seems to me like they were changed but not by much.
• Notice the implicit assumptions: that neighboring states are comparable

***

Finally, what is the evidence for "effects [of texting bans] are slight crash increases"? The HLDI President elaborated: "In a perverse twist, crashes increased in 3 of the 4 states we studied after bans were enacted."

Problem: that is simply not true. It turns out California is considered one of the 3 states. Look back at the graph shown above, and try to convince yourself that accident rates have increased after the text-ban went into effect. I can't. You don't even need statistical software for this.

Still not convinced... this is the chart for Minnesota compared to Iowa and Wisconsin, where the accident rate for Minnesota after the ban went up by 9%, the largest among the 4 text-ban states. First thing you noticed is that there is virtually no difference between the states after the law went in. All of that 9% difference had to do with something that happened about 5 to 10 months prior to the ban, and if anything, that period should be investigated. But I can't see how one can conclude that the text-ban caused accident rates to go up.

***

But the researchers didn't stop there. They now declare story time! First, they managed to convince themselves that noncompliance was not really an issue:

But [noncompliance] doesn't explain why crashes increased after texting bans. If drivers were disregarding the bans, then the crash patterns should have remained steady. So clearly drivers did respond to the bans somehow...

Absolute shocker. They say that because the texting ban increased accident rates (not even true), that proves that there is compliance, and therefore, they could conclude that texting bans did not reduce accidents!  My advice: go find statistics of how many drivers have been booked for texting while driving in those states.

Then, they spin more fiction, following on that last sentence:

... what they might have been doing was moving their phones down and out of sight when they texted, in recognition that what they were doing was illegal. This could exacerbate the risk of texting by taking drivers' eyes further from the road and for a longer time.

Great sounding story but the problem: this sounds smart if the accident rate went up, it sounds silly if it didn't. Imagine if I buy this story--then the onus is on the researchers to explain why drivers moving their phones down and out of sight did *not* in fact lead to increased accident rates.

This is the discontent of story time. These stories are made up to explain one particular interpretation of the observed data, and do not hold up if the data were to be interpreted differently. Either the stories had to be reversed, or ever more elaborate extensions have to be developed.

 

 

 

Immigration story time

Reader John M. complains about "story time" in this New York Times article, which tells us about the trend among immigrants to no longer change their names after arrival. The subtext is a celebration of reduced discrimination and/or xenophobia against foreigners. (They did not say anything about Arizona.)

***

I agree. The article is prototypical story time: the promise of a data-driven analysis but nothing that approaches quantitative rigor ever materializes; instead, we get free-form speculation.

The only number offered in this entire article is that only about 6 out of 500 name change applications in June 2010 in New York "appeared to be obviously intended to Anglicize or abbreviate the surnames that immigrants or their families arrived with from Latin America or Asia", according to an accounting by the Times.

Even without making a fuss about this number, we must ask how one number can make a trend? What was the rate of change in the past, and what part did discrimination play in these changes?

***

What's wrong with no data? It becomes "he said, she said".  For example, one of the examples given of someone who changed his name was "Tom Lee" aka Wong Ah Ling who was "the unofficial mayor of Chinatown". After mentioning Lee, Anne Bancroft and Charles Steinway, the author wrote this without irony:

The rationale was straightforward: adopting names that sounded more American might help immigrants speed assimilation, avoid detection, deter discrimination or just be better for the businesses they hoped to start in their new homeland.

I don't know who Tom Lee was but come on, if someone was so prominent in Chinatown, how did a name change change his ethnic identity? How did "Tom Lee" sound less Chinese, more American? Okay, I know there is a white dude called Tommy Lee but still, come on. (The author did manage to quote Prof. Nancy Foner later in the article, who essentially contradicted the earlier assertion.)

What's wrong with no data? You get platitudes like "most experts agree," "experts say", "they say," "sociologists say". And for some reason, these experts and sociologists don't use data either.

***

More on story time here and here.

 

 

Statistics calling for inequity?

Jerome C. points us to this ad by the French car-marker Renault:

 For those who don't speak French (including me), this is a marketing campaign in which they will refund 1 out of every 300 car purchases, picked at random.

Jerome says:

How would it sound if a major car brand offered you a $50 rebate on a $15,000 car? ridiculous, and possibly insulting. Instead, car maker renault proposes to refund one car purchase out of 300. The ad campaign suggests that such an outcome is likely, compared to all the losing bets we are implicitly making every day. They are banking on the fact that this proposal is perceived as being more advantageous than typical discounts, although the expected value of the gambit is probably higher.

This lottery is no different from any lotteries out there. For positive outcomes, we tend to over-estimate the chance of success, something I discussed in Chapter 5. In any lottery, the average outcome (adjusted for probability of winning) is a useful number to summarize the universe of potential outcomes and to make a decision to play or not, but no player will win the "average" win; you either win the prize or you get nothing. Why would anyone play a lottery with negative expected return? There are lots of possible answers to this question. The one I like is: those who play like to think if someone has to win the lottery, then it would be me. (If you are the winner, the decision to play has been wise, in hindsight.)

In the case of Renault's campaign, Jerome considers the choice between a $50 rebate for everyone or a lottery offering a 1/300 chance to win a free car. One is a fair, equal distribution of spoils; the other, a winner-takes-all scheme.

Assume the expected values are the same. For Renault, the cash outlay would also be the same, assuming no difference in claim probability. But many customers probably prefer the lottery, the same people who buy lottery tickets every day. Jerome guesses, and I agree, that the latter would be more popular among customers, and thus have a higher marketing impact.