Big Data, Plainly Spoken (aka Numbers Rule Your World)

The language of data, the math of retail

Clive Thompson tells Wired readers that we should all speak the language of data. (The online version is here. The article also appears in the May issue.) He argues that statistical illiteracy is the nation's political problem. "If you don't understand statistics, you don't know what's going on -- and you can't tell when you're being lied to."

In other words, everyone should think like a statistician... my pitch to the Baltimore Sun reporter is here.

***

As happens with these interviews, a great deal was said, much of it vanished during the editing process.

The example I provided illustrates "survivorship bias". The most important metric for the retail sector is same-store sales growth, the average change in sales experienced by individual stores. Less known is the fact that only stores opened for at least one year are considered eligible for the sample: this is sensible since new stores may experience "growing pains" and thus unfairly drag the metric down.

Even less known is the fact that stores closed during the reporting period are excluded: this is sensible in a stable economy; removing transient activity can be justified if one wants to measure the average trend. In a recessionary environment, such an exclusion creates a bias in the sample, which has the effect of magically creating sales growth when sales is merely migrated from one store to another.

Imagine there are four Starbucks in your neighborhood (quite likely in NYC). Starbucks decided to shut one of the (low-performing) stores down. Because this store was not reported as open at the end of the month, it was deemed ineligible for the sample. Meanwhile, all the customers of this store took their business to the other three Starbucks in the neighborhood. Those three stores saw a jump in sales. The jump in sales constituted same-store sales growth, when in fact they merely migrated from the closed store.

It's a form of "survivorship bias" because the sample used to estimate same-store sales selectively included only "survivors". If the closed store were also added to the sample, then the drop in sales (to zero) at this store would cancel out the jump in sales in the other three stores, which would reflect the economic reality.

So, yes statistics is tricky but it can be learned.

Thanks, Clive, for devoting a column to make this important point.

Update: Friday talk

Dona Wong and I are giving a joint talk at the NYU Stern School on Friday. Dona is the author of the recent book, Wall Street Journal Guide to Information Graphics (reviewed at Junk Charts). My presentation will be titled "Five Years of Chart Reading."

Reading SuperFreakonomics 3

The third chapter of SuperFreakonomics is simpler in structure than the other two chapters, containing just two parts, one dealing with the Kitty Genovese murder and the other with the research of Chicago economist John List, a colleague of Levitt.

The chapter touches upon a number of keystone experimental results from psychology, exposed to anyone who have taken PSY 101. The Kitty Genovese case is a real-life example of the "bystander effect", the tendency of human beings to offer no help to strangers because they are expecting others to offer help. 

The Ultimatum and Dictator games are used to study human rationality and the attitude toward inequity.

The Stanford prison experiment and the Milgram experiment at Yale were also briefly touched upon, these being studies of "obedience to authority", used to show that people have a capacity to do really bad things when given roles as authority figures (Stanford), or subject to authority figures (Yale).

***

Here are, again, my thoughts on statistical topics as I read the chapter:

pp.100-1: Raises an important point that randomized controlled experiments in criminology that are also ethical and politically acceptable are very difficult to design. The thought experiment of a randomized experiment they concocted to "know whether putting more people in prison really lowers the crime rate" doesn't make much sense to me:

Pretend you could randomly select a group of states and command each of them to release 10,000 prisoners. At the same time, you could randomly select a different group of states and have them lock up 10,000 people, misdemeanor offenders perhaps, who otherwise wouldn't have gone to prison. Now sit back, wait a few years, and measure the crime rate in those two sets of states. Voila!

In case you are interested in running randomized experiments, here are a few pointers:

• Even though there are two groups of states, there should be only one randomization that puts states into two groups. They did use the phrase "at the same time" but I find the sentences a bit confusing.
• Think clearly about the unit of analysis (also the sampling unit): for this example, it could be "state", "locale", "prison", "prisoner".
• Think about the nature of the difference in treatments applied to the groups: what is really being measured when one group releases 10,000 guilty while the other group locks up 10,000 "innocents"?

p.115: L&D wrote: "In Virginia, List cruised the trading floor and randomly recruited customers and dealers, asking them to step into a back room for an economics experiment."

This hits on one of my pet peeves. Whenever I see an assertion of "random selection" in an observational study, I want to ask what mechanism was used to ensure randomness. Think, for example, when the NYC subway police tell us they "randomly" inspect bags, how is it precisely that they enforce such "randomness"? Are they picking on every Nth passenger? Do they carry a pseudo-random generator?

Not knowing the precise selection rules is not an excuse for assuming random selection. In this example, I'd like to know how List "randomized" the recruiting process.

pp. 118, 122: would love to see sample sizes cited alongside the results from the Ultimatum/ Dictator experiments, all reporting fairly large differences between groups. I'm sure they were large enough, but statisticians always worry about the so-called "law of small numbers" (attributed to Kahneman and Tversky). When sample size is too small, even very large differences may occur by chance.

pp.120-3: Discusses the shortcomings of the experimental economics field, which has become quite influential (see Dan Ariely's book). Many of these issues are exactly the things statisticians worry about... selection bias, nonresponse bias, generalizability, observer effect, etc. Well worth reading and pondering.

What I don't get about this section is how they can be so negative on these "lab studies" while simultaneously so admiring of John List's research. As far as I can tell, List also ran "lab studies". The main difference is that in some cases, his subjects were not students but traders taken from baseball-card trading floors. These subjects knew they were part of an experiment and were instructed to do certain things.

But I agree with their general points about lab research, and also think List's research is useful. I hope List and other researchers will reach out to the market research and political polling communities because many of the problems they face are not new problems; they have been studied for a long time, and these communities can help each other.

pp.122-3: They continue the takedown of lab research, citing a researcher saying "lab experiments have the power to turn a person into a 'stupid automaton' who may exhibit a 'cheerful willingness to assist the investigator in every possible way by reporting to him those very things which he is most eager to find'." This is where they introduce the Stanford and Yale experiments as proof of "forced cooperation".

When I was taught about these experiments (admittedly many years ago), they were evidence of the human capacity to do bad things but L&D's point here is contrarian: they say these bad things could only happen in labs but not in real life. This is a new interpretation to me.

p.124: am glad to hear about "warm-glow altruism" research, would love to learn more.

pp.125-: these pages are a take-down of the New York Times reporting on the Genovese murder. Interesting story, looking for a rejoinder.

The culture of misinformation that pervades the business press

Carl Bialik (The Numbers Guy, WSJ) is the first business journalist to talk about the "margins of error" around the economics statistics that are a staple of the business media empire.

In a previous post ("Agreeing to disagree"), I documented how the entire business press decided to ignore the Census Bureau's warnings about the unreliability of many statistical estimates based on very large margins of errors, due to very small samples. This is the article that wasn't deemed fit to print.

My example was the retail sales growth statistics, where the previous month's "revision" was larger than the reported growth in the current month. Bialik showed the same problem with unemployment estimates, new home sales data, and so on. Specifically, he cited a margin of error for the February unemployment to be between -200,000 and +511,000. The width of this margin of error is such that it is utterly useless. Did it stop the business press from flushing out stories about an unemployment decline? No.

(Glossary: The 90% confidence level means that 1 out of 10 times, the true value of the economic statistic will fall outside the advertised margin of error. I'm warning you: do not count how many statistics are issued by the Census Bureau each week.)

***

I happen to also be reading Dean Starkman's "How Could 9,000 Business Reporters Blow It?" (Mother Jones), recommended by Brad Delong. Starkman attributed the problem to: "Increasingly, business coverage has addressed its audience as investors rather than citizens", and the "triumph of Wall Street insiderism." He has a dim view of the future, given the dire straits print journalism finds itself at the moment.

I prefer Felix Salmon's simpler explanation, and a very courageous act to flog himself in public: 

journalists, myself included, read economics papers: we generally have no ability or inclination to try to understand the details of the formulae and regression analyses, so we confine ourselves to reading the stuff in English, and work on the general assumption that the mathematics is reasonably solid.

Yes we hate passwords

This is an article that should have been written a long time ago. It's great that Mark Pothier (Boston Globe) finally wrote it. (Another article that begs to be written: how much productivity Microsoft has destroyed with the new Office. I don't need to see another review of the iPad; how about a proper review of the new Office?)

The bottom line: most online security measures are time-wasters that have no proven benefits, the equivalent of snake oil. Don't you ask us to make up another password.

***

Here are some passages that we can approach with a statistical mindset:

While doctors can cite statistics showing smoking causes cancer, and road-safety engineers can produce miles of numbers supporting seat belt use, computer security professional lack such compelling evidence to give their advice clout.

Great point about having data to support the decisions to roll out irksome security measures.

However, Mark totally under-estimated the immense intellectual gyrations statisticians went through to prove smoking causes cancer. I mentioned this briefly in Chapter 2 of Numbers Rule Your World -- even a giant of the field, Fisher, disputed the causal link, claiming that one day a gene would be found that caused us both to smoke and to get cancer!  Much of Chapter 2 concerns the tremendous resources devoted to finding the source of one E.coli outbreak. None of this is simple stuff.

The connection between seat belt use and road safety has to be even more tenuous. This is a practical situation where statisticians clearly cannot run a randomized experiment (which would force a randomly selected group of drivers to not wear seat belts), and in these cases, cause--effect relationships are notoriously hard to establish.

 

Security professionals need to consider that user education costs everyone (in time), but benefits only the small percentage who are actually victimized, [Cormac Herley, a principal researcher at Microsoft] wrote.

This sentence is too simplistic. Yes, only a small percentage will be victimized. The key to understanding this situation is to realize that who gets victimized is not known beforehand! This is a classic case of "decision making under uncertainty". We have to take action before we know who among us may become victims.

This issue is analogous to the insurance problem discussed in Chapter 3. All insurance schemes -- when all is said and done -- result in subsidies, transfers of wealth from one group to another. However, many of us happily participate in these subsidy schemes. Why? It's because at the time we pay the premiums, we cannot predict whether we would be stricken by a horrific illness or a freak accident. In that sense, everyone has equal opportunity to claim the rainy-day funds. (I'm simplifying a bit here. There are moral hazard and adverse selection problems to deal with.)

 

Banks and other investment companies often guarantee to reimburse customers if unauthorized withdrawals are made from their online accounts, so the customer does not pay a direct price. The banks face losses, but they are relatively modest -- the annual cost nationwide as a result of phishing attacks is $60 million, Herley estimated.

Herley thinks banks are willing to foot the bill because the total cost to the industry of phishing attacks is small on average. That is one reason but something else - statistical - is happening here.  The law of large numbers (see also here). It is very difficult to predict how much loss will be suffered by any individual customer; however, the total loss inflicted on all bank customers is very predictable due to the large sample size.

Update: Kindle, events

The Kindle version of Numbers Rule Your World is now available from Amazon! You can now read my book at the bargain price of 9.99, delivered in "under a minute".

***

On April 23, I will be attending the 10th anniversary celebration of Princeton's Operations Research and Financial Engineering department. There will be two panel discussions on "Lessons from the Financial Crisis" and "Energy and Climate Change". This will take place in their brand-new building, Sherrerd Hall.

ORFE has become one of the most popular majors at Princeton. It's perfect for students who want to have one foot in engineering and the other in the social sciences. It's always been a great feeder program to Wall Street banks and management consulting firms although if any of the students are reading this, there is life outside of banks and consultancies! There are plenty of businesses who need quantitative thinkers like you.

The department is much older than ten. When I graduated, it was combined with Civil Engineering. This is their 10th year since splitting from Civil.

***

On April 30, I will be giving a talk at NYU's Stern School on "Five Years of Chart Reading" (Kaufman Management Center, Room 5-90 at 11:30 am). This is a joint event with Dona Wong, the author of The Wall Street Journal Guide to Information Graphics. Please come and see us.

Bad scores

In Chapter 2 of Numbers Rule Your World, I explained why most of the accusations hurled at credit scoring miss the mark. For most consumers, especially those behaving honorably, credit scoring offers many benefits.

However, success breeds over-reach, and when credit scoring companies try to sell their wares as indicators of good/bad employees, they surely are selling snake oil. This reminds me of the effort by pharmaceutical companies to find other uses of drugs that have been granted patents for one use.

***

Andrew Martin of the New York Times wrote about the use of credit scoring by companies to screen employees. Eric Rosenberg of TransUnion, one of the big three credit reporting services, admitted that there is no research that exists to show any correlation between high credit scores and likelihood to be "good" employees. In the meantime, his employer continues to sell to employers, and is fighting legislative challenges to using scores in this way.

Employers say they use credit scores hoping to prevent "theft and embezzlement". Credit scores predict the probability of someone to fail to repay a loan in the not-too-distant future. How is that related to stealing I cannot fathom.

A couple of additional points (not covered by Martin) to ponder:

• The biggest fallacy of this use of credit scores is the underlying model that says an employee's behavior at home is the same as his or her behavior at work. This is patently untrue.
• The second fallacy is the mistaken interpretation of a correlational model as a causal model. Credit scoring models do not make the distinction between someone failing to repay a loan because of questionable personal behavior or falling on hard times. They just tell us that this someone is likely to miss payment. When employees use these scores, they over-interpret the model when they assume missed payment is always due to personal misbehavior.

 

Reference: "As a Hiring Filter, Credit Checks Draw Questions", Andrew Martin, New York Times, April 10 2010.

A novel view of the placebo effect

This item in the latest RSS News caught my attention since I have written about the placebo effect (here, here).

Stephen [Senn] argued that the placebo effect is really an example of [regression to the mean]. Selecting only patients for a clinical trial on a treatment of diastolic blood pressure (DBP) that have a DBP above a certain cut-off, for example, means that as a group the patients are bound to improve since their starting point is at the extreme. Excluding patients with lower DBP misses those in that group that would actually get worse (ending up with 'high' DBP).

Senn is the author of one of my favorite non-technical introductions to statistics (Dicing with Death), and I think he's pointing out something very intriguing. He is, in effect, arguing that the so-called placebo effect does not exist; it is a statistical artefact.

Is this a crazy idea? Or is this brilliant? What do you think?

Agreeing to disagree

This is a post I wrote several weeks ago. It's a must-read just because no one in the mainstream media is interested in carrying it. So I must be saying something right.

***

Last week, the business press was abuzz in reaction to the newly-released retail sales statistics. Everyone agreed that (1) retail sales went up 0.3 percent month to month and (2) this statistic proves a sign of economic recovery and/or the resilience of U.S. consumers. Here are a sampling from the usual suspects:

Associated Press

• Headline: Retail Sales Post Surprising Gain in February
• Lede: Retail sales posted a surprising increase in February as consumers refused to let snowstorms stop them from stepping up purchases for everything from clothes to appliances. The improvement provided hope that the recovery from the Great Recession is gaining momentum.
• Data: For February, sales rose 0.3 percent, the Commerce Department said Friday. That surpassed expectations of a 0.2 percent decline. (paragraph 4)

Wall Street Journal

• Headline: Shoppers Buck Snowstorm to Push Retail Sales Higher
• Lede: U.S. retail sales posted a surprising gain in February despite falling car demand amid trouble at auto maker Toyota Motor Corp. and fierce blizzards that crippled the East Coast for days.
• Data: Retail sales rose last month by 0.3%, the Commerce Department said Friday. Economists surveyed by Dow Jones Newswires had forecast a 0.3% decrease in February sales. (paragraph 1)

Washington Post

• Headline: Retail Sales Up Slightly in February, Despite Nationwide Snowstorms
• Lede: February's cross-country snowstorms couldn't freeze out consumers as retail sales rose 0.3 percent for the month, according to government data released Friday. The increase was higher than many analysts had expected considering that almost every state had snow on the ground at one time last month. ...
• Data: ditto (paragraph 1)

Time

• Headline: A Hopeful Economic Sign: February Retail Sales Jump
• Lede: Despite blistering snowstorms and a dip in consumer confidence, retail sales rose strongly in February, signaling that recession-weary shoppers may finally be opening their wallets. Yet experts quickly noted that consumers have not convincingly abandoned their penny-pinching ways and likely won't for a long time.
• Data: Seasonally adjusted figures released by the Commerce Department on Friday show U.S. retail and food-services sales rose 0.3% from January and 3.9% from a year ago. (paragraph 2)

CNN Money

• Headline: Retail Sales Beat Expectations
• Lede: Retail sales rose in February, the government said Friday, surprising economists who expected a decline.
• Data: The Commerce Department said total retail sales edged up 0.3% to $355.5 billion last month. Economists surveyed by Briefing.com had anticipated that February sales would drop 0.2%. (paragraph 2)

Business Week/Bloomberg

• Headline: Sales Rise as Buyers Overcome Snow, Job Concerns
• Lede: Americans braved blizzards and overcame job concerns to propel retail sales in February, pointing to a broadening in growth that will help sustain the expansion.
• Data: Purchases unexpectedly climbed 0.3 percent, the fourth gain in the past five months, Commerce Department figures showed today in Washington. (paragraph 2)

New York Times

• Headline: U.S. Retail Sales Rose in February
• Lede: February was hardly an ideal month for retailers: snowstorms blanketed many parts of the country, and car dealerships faced an uproar over safety concerns.
• Data: The Commerce Department retail sales increased 0.3 percent from January, much better than the 0.2 percent decline Wall Street analysts had forecast. (paragraph 3)

The NYT was sufficiently impressed to designate 0.3 as a "prime number". 

***

Everyone agrees about the number and what it means. So many media outlets, one echo chamber. But...

***

No one agrees with the Census Bureau, the source of the data. Here are the salient parts of their press release:

Headline: Advance Monthly Sales for Retail Trade and Food Services

Lede: The U.S. Census Bureau announced today that advance estimates of U.S. retail and food services sales for February, adjusted for ..., were $355.5 billion, an increase of 0.3 percent (+/- 0.5%)* from the previous month and 3.9 percent (+/- 0.5%) above February 2009.

* The 90 percent confidence interval includes zero. The Census Bureau does not have sufficient statistical evidence to conclude that the actual change is different from zero.

(Also of note, the estimate for the percent change between Dec 2009 and Jan 2010 was revised down 0.4 percent, larger than the supposed change in Feb 2010.)

What they released was an "advance estimate" based on a small subsample (5,000 firms) of the usual sample used to estimate these statistics. If the conventional 95 percent confidence level were used, the margin of error would have been even wider.

So, our journalists agree with each other in disagreeing with the Census Bureau.

***

In the second-to-last paragraph of the release, the Bureau statisticians warned:

individuals using estimates in this report [should] incorporate this information [about margin of error] into their analyses, as sampling error could affect the conclusions drawn from the estimates.

Maybe they should be more direct: the 0.3 percent number is likely pure noise.

Learning from climate data

This is the promised second post in reaction to Phil's piece on Andrew's blog about dealing with dirty, complex climate data. In a prior post, I considered the issue of a perverse incentive in data processing, and showed how it also affects credit reporting and scoring.

***

At the end of his post, Phil surfaces a topic that will clearly irk some -- when there is a gap between the data and the model, should one fix the data or fix the model?

Since I wrote about this topic here as it relates to predicting Olympic medals, in a post called "False belief in true models", you might understand that my first reaction was: fix the model, the data is reality! By contrast, Phil indicated that it is often prudent for climate scientists to fix the data to bring it closer to the model. How might one reconcile the two points of views?

The reason for my post on "false belief in true models" was my displeasure with many business and economics folks who talk incessantly about over or under performance relative to a "model". For example, the employment statistic did better than "expected" even though the growth in jobs did not keep up with population growth, meaning the nation was worse off. This type of statement is tantamount to saying the model is always true, but a statistical model can never be true.

Then, Phil brought up this scary prospect:

The models are close to being correct. In this case, gross discrepancies between data and models will indicate problems with the data. Fixing those problems will lead to data that are in better agreement with the models.

In effect, he is saying Data is Not Reality. Uh oh.

He explained further:

When the data are complicated ... then it's not necessarily a surprise to find problems with the data, and to find that when those problems are fixed, the result is better agreement with a model.

I fully understand what he means. The data environment faced by climate scientists is many orders of magnitude more complex than for say businesses. If I need to count the number of gadgets sold through a website, these transactions are recorded, and the data is relatively clean. Climate data is extremely hard to collect... Phil talked about thermometers installed on 3000 undersea robots, for example. The errors, such as forgetting to notice that satellites were moved closer to earth, are not easy to catch since presumably the data analysts were not the ones ordering the satellite locations to change.

***

In other words, adjusted data is reality. Unadjusted data is not. I should have made this clear. In statistical analysis, the first step is to inspect the data, and correct any errors. It is best if such data cleansing is completed prior to the analysis; what Phil appears to be saying is that some of these errors are so subtle that it is only when compared to a reasonable model that they would come to light.

At the end of his thoughtful piece, I again feel that climate scientists are not giving themselves enough credit. I don't think it is correct to describe the data cleansing activity as "bringing the data closer to the model". Instead, he should describe it as correcting obvious errors in the data, or reducing measurement error.