Big Data, Plainly Spoken (aka Numbers Rule Your World)

Small numbers and scams

Statisticians hate small numbers (samples); now there is another reason to hate small numbers. In one word, scams.

The FTC has shut down a scam in which the crooks have sneaked through 1.35 million fraudulent credit-card charges, each valued at $0.25 to $9 -- after letting it run for four years. What's shocking is that less than 5% of the victims (78,724) noticed and reported the charges. So, instead of stealing $1 million from one person, steal $1 from a million.

Reading this article, I'm reminded of similar schemes:

• In 2006, a "student entrepreneur" asked Web users to buy $1 each to own one piece of a million-piece on-line jigsaw puzzle. The proceeds would "pay for his degree" -- how a degree can cost a million bucks I'm not sure. (his site).
• Once in a while, I get a notice of the results of some class-action lawsuit that I have been made a party to without applying. They would tell me that I am eligible to receive some tiny amount of compensation (say, $5). If less than 5% bother to collect their share, I wonder who gets the spoils (the lawyers?). I'm not a lawyer so I don't know the answer but I have my suspicions.

The onlyOne difference being that our society does not call these "scams".

Some comments on football statistics, why it is hard

Even the ESPN announcers are realizing that the statistics being printed on the screen have little to do with what is transpiring in the World Cup matches. In a recent match, I noted that ESPN showed or spoke about these statistics: number of shots, number of shots on target, possession time (as proportion of playing time), total number of touches (or passes), number of corners, number of miles run.

A friend of mine asked me to predict the World Cup winner, which got me thinking about how one would go about predicting such a thing. One of the ideas in Chapter 2 of Numbers Rule Your World is that good statistical thinkers have a feel for the nature of problems. In sports, baseball is much easier to deal with than football (soccer) from a statistical perspective. Why?

***

Trying to predict the World Cup winner is fraught with challenges. Here are some starters:

• National teams do not play together regularly. They get together every four years to play in the World Cup, and in between, they play some continental championships like the Euro, and some inconsequential tournaments and/or friendlies. There are very few repetitions of competitive, meaningful matches between any two countries. Statistics require repetitions and this is one big problem.
• There are also not many situational repetitions. Unlike baseball, where every at-bat can be thought of as a repetition (or perhaps, every at-bat plus on-base configuration is a repetition), football is a fluid game, making it difficult to extract comparable situations. While corner kicks are easier to identify as "a situation", they too present challenges because teams use different offensive tactics and different defensive tactics and different overall strategies, which make one corner kick somewhat different from another. There are probably not more than a dozen corner kicks per match anyway.
• Like baseball, football also turns on one or a few moments during any given match. The Swiss had barely any action against the Spaniards except for that one moment. Whether the US-Slovenia match was tied or not came down to a blown call. Calling one football match is as difficult as calling a single baseball game.
• The types of data that are easiest to collect in a football match have very little predictive power... possession time really doesn't mean anything if one side is playing a defensive strategy; number of shots doesn't mean anything if one team is unable to penetrate into the box and keeps taking long-range shots; number of miles run also means little if one side is playing defensive tactics.
• Football is much more of a team game than baseball. Goals typically come about based on coordinated actions by several players, and so statistics that measure one player or the entire team are unlikely to capture the information needed to predict success.
• Football is very tactical. Statistics reflect innately the tactics adopted by the team in any given match. I think if teams hire statisticians to analyze their performance based on intimate knowledge of the tactics they were deploying during the match, the analyses could be very useful. It is much harder for outsiders without knowledge of the intended tactics to analyze performance effectively.

***

Predicting the winner of a match is probably not very useful except for those gambling on sports. What is, to me, a better question to ask is whether there are statistics that if provided during the course of the match will allow us to understand the relative strength of the two teams, how the match is going, and how it might end. For instance, in baseball, knowing the strike/ball ratio and the number of pitches thrown gives us a good sense of whether the pitcher is having a good or bad day.

Typos show the value of statistics

Over at Junk Charts, I examined Nate Silver's ranking of New York neighborhoods (first published in New York magazine): Which factors affected the rankings? How did the factors correlate amongst themselves?

While analyzing the data (which I hand-transferred from the printed pages), I found a moderate number of typos, scores and ranks that don't make much sense. Now, I am not here to criticize their editors because as anyone who makes a living analyzing data knows, typos and other data issues are the norm, not the exception in this business. What I want to do here is to describe how I uncovered the typos, and more importantly, why statistical analyses are often immune to such typos.

***

On the right are plots of the scores against the ranks for each category (factor) being evaluated. We expect to see a monotonically decreasing function, i.e. as rank increases (moving from left to right), score must decrease (or stay put), score should not increase.

The sharp valleys and peaks in almost every one of these charts are typos. For example, the sharp valley in the "Creative Capital" corresponds to Parkchester, ranked 29th in this category, but its reported score of 63 is much lower than the Harlem (75, rank 28) and Astoria (74, rank 30).

I spent quite a bit time trying to fix these errors, trying to use the surrounding data to reason whether the rank or the score was mis-typed. It was a fruitless exercise.

(Look at "Green Space" for example. The line went up and down, indicating that there were many typos, and any fix would have involved a whole series of changes.)

***

In practice, data analysts do not fix typos unless they are extremely egregious and unambiguous -- and even then, the fix may just be to restate the value as "unknown".  One reason is that one doesn't want to make a bad situation worse. Another is that statistical techniques by definition generalize the data, and thus are not very sensitive to individual values.

To illustrate this point, I did a linear regression of category scores and overall scores. According to Silver's ranking formula, the overall score should be a weighted average of the category scores, e.g. housing affordability had a weight of 25% in the formula.

The regression answers the question of how much of the overall ranking is explained by the individual category rankings. It should be 100% if there were no typos -- if you know the category scores, you should be able to derive the overall score without uncertainty. Because there are typos, the correlation will be slightly off.

The chart on the right shows that the correlation is almost but not perfect. The chart compares the actual overall score as reported in New York magazine with the "predicted" overall score as per the regression analysis.

The regression in effect "recovers" the weights used by Nate Silver in his algorithm (shown to the right). Despite the "noise" introduced by the typos, the weights found by the regression (shown in the column labelled "Estimate") are almost exactly those used by Silver.

This is why many statisticians are not overly concerned with small errors in the data. We expect that data is not clean, and we know many of our techniques can overcome those errors.

***

PS. Here is my post on Junk Charts on Nate Silver's rankings.

Tip #1 for reading economic stats: pull apart threads

I will be writing an occasional series of posts, discussing useful tips for reading the volumes of economic statistics that fill the airwaves daily, starting with this post on pulling apart time-series data.

***

In "True Lies and Economic Statistics", I pointed out the statement that May 2010 had the "biggest monthly increase in jobs in the past ten years" is a "true lie". Anyone citing the job growth number is making a judgment on whether the economy is healthy enough to create jobs. But the 431,000 number does not answer this question.

The key to reading the job growth statistic is to pull apart the multiple threads that it contains. Within the 431,000 figure are the 411,000 temporary workers for the Census, hiring that only happens once every 10 years; the remaining 20,000 new jobs are more reliable but to be meticulous, we ought to further ask:

• how many of these were temporary versus permanent jobs?
• Were there any statistical adjustments (e.g. seasonal adjustments, so-called birth/death model adjustments), and
• if so, how large were those adjustments?

***

Think about your electricity bill. It was $100 in May; it went up to $150 in June. You want to know why: in particular, you want to know if the $50 jump reflects a persistent shift in your usage level.

You realize that your energy use jumps at the start of every summer, because of air conditioning. Looking up the GE energy use website, you estimate that your room a/c should add about $30 per month to your bill during the summer. So at least $30 of the $50 jump is due to "normal" usage. This, in essence, is what statisticians mean by "seasonal adjustment".

Next, you throw a World Cup launch party for your college buddies, in June every four years. So every four years, your June electricity bill shoots up. Looking up past June bills, you can estimate how much energy your buddies use while they are in your house. This is what is known as a "cyclical adjustment" because such extra spending also says nothing about your "normal" usage level.

***

Similarly, we can pull apart many threads in the job growth number. Roughly speaking,

Observed job growth - seasonal effect - cyclical effect = economic growth effect + margin of error

For May 2010, according to the Census Bureau, this equation read thus:

1,090,000 - 659,000 - 411,000 = 20,000 +/- error.

Yes, the survey result reported an increase of 1.1 million jobs from April to May (you wouldn't know from published reports). However, 60% of this growth represent a spurt in hiring at the start of summer, something that happens every year and thus not an indication of a growth spurt. In addition, almost 40% of the observed growth are the temp Census jobs, which happens once every ten years, and also not an indication of a growth spurt.

To make matters worse, the sample size of the survey leads to a margin of error of +/- 100,000 jobs. And the 20,000 jobs created is much smaller than the possible error and so the best we can say about May jobs is that the job market flat-lined.

***

The Census automatically makes the seasonal adjustment (something they find so routine that they do not even mention, in the main text, that the published numbers are post-adjustment). But Census does not adjust for the once-in-a-decade spurt in Census hiring. I find that odd.

They may not do it, but you and I should do it.

P.S. Please see my older post ("The Real Climategate") for an example of how statisticians decompose time-series data into various components, basically a graphical representation of the above equation.

True lies and economic statistics

Or, the treachery of objective journalism

***

Those were my thoughts as I made my way through the New York Times article about today's jobs report. (Not picking on the Times in particular; this is a disease of the so-called "objectivity" in journalism.)

The article keeps stating the "facts". For example:

Employers added 431,000 nonfarm jobs nationwide in May, the biggest increase in a single month in a decade, the Labor Department said Friday. (my emphasis)

This sounds objective -- it is a "true" statement that this number is the "biggest increase in a decade". But it is also a "lie" because this is also the only month in the past decade for which the Census took on hundreds of thousands of temporary workers. The article also stated this additional "fact":

But the bulk of the growth was in government jobs, driven by hiring for the 2010 census, and private-sector job growth was weak.

A few paragraphs later, the article stated:

Altogether, 411,000 of the jobs added were for census workers whose positions will disappear after the summer.

Here lies the rub. We cannot argue with this paragraph because all statements are true as stated but this is not what I consider "good reporting". This reporting misrepresents "what the data means".

Here's one version that calls a spade a spade:

Employers added 20,000 nonfarm jobs nationwide in May, the Labor Department said Friday. In addition, 411,000 jobs were added for temporary workers for the once-in-ten-year Census, positions which will disappear after the summer.

***

In data reporting and analysis, it is very dangerous to be "objective" in the sense of "just stating the numbers".  This is a very important point to realize. You want your data analysts to interpret and understand what the data means. The fact that this increase was "the biggest in a decade" has zero value, none whatsoever! To mention this in the first sentence of this report is a travesty, an embarrassment.

***

Many economics blogs are vigilant about reporting of economic statistics in the media. I recommend Dean Baker's Beat the Press, and Calculated Risk even had a preemptive strike about the reporting of this jobs announcement (CR has analysis here).

PS. My related post about how to read these type of data.

Judging predictions of rainfall

The overheard conversation from the other day suggests a curiosity towards a supremely important aspect of statistical predictions -- that of validating models.

In essence, the men on the street were using the outcome (rain drops) to lay down a judgment on the weather forecasters.

How should one use statistical thinking to deal with this problem?

***

Forecasters issue probabilities e.g. 20 percent chance of raining today. At the end of each day, the one-day-ahead forecast from the prior day could be evaluated. Either it rained or it did not rain. While the forecast is in probability, the outcome is a binary yes/no. And this is why validation is not straightforward. As the men of the street showed, the one day's worth of data (one Yes or one No) does not provide sufficient evidence to judge the accuracy of the forecast.

When a statistician predicts 20 percent chance of something happening, she is imagining alternative universes. As she exists today, she imagines all possible tomorrows; in one universe, tomorrow would bring 1 inch of rain, and in a different universe, tomorrow would bring 0.1 inch, etc. etc. When she says 20 percent chance, she means in 20 percent of all possible scenarios for tomorrow, she expects to see rain.

Since tomorrow only occurs once, we have a problem: we cannot directly verify such an assertion. What to do?

***

We can indirectly verify this assertion. Do the following:

• On every day in which the forecast is given to be 20 percent, note the actual outcome (rain or no rain).
• Calculate the proportion of such days in which it actually rained.
• Do this also for 30-percent forecasts, 40-percent forecasts, etc. (in reality, you would establish ranges of probabilities rather than exact probabilities)
• If the actual proportion of rainy days mirrors the forecasted probabilities, then we have confidence that the forecasters are doing a good job.

Notice the sleight of hand. In the idealistic version, the base consists of all possible tomorrows, and each day-ahead forecast could be validated. In this realistic version, we abandon hope of validating each forecast and shift our attention to validating groups of forecasts.

Please leave a comment if you have seen this sort of validation for weather forecasts, or if you know of other ways to do validation.

Statistically improbable words 3

Background: We, in New York City, have been basking in the summer sun the past few days. Of course, rain threatened each afternoon. And when a few drops started on Sunday, I overheard the following exchange, outside a gym, under an awning:

Man 1: No way this is 20 percent.

Man 2: Damn right, this is more like 50 percent, not 20 percent.

Man 1: Could be 40 percent but 20 percent is just stupid.

Bless the amateur weathermen.