Big Data, Plainly Spoken (aka Numbers Rule Your World)

Good guys in sports need a dose of reality

I will be speaking at the Agilone Data Driven Marketing Summit (link) in San Francisco on Thursday. I will be talking about hiring for numbersense. Drop by if you are in the area. Future events are listed on the right column of the blog >>>

***

I feel bad piling on the "good guys" in the sports doping spectacle but sometimes, you need someone to point you to the mirror.

Here are the breathtaking first sentences from an article in Canada's The Globe and Mail about the scarcity of positive doping results in Sochi 2014:

At the midpoint of the Sochi Games, not yet marred by a single case of doping, the IOC’s top medical official said its efforts to catch drug cheats were so successful they had scared them all away.

A week later, after the disclosure of a fifth doping case on the final day of the games, IOC president Thomas Bach cited the positive tests as the sign of success.

If you have been reading this blog, you already know the people in the anti-doping business set themselves a really low bar. The title of Chapter 4 of Numbers Rule Your World (link) contains the phrase "timid testers" for a reason.

The statement by the unnamed "top medical official" is the more shocking. If there are no positive test results, and such is considered an accurate portrayal of the doping situation, then we must believe that there are no dopers. Apparently, this official believes no athlete that has been tested doped. Not a single one.

In the article, the reporter continued:

Who’s right? To [IOC president] Bach, it doesn’t much matter.

“The number of the cases for me is not really relevant,” Bach said. “What is important is that we see the system works.”

Now, it's Bach's turn to display his ignorance of the statistics of anti-doping. As I explained years ago in the book and also on this blog, the proportion of tests that come back positive is one of the most important numbers to look at when judging the success of an anti-doping program. So far, we know that six out of 2,630 athletes tested positive, meaning the rate of testing positive is 0.23%. (Much less than 1 percent is the norm in all large international events.)

What does that mean? If one percent of athletes doped, then we should expect 26 positives if the tests were 100% accurate. Since they only caught six, at least 20 of the 26 dopers passed the test. Yes, that means over 80% of dopers passed. (And I'm only assuming one percent doping, and not allowing the possibility of false positives.)

***

This leads me to the as-yet unrecognized scandal. Lance Armstrong, Ryan Braun, Mark McGwire, Alex Rodriguez, etc. etc. None of these confirmed dopers were caught by steroids tests. In fact, all of them boasted at one point or another that a long string of negative test findings proved that they were innocent.

Rather than gloating about the "success" of anti-doping measures, they should try explaining how the most notorious dopers in sports were repeatedly given a clean bill of health.

I am a supporter of anti-doping. I just want some discussion of the false negative problem.

Numbersense and true lies

Long before I came up with "numbersense," I wrote about "true lies" in data analysis. (link)

The nature of data, especially Big (as in multidimensional) Data, is that one can come up with an infinite number of statistical computations, all of which are "true" in the sense that one would obtain such statistics were one to plug the data into textbook formulas. Inevitably, some of these statistics lead to contradictions.

An example I give in the Prologue of Numbersense (link) is a case of Simpson's Paradox. There are two ways to compare two airlines' rate of delays during a given window of time at a common set of arriving airports. One can aggregate the number of flights across all airports, then compare the average rate of delay. Alternatively, one can compute a pair of delay rates for each airport, then compare the rates by airport. In the example given in the book, airline A came out ahead in the aggregate measure but was the more delayed at each of the individual airports. This is an instance of "true lies". Airline A is either better or worse, not both. Given that answer, one of the two methods leads to the wrong interpretation. But one cannot complain that there is anything wrong with the data or either formula used to compute the average.

***

I was thinking about "true lies" while reading the exchange between Alberto Cairo and Andy Kirk about the following chart, which prints a "truth," that half of US economic activity occur in major urban areas that constitute a tiny proportion of US territory.

Cairo complains that this chart is silly because about half the US population live in those orange urban areas so in reality, anyone who accepts the meme that this map has "incredible" insight is just surprised that half the US population live in major urban areas.

Kirk, who said he retweeted this map, wants us to stop whining:

I get that GDP is essentially a proxy indicator for where people are living yet I still have a novel interest in learning about the dynamics of the US. I *know* that there is not a uniform distribution of where people live (nowhere on earth has this) but it is still revealing for me to see anything that represents a proxy of this skewed population. I don’t think the map claims to be doing anything different to this so, in that sense, it doesn’t mislead or make false claims.

I will be writing on my other blog about the educational aspect of a chart like this, which is the other prong of Kirk's argument. That last sentence, which I bolded, strikes me as the argument that the true lie is true and therefore is beyond reproach. This is a crucial difference between doing statistics and doing pure math. In statistics, you can't win arguments by invoking the truth... if the truth is knowable, statisticians would all be unemployed.

The map does not make false claims but it leads readers to the conclusion that the orange areas are much more important than the blue region (equal economic activity but much smaller area). The first problem is that the types of economic activities are vastly different between those regions, and this significant factor is ignored.

The second problem is that the designer over-aggregated the data. All counties (or zip codes) are classified into two groups ("split in half") when in fact, the level of economic activity at the level of counties (or zip codes) is a gradient. Imagine plotting the economic activity index by county, ordered from the highest to the lowest. Do we see a dramatic drop-off after counting out half the counties (i.e., the pattern shown on the left chart below)? Or are we more likely to see the pattern shown on the right? If you see a distribution like the one shown on the right, would you summarize that with just two segments?

 ***

Cairo's general point is that good data visualizations require good data analyses. In turn, good data analysis requires numbersense.

***

Chapter 3 of Numbers Rule Your World (link) explores the question of aggregating data, which is central to statistical thinking. Aggregation features throughout Numbersense (link), particularly in Chapter 1 (school rankings), the chapters on economic statistics, and the chapter on fantasy football.

Also, you can learn statistical concepts from me at NYU. New course starting first week of March. More information here.

The post on Junk Charts about this map is here.

 

An example of why data will not end debate

One oft-repeated "self-evident" tenet of Big Data is that data end all debate. Except if you have ever worked for a real company (excluding those ruled by autocrats), and put data on the table, you know that the data do not end anything.

Reader Ben M. sent me to this blog post by Benedict Evans, showing a confusing chart showing how Apple has "passed" Microsoft. Evans used to be a stock analyst before moving to Andreessen Horowitz, a VC (venture capital) business. He has over 25,000 followers on Twitter.

I'll get to this chart later but feel free to tour around the comments area. You will get a feel for the types of conversations that happen when an analyst offers data to a corporate meeting. It turns out that everyone has an opinion about everything, and while data people think the only way to make an argument is by presenting data, there are plenty of other people from other background who use other modes of persuasion, like rhetoric, anecdote, philosophy, and HIPPO (i.e. HIghest Paid Person's Opinion).

The original post plus these comments present a mishmash of metrics that can be used to measure the gap between Microsoft and Apple. Here is the list roughly in chronological order up to the point where the post had 70 comments:

From the original chart, we have PC shipments, iphone and ipod shipments, shipments of all devices running MacOS or iOS (possibly excluding AppleTV), PC shipments against all devices running iOS, PC plus smartphone (excluding tablets and music players) shipments against all devices running iOS.

Then in the comments, people are bringing up: Xboxes (a gaming device running on Windows OS which is almost a PC), Android devices (which is part of the Google universe), profits instead of units, servers, Google, Macs that are running Windows OS, whether it is accurate to say "Apple passes Microsoft" using the blogger's own metric, whether one needs to wait more than one quarter to make such a conclusion, TVs, printers, the difference in technology that resides inside the same device (say the PC), whether the fourth quarter result can be generalized, embedded devices, self-built PCs, video-editors on the Web, and Amazon.

***

Evans responded to many of these comments by complaining that readers are not getting his message. That's an accurate statement, and it has everything to do with the looseness of his data. This reminds me of Gelman's statistical parable. The blogger here is not so much interested in how strong his evidence is but more interested in evangelizing the morale behind the story.

His primary thesis is quite likely correct. There is a huge trend of U.S. consumers spending more time on mobile devices, mostly smartphones and also some tablets. I see this at work, and the trend is widely recognized in the tech world by this time.

But the readers are also right to point out the deficiency of that chart.

***

Using my Junk Charts trifecta checkup, we'd say the failure of the chart is due to the Data corner. The chart addresses an interesting Question, and the Graphical elements are acceptable but the Data just do not do justice to his credible thesis.

This reader for example supports Evans, and blames other readers for an outdated way of thinking about the computer industry:

That's fair, except that Evans is also guilty of old-school thinking when he used unit sales of devices as the metric to compare Apple and Microsoft. What he really needs is data on time spent. This can be obtained from Nielsen or Gallup polls or some such venue. Just count the number of hours the average person spends on iOS devices versus Windows devices - regardless of whether they are phones, PCs, laptops, gaming consoles, music players, or embedded devices. This is sometimes called a "mindshare" metric.

To get even more sophisticated, we should merge mindshare with revenues (or profits) generated. Different companies have different business models. Those that give away services (or hardware) for free (or close to free) can gain mindshare at the expense of revenues (or profits). I like a metric such as dollars generated per hour used.

***

Back to my original point. Doing data analysis is a good first step. The bigger challenge is pushing people with preconceptions to believe the analysis (with all its imperfections and assumptions) and to change their minds.

 

 

Hat tip to the Times editors

While I wasn't happy with how the New York Times business page covered the recent unemployment report (link), I was pleasantly surprised to see this editorial today titled "Making College Pay". Here are the key sentences:

The recent jobless rate for college graduates ages 25 and older was 3.2 percent, and their median pay at full-time, full-year jobs was $75,300 for men and $53,700 for women. That is a far lower rate of joblessness and a far higher pay level than for high school graduates and people without high school diplomas.

But that doesn’t mean that enough good jobs are, or will be, available for college graduates. Though joblessness for college graduates ages 25 and older looks tame, the jobless rate for those under 25 averaged 8.2 percent in 2013, compared with 8 percent in 2012 and 5.4 in 2007, before the Great Recession hit in full force.

That is one of the points I have been making the past few days.  If you want to evaluate whether we should be making more college graduates, it is pointless to look at a metric that combines people who just graduated and people who have been working for forty years, and everyone in between. The only thing that really matters is whether people who are newly graduating are finding jobs. I imagine an economist would call this a "marginal" analysis.

And not just any job... are they finding permanent, full-time jobs that require college education? This is where even these editors went off rails. The U3 unemployment rate of 3.2 percent does not care if the jobs require a college degree; it also counts temp jobs and part-time jobs as employment. This means even the 8.2 percent rate for the under 25 age group may be misleading once we dig into the nature of the jobs being counted as jobs.

Actually, the editors kind of got this message too when later they pointed out research showing "the rise in underemployment for graduates ages 22 to 27 never fully retreated after the recessions of 2001 and 2007-9; in 2012, it was a dismal 44 percent." If they realized the official unemployment rate hides the problem of underemployment, they really should have used a different unemployment metric in the beginning. Other metrics are all reported by the Bureau of Labor Statistics.

Read that last paragraph again, and tell me why we should make more college graduates. And let's not forget that these graduates are saddled with unprecedented debt on graduation.

Here's another gem: "Pay, meanwhile, has stagnated for college-educated workers over the past 12 years."

I wrote previously about why the structural unemployment thesis is bankrupt (link).

All in all, a great NYT editorial that will hopefully change the policy debate for the better.

 

An offer for New York Times Business reporters

Reading the New York Times's coverage of the unemployment report irritates me so much that I'm putting an offer on the table: if they contact me, I'll send them a copy of Numbersense (link) for free so that they can educate themselves about employment statistics (Chapter 6).

***

In the meantime, let me run through all the irritating errors they keep reporting month after month (link to article). By the way, you will be much better guided if you just read the New York Post column by John Crudele on this stuff.

First of all, the Times insists on reporting the most meaningless metric (the so-called U3 unemployment rate) when the Bureau of Labor Statistics publishes several other much more telling statistics every month. In the first paragraph, they highlighted:

The unemployment rate inched down in January to 6.6 percent, the lowest level since October 2008, from 6.7 percent in December.

The very first chart in the accompanying graphic shows a bold black line that declines from a high of 7.6 percent to the current rate of 6.6 percent. (The online version is even more gung-ho, stretching the timeline back to 2011 when the rate was over 9 percent.)

What's the problem, you ask? The U3 unemployment rate paints an incomplete picture of the nation's jobs market. The chart below shows the other side... the green line, which has in constant decline since 2008, is the proportion of population considered part of the "labor force". It turns out we don't compute unemployment rate off the base of population (or working-age population). There is a convoluted set of rules which define somebody as part of the "labor force" and if you are not in the labor force, you cannot be unemployed!

(I don't usually do dual axis charts. In this one, I just want to quickly show you that these two time series show remarkably different trajectories from the start of 2008.)

Just for example, if you graduate from college, you first become part of the labor force, then if you give up on your job search after several months, you will no longer belong to the labor force (so you can't be unemployed). If you took a part-time temporary job for a day, and never worked another day during the month, you can be considered employed (and part of the labor force). In the book, I explain how to interpret numbers in the light of these complex rules.

Most importantly, prior to the regression recession [Ed: Oops, that was a statistical slip of the tongue], the labor force participation rate (green line) was extremely stable fluctuating around 66 percent but it has been on a one-way slide since 2008 with no end in sight. Currently, it's at 63 percent.

***

Another chart worth printing is the following, which shows an alarming rise in the ratio of U6 unemployment versus U3 unemployment. It paints the picture that the nation's employment situation is not as good as advertised. U6 unemployment is more expansive (but also probably more in line with intuition): for example, a so-called "discouraged worker" is considered unemployed in U6 but not in U3.

This ratio would stay put if U3 and U6 are both moving in lock step. However, it is rising meaning that the rate of decline in U3 is faster than the rate of decline in U6. Another reason why focusing on U3 only is a mistake.

***

Later on in the article, the reporter quotes economists as blaming the "wintry" conditions for the poor jobs report in December. Anyone who makes such a statement is someone who did not do his/her homework. The BLS discloses exactly how they count employment. The most important thing to note is that someone is considered employed if that person has worked at least one hour during the "reference week" of the survey. Working is not defined as going to the office. If you work from home or if you called in sick, you are still considered employed.

The aforementioned economists then said something even more inexplicable, which the reporter duly repeated:

Initially, the weak report for December was blamed on wintry conditions that inhibited hiring, but... a second straight month of disappointing job gains led him to conclude that the cold and snow could not be blamed this time.

Where is the logic in this?

***

The end of the article brings yet another one of the great economic fallacies of our time: the claim that the unemployment problem is due to "only one-third of American work force" having a college degree or higher. This is one of these "Big Data" nonsense referring to the correlation that college grads have much lower unemployment rate than non-college grads.

If this reporter cared to look back at his own headline -- that wage growth has stagnated, and "in fact, white-collar workers did a bit worse than blue-collar workers last year in terms of wage growth", then he might realize that jobs are not chasing college grads these days. If having a college degree would magically find one jobs, then there must be too many jobs requiring college degrees and not enough graduates, and we'd expect our white-collar wages to be rising sharply until that inequilibrium is corrected.

***

This stuff is not that hard. It just requires reading the source materials and understanding the details of how jobs are counted. And I have summarized all of that materials into one book chapter. So please do us a favor and read it!

All else is equal except that one is more equal than the other

The mis-use of "all else being equal" in statistics and much of the social sciences is a disease. The key to diagnosing this abuse is to recognize that there are two modes of "all else being equal".

I call it the "design" mode and the "pretend" mode.

An example of using "all else being equal" in the design mode is the classic randomization principle. If we run an A/B test (test of two means or proportions), by virtue of the random assignment, we are assured that groups A and B are statistically indistinguishable except for the known treatment. There is little danger of abuse in this mode. Unless we draw a bad lot, we know all else is truly equal.

Most of the time, though, when a researcher invokes "all else being equal", he or she is using the "pretend" mode. A classic example is a regression analysis run on observational data, otherwise known as "controlling for all other factors". First of all, you can only control for the known factors that you include in the regression equation. More seriously, in the "pretend" mode, all else being equal is an assumption. The effect given for any factor is under the assumption that the unit is average on all the other known factors. When you have observational data, you are unable to verify that all else is truly equal!

So next time someone mentions all else is equal, first figure out if this is used in design or pretend mode. Adjust your expectation accordingly.

Two new reviews of Numbersense

I've learned one thing about book readers. There is a lag between buying a book and reading it. In fact, I imagine an author has two battles to win: one is at the bookstore (or Amazon) getting you to purchase the book; the next is to get you to pull out the book from your shelves and start reading it. I mean, what am I to say? I own shelves of unread books.

Anyway, reviews of Numbersense are trickling in. The following two long reviews make me happy, as these readers get the points I'm trying to get across.

Here's Thinkanator's review (link). He apparently is tired of the heavily-groomed media coverage of Big Data, and wanted the world to know about "Real Big Data," the less glamorous side of the job. He noted: "In the epilogue Kaiser shares one of his own Real Data Science stories and I found myself nodding my head and saying, “Yup, that’s how I spent several days in the last couple of weeks!”

Thinkanator probably wrote a better summary of the book than I have: "Numbersense is a wonderful and accessible book that consists of a series of stories about data that illustrate how to think about the kinds of statistics you read about on a daily basis. The emphasis isn’t mathematical, it’s more about when you should think, “Hmmm… that doesn’t sound right”, when you hear some statistics thrown around."

***

Veronique Frizzell wrote not one but three posts about Numbersense.

Post 1, Post 2, Post 3

In Post 1, she highlights the sections of the book that she enjoys most. In Post 2, she applies the Groupon analysis to her own business. In Post 3, she draws lessons from the chapter on unemployment statistics.

***

Get your copy of Numbersense here.

 

 

MailChimp Gmail study as an example of Big Data studies 2/2

Previously, I analyzed the data analysis by MailChimp on the impact of Gmail's new tabbing feature, and noted a potential data issue (link). In Part 2, I will look at the MailChimp study as a typical example of "Big Data" studies.

The Gmail study has several features that are hallmarks of Big Data. First and foremost, the analyst boasts of a staggering amount of data ("29 billion emails, 4.9 billion opens, 4.2 billion clicks, and 43.5 million unsubscribes"). Because of such "wealth", the analyst now feels empowered to ignore statistical variability. This is sometimes known as "N = All", the illusion that the analyst has collected all possible data points. In my talks, I like to call this "seemingly complete" data, calling it what it is. The assumption of "N=All" is akin to the assumption of "perfect information" or "completeness" in economic models, and we know where that led us.

Take this chart from MailChimp's post:

The difference in click rates ranges from 0.3% to 2.5% among the top 4 email providers. This is against a base rate in the 85% range (see Part 1 for why this base rate is probably miscalculated). So we are talking about tiny percentage differences. The email provider-to-email provider difference is even smaller: for example, it's less than 1% between Gmail and Hotmail.

Now imagine you are tracking the day-to-day changes in click rate for a single email provider (say Gmail, which according to the other chart clocks in at approx. 84%). Do you expect Gmail's click rate to be 84% day after day with no fluctuations? Probably not.

Here's the problem... the "seemingly complete" paradigm is a claim that there is no noise in the data. And that is the basis of the claim that email provider A is worse than email provider B in their click rates.

I see this as the "moral hazard" problem of Big Data. The comfort of abundance causes us to let our guards down; when we should see fewer false results, we end up seeing many more!

***

I will quickly run through a few other ingredients of the typical Big Data study. The data is purely observational, has no control group, and is coopted from other studies with different objectives. Just think about other Big Data studies, and you'll notice that they fit most of these criteria.

The MailChimp analyst, to his credit, attempted to manufacture a control group by appealing to other email providers who do not have the Gmail-style tabs. This is a simple and useful control but readers using numbersense would want to investigate these questions:

• are there differences in the types of people who have Gmail accounts from those who have AOL accounts?
• are there differences in the types of email marketers who spend on Gmail from those who spend on Hotmail?
• During the period under study, did any of the other email providers implement any changes to their email service, other than Gmail introducing the tabs?
• Did email marketers change their behavior on the Gmail platform as they react to the changes put in by Google, which presumably they would not have replicated on other email platforms?
• Did some email users migrate out of Gmail to the other email providers during the period under study?

Any of these factors, and more, can explain any gap in click rates between email providers during the period of the analysis. This is the discontent of not setting up a control group ex-ante. In Big Data, analysts often have to construct control groups after the fact. This is one area I hope to see theoretical advancement.

In the meantime, realize that whenever we create control groups like this, the resulting analysis is heavily laced with assumptions (e.g. Gmail and Hotmail have the same types of users and attract the same types of advertisers, and that none of the other platforms implemented any major releases during the period, and none of the marketers changed their Gmail marketing strategies, etc.)