Big Data, Plainly Spoken (aka Numbers Rule Your World)

What is Numbersense about?

Michael Elves interviews me for UMFM, the University of Manitoba radio station. We chatted at length about pretty much all the different chapters of Numbersense. He was very interested in the fantasy football materials since the draft is happening this week. You can hear the interview at 11:30 am CST on Thursday Aug 29. Michael tells me a recording will be made available later.

***

JMP will be giving away copies of Numbersense. In this interview, they asked me what problems statistics can't solve. Read my answer here.

 

 

These journal editors probably forgot to drink their morning coffee

Here we go again. Another useless study published in a peer-reviewed journal (Mayo Clinic Proceedings) with a relatively high impact factor and promoted as "Breaking News from the Editor" to the press who then attached a sensational headline and reported it as "science".

This is what caught my eye: "Drinking more than 28 cups of coffee a week may be harmful for people younger than 55, according to a study." I saw this on USA Today (link)... but many other outlets also carried the story, including NPR, CBS, AJC, Guardian, etc.

Just the headline should sound alarm bells. Why 28 cups a week? Is it ok to have 27.5 cups a week? Why 55 years old? On your 55th birthday, should you stop worrying about the number of cups?

Of course, even that speculation betrays a causation creep. Nothing in this study, nor any study of this type, can prove a causal link between the accused food and harm.

Now let's check out the summary by the authors:

Conclusion: In this large cohort, a positive association between coffee consumption and all-cause mortality was observed in men and in men and women younger than 55 years. On the basis of these findings, it seems appropriate to suggest that younger people avoid heavy coffee consumption (ie, averaging >4 cups per day). However, this finding should be assessed in future studies of other populations.

Note that construction: "a positive association between .. and .. was observed ... on the basis of these findings, it seems appropriate to suggest that". All those weasel words. What they really mean is "we have data showing a positive association and we make an assumption that correlation = causation, therefore you should ..."

Do you think you'll learn anything in the journal paper about the biological or chemical mechanism by which coffee causes death? Take a guess.

***

USA Today commits the other sin of health reporting: failure to explain the level of harm, and the context to interpreting such. USA Today tells its readers:

Men younger than 55 who drank more than 28 cups of coffee a week (four cups a day) were 56% more likely to have died from any cause.

56% compared to what? Turns out it's compared to men younger than 55 who do not drink coffee at all. There is a wide gap between drinking over 28 cups a week and drinking zero.

What's also missing is the error bar. According to the paper, the 95% interval is 30% to 87%. Not kidding, it's 25% in each direction.

Absent is the context for understanding what 56% means. How many additional deaths for every 10,000 heavy coffee drinkers? Amusingly, you can't figure this out even after reading the entire paper. The authors got away with presenting data in aggregate (33,900 males, 2198 male deaths, etc.) without showing age group breakouts (Where were the editors??) Stymied, I glance at their other result, the one for "all" men.

In men, those who drank more than 28 cups of coffee weekly had a 21% higher risk of dying compared with their non-coffee-consuming peers.

By the way, the error bar on this result is 4% to 40%. Now, I can't interpret this result either. The baseline death rate in the study was 6.48%. Nowhere in the paper does it break out the number of deaths by the level of coffee drinking. There is no way to know how many of those 2,198 male deaths were men who did not drink coffee at all.

While the important data on the outcomes being analyzed are not published, the authors of the paper regale us with numbers such as the N=11 people who were excluded from the population because they had a history of stroke.

So I ask again: where were the editors? how did they miss this?

***

Readers should also check out Andrew Gelman's recent rant about the excessive reliance on statistical significance, and the neglect of data quality in journal editing (link). It's not that hard to draw up a list of data required for publication.

***

I may write another post about the other issues with the analysis. But here is the most important one in case I don't get to it. This is the chart that supports their finding about men under the age of 55 who drink more than 28 cups of coffee a week. They want us to look at the blue bar on the far right of the chart.

Look at the other blue bars. Need I say more?

 

 

Round up of book news

Here are links to coverage of Numbersense.

David Walker and I debate whether statisticians should be a part of Big Data in Significance. (link - requires registration). Significance is a joint publication of the Royal Statistical Society and the American Statistical Association, and a great source of applied statistics. Walker fears Big Data is pure froth while I outline several problems in which statisticians can play a key role.

Princeton Alumni Weekly featured Numbersense on its book blog

Joe Dager (Business 901) has posted a two-part podcast: Part 1, Part 2

The recording of my webinar on Big Data and Marketing is here.

For those shopping at Barnes and Noble, Numbersense is inexplicably and incorrectly classified as "business reference". I have had no luck so far getting either McGraw-Hill or B&N to classify the book properly.

 

 

An ignored issue in Big Data analysis

The Sunday Review section of New York Times on August 11 contains two pieces I'd like to discuss.

The first piece, by Ian Urbina, an investigative reporter, is called "I Flirt and Tweet. Follow Me at #Socialbot". He tells a fascinating story about how programmers create "bots" that impersonate people surfing the Web. For instance, researchers at a Brazilian university created a bot named "Carina Santos" who was rated by certain social media ranking companies as being more influential than Oprah Winfrey.

Urbina reports that:

Last year, the number of Twitter accounts topped 500 million. Some researchers estimate that only 35 percent of the average Twitter user's followers are real people. In fact, more than half of Internet traffic already comes from nonhuman sources like bots or other types of algorithms.

Somehow Urbina missed the implication of these bots. A huge part of Big Data is internet traffic, including tweets. Recall the Library of Congress made a big deal about saving every tweet ever produced (link). What does this say about the use of tweets to analyze "human" behavior and psychology? Would you ever believe a study that does not adjust for bot traffic?

***

Ironically, readers have the answer in the same section of the paper. There is another article, titled "Dr. Google Will See You Now", penned by Seth Stephens-Davidowitz, a recent Harvard PhD in economics. This article takes Google search data and draws conclusions about "depression" in America.

I could barely get past these words:

Not every health-related search using "depression" is a sign that someone is depressed, and not everyone who is depressed queries Google. But thanks to the incredibly large sample size, meaningful patterns emerge.

This has the same symptom as "causation creep," in which the researcher admits he/she doesn't have evidence of causation but immediately ignores that inconvenient fact, and acts as if he/she has proved causation. Here, Stephens-Davidowitz goes on and assumes that every health-related search using "depression" is a sign that someone is depressed, and that everyone who is depressed queries Google.

It's a sad state of academia if a PhD at Harvard thinks that an "incredibly large sample size" is a cure for badly conceived analysis.

 

Numbersense Pros: Interview with Dean Baker

Today's Numbersense Pros features Dean Baker, an economist and co-director of the Center for Economic and Policy Research. His blog, Beat the Press, is my first stop for economics news. What attracts me is his data-first attitude. Every post on his site is about interpreting economic data--there is none of the data-free theorizing that is typical on other economics blogs.

Baker adopts a cynical, almost satirical, tone, which is fun for readers but perhaps unamusing to the targets of his scorn. To give you a flavor, one of his recent posts is titled "Larry Summers Thought the Stock Bubble was Cool and Missed the Housing Bubble".  Another post on how to fix unemployment begins with "there are two types of people in the world: those who make complicated things simple, and those who make simple things complicated."

 Here is the interview.

KF: How did you pick up your impressive statistical reasoning skills? What sets you apart from other economists?

 
DB:

I don't know that my skills are so impressive, but insofar as I do things differently from other economists I think it is the result of thinking more about the data, meaning how it is gathered and its underlying patterns, rather than trying to beat it up with complex statistical analysis. Certainly our econometric techniques have improved substantially over the last 2-3 decades, but I think it will be a rare that a new technique will allow us to recognize a trend or relationship that we couldn't see with more simple tools.

***

KF: What is your pet peeve with published data interpretations?

DB:

People often fail to consider the most basic issues about the data they are looking at. I recently saw a news article commenting on how the Case-Shiller data released in July showed that the housing market was not being hurt by the jump in interest rates at the end of May and beginning of June. In fact, the Case-Shiller data released in July told us nothing about the housing market in June. The data was a 3-month average for the period between March and May. Furthermore, the data is based on closings. Since it typically takes 4-8 weeks between a signed contract and a closing, most of the sales that went into the May data were contracted in the period from January to April.

There are many other cases I could cite like this in which people analyzing the data have not even thought for a minute about the nature of the data they are looking at. It really helps to know what the data are.

***

KF: Which source(s) do you turn to for reliable data analysis?

DB:

That one is not easy. There are plenty of economists whose work I respect and when I see them write on a topic I can be confident that they have serious things to say, but in terms of regular analysis of the economy, I can't say that there is anyone who I think is especially good.

 

KF: Thank you. You are indeed one of the few people who really know the economic data. We are very lucky that you blog.

 

***

Other Posts in this series:

Interview with Felix Salmon, finance blogger at Reuters

Interview with Avinash Kaushik, Web analytics guru

The case method and statistics

Readers of my books will not be surprised to hear that I am a fan of the case method in teaching. The case method was pioneered by Harvard Law School in 1870, and I was exposed to it at Harvard Business School, where it is used to teach everything from marketing to leadership, and from economics to operations, and even accounting.

Since the 1920s, the case method replaced the "lecture and drill" style of teaching at HBS. It is a radical approach which requires students to think deeply about the case materials, engage one another in constructive debate, and draw general knowledge out of specific examples without getting drowned in the minutiae. It is radical in that the professors do not present any slides or lectures at all.

Let me describe a typical class. Case materials are distributed in advance of the class. Students are responsible for reading the case, and preparing detailed responses to some open-ended questions based on the case. The most important question is always "what would you do if you are faced with this business decision?" The case materials are presented as a narrative, often divided into a sequence of multiple stories, each one ending with a key moment in the company's history. The goal is to simulate the decision-making environment of the protagonist of the case. This means irrelevant data is included. A proper response starts with an "action plan" describing the student's decision, and what follows is the argument marshalling the facts of the case, any analysis of the data, and attention to specific details of the company, the culture, or the environment.

The teacher enters the classroom of 70 to 90 students, and alerts one that he or she will lead off the discussion. The teacher acts primarily as traffic cop during the class, encouraging students to debate each other, occasionally nudging the direction of the dialogue, but takes care not to present a "correct" answer. The teacher uses the blackboard to organize the key ideas as they are raised by students. If a key idea never surfaces, it will remain untouched; it is considered bad form for the teacher to stop the flow of conversation and introduce a foreign concept. A minority of teachers reserve the last five minutes for a summary--but the summary is almost always of the issues, the conflicts, rather than a clean description of the "correct" course of action. Sometimes, it is just a review of what the protagonist in the case actually did, and whether the actions taken panned out.

***

The case method is controversial. Students who hate it typically complain that they "learned nothing." They are looking for a bullet-point summary of the day's learning. They believe in the black-and-white view of the world: there is a "right" way to do something. They show little patience for uncertainty.

Others complain about the inefficiency of the case method. As you can imagine, there are long stretches of time in which the class is collectively confused.

***

I like the case method, and I think it is particularly suited to teaching statistics.

One of the key philosophies of the case-based teaching is that the business environment is filled with uncertainty, with incomplete knowledge, and therefore there is no single right way of doing things. That sounds like a typical statistical problem!

The case method also acknowledges that general principles are useless in running a business unless they are adapted to specific situations. Business managers must adapt to changing circumstances and cope with uncertainty.

***

In writing my two books, I have taken half of the case approach--the use of case studies, but also adapted it to the printed form. The Socratic method is missing, and I've included a few takeaways. I believe strongly that you can't reducing managing a business to a cookbook, nor can you provide detailed recipes to solve business analytics problems.

The case method demands a lot from users, and that's why 50% of grades at HBS are determined by class participation. You'll get the most out of Numbersense if you engage with the materials actively, discuss them with other readers, figure out other ways to interpret the data, and look for general principles.

***
A good article about how several Harvard professional schools implemented the case method is here. What strikes me are the different ways of expressing the same philosophy. Forbes has a nice piece last year with an inappropriate title.

 

A letter to reporters on the economy

The New York Times took over 1,000 words to tell us that Big Data won't change the economy (or is it the economists' profession?) ("Is Big Data an economic Big Dud?") I'm less pessimistic; I think the collection of vast troves of observational data is ultimately beneficial but only if (a) we set a high bar for analytics, such as requiring multiple corroborating data sources pointing to the same conclusion; (b) we initiate efforts to collect specific data through well-designed surveys or experiments with the goal of providing a degree of quality control; and (c) we advance the methodologies of analyzing observational data.

The bar is not high, given the woeful state of research on the economy, exposed by the Great Recession. Even worse are the "consensus forecasts" issued about business performance. However, don't for a second fall into the trap of thinking that more data automatically leads to better research. More data does not automatically lift that low bar; quite the contrary, without meeting the preconditions listed above, more data could push that bar even lower than it is now. That is one of the central concerns of Numbersense.

***

Now, reporters take note. Please don't repeat the following fallacies that pollute most articles written about Big Data.

The NYT starts by telling us how fast the volume of data is growing: "The astounding rate of growth would make any parent proud." Give me a break. The simplest way to generate tons of data is to introduce bugs and mistakes into code. I will give you two examples which count as "data" generation.

First, when you import data to a data base, a log file is produced containing entries like "Import started.", "Read 50,000 entries." "Warning: Found date in non-standard format", etc. plus overhead entries on the time of execution, the name of the software, etc. Such log files are regarded as data, specifically "unstructured" data.

You can use say an ODBC connection to import an external file but in many such arrangements, the database only processes a small number of rows per instruction, say 500 rows. So, if your data comprises 20 million rows, the import requires 40,000 instructions for one upload. Even without errors or warnings, you've generated 1.2 to 2 million rows in the log file. If you have a column of "non-standard dates", you've generated at least 20 million rows of warnings. Your log file now has more rows than your data. You've truly contributed to the growth of data.

The second example is really an analogy. i am reminded of college days when I accidentally printed machine code in computer lab. The human-readable code might be 50 pages long but one typo and the printer started churning out thousands of pages of gibberish. We did the sensible thing in those days--we threw out the gibberish.

***

The second fallacy is encapsulated in this statement: "the economy is, at best, in the doldrums and has stayed there during the latest surge in Web traffic." Behind this kind of blather is the presumption of a simple, direct link between data and "the economy".

Maturity in data-analytic thinking starts with complex, multivariate, stochastic and dynamic systems. There will never be F=ma or E = mc^2 for the "economy". What does the "economy" mean anyway? Is it GDP growth? Is it employment level? Is it inflation? Is it happiness? Is it social wellbeing?

Such statements show a shallow appreciation for the factors leading to the economic collapse. Chief among these is the housing bubble, which was promoted by bad public policy, and in many cases, blatant fraud. Data cannot solve such problems; we need political will, ethics, and enforcement of the law. In fact, as I argue in Numbersense, data worsens these problems as more data allows more theorists to cite "evidence".

***

The third fallacy is causation creep, something I discuss here frequently. This involves first acknowledging correlation is not causation, and then slipping in the causal interpretation anyway. For example, this sentence contradicts itself: "The overall economic trends are complex, but an argument could be made that the slowdown began around 2005 — just when Big Data began to make its appearance." If you believe the antecedent, then you have to conclude that the second clause is too simplistic; conversely, if you believe the second clause, then the antecedent is falsified.

***

The point of Chapters 6 and 7 (on economic data) in Numbersense is to raise awareness of the complexities of developing metrics for something as amorphous as the "economy". One then realizes the challenges of interpreting such metrics. To get the full argument, get a copy of the book today.

 

 

 

Numbersense Pros: Avinash Kaushik

Today, I have the honor of interviewing Avinash Kaushik, author of the bible known as Web Analytics 2.0, and a digital marketing evangelist at Google. He also has a must-read blog called Occam's Razor. Occam's Razor is a principle championed by statisticians that can be summarized as "as simple as possible but not too simple". It is a principle and therefore it also draws controversy from some quarters. Kaushik's blog is a place where you can pick up Numbersense. He writes about Web data, which is a huge part of Big Data.

***

KF: How did you pick up your impressive statistical reasoning skills?

AK:

I'm a Mechanical Engineer with a MBA in Finance and Marketing. Somewhere along the way, data, analysis, statistical reasoning became an integral part of my life. :)  Beyond what our traditional education blesses us with... using new analytical software has been an opportunity to refresh old concepts and learn new ones. I'm also partial to books. I'm attaching a photo of one that's on my desk right now!

I've also come to believe that application of cool, sexy, statistical techniques yields poor results not because someone was unfamiliar with them, rather because they don't quite understand the underlying business drivers, data collection mechanisms, impact of the business culture, the people involved and other "soft factors."  I've attempted to invest an enormous amount of time in all of those areas to improve the impact of my analytical efforts.

***

KF: What are your pet peeves with published data interpretations?

AK:  Ignorance of the simplest rules. 

I get mad, still, when I see the classic correlation = causation. My expectation was that after that problem has been made fun of in Dilbert comics, we've reached a level where no one would fall for this trap. And yet, every day!

Not stating the assumptions clearly or, worse, not understanding their impact is another cardinal sin that I've come to dislike a lot. Assumptions are everything! Dig, dig, dig, find them and understand their impact on the data you are looking at.

Perhaps my favourite is the utter unwillingness to actually think about the data in front of them. For example, a prestigious digital company shared the other day that "67% of US buyers check the social profile of the company before they buy the product." Now that is complete crap. You just have to think about all the people making purchases, or yourself, and how often do they/you actually check a company's social presence before making the purchase? Yet this fact was provided with a pretty infographic, and reposted on many "popular tech/social blogs." So few people bring their nose close to the data to check if it smells. It is distressing.

***

KF: Which sources do you turn to for reliable data analysis?

AK:

(This is 100% genuine, and not an attempt to play to the gallery...) For years and years the posts on this blog, Junk Charts, and the comments have been a valuable source of learning for me.

I'm a big fan of the Guardian's Data Store (link). Such a great collection of data visualizations, types of data, thought provoking analysis.

I also use information aesthetics (link), as a gateway to discovering new sources.

Attribution in online marketing: a Big Data problem

Avinash Kaushik's masterful post on the "mutli-channel attribution problem" in Web analytics is required reading for anyone seeking an understanding of what Big Data is really about. Kaushik's posts are marathons; I provide here a little background, plus some highlights from his post to save you some time. But you absolutely should read the whole thing!

I will start from the elementary. Big Data is big because the Internet was set up in such a way that there are extensive log files (Weblogs) documenting details of how information flows from point A to point B. When you click on a link in your browser, it sends a bunch of requests to a remote server, asking for the elements (images, text, buttons, etc.) that form a Web page. The remote server keeps a log of all such requests from all browsers from everywhere around the world. One of these requests goes to the Facebook server (this is why Facebook blankets huge numbers of pages with their Like buttons.) The Facebook server then has a record of what browsers asked for what pages, in other words, Facebook knows you navigated to that page. Something called a cookie is involved here; the cookie identifies your browser.

Let's say you buy flowers at FTD.com. The retailer uses Google Analytics to track user movement on its website (This is a hypothetical example. I don't know what tools they use.) This means that some computer code is inserted into the web pages at FTD.com, and each time your browser requests a page, a call is made to the Google Analytics server with your cookie information.

The point is that lots of data reside in these Weblogs. They tell when you visited which Web page. Eventually, you may end up buying flowers - that is known as an "action". If you purchased flowers, the FTD.com marketing team wants to understand how you arrived at that decision. They assume that the path by which you ended up on the purchase web page provides hints as to why you bought flowers that day. Note I used the words "assume" and then "hints".

Here is an example of a path analysis:

Take a look at the first row, which represents all the recorded interactions that took place prior to a user making the purchase. Read from right to left and you are reading time backwards from more recent interactions. The "action" of purchasing is off the chart on the right side. The five interactions immediately preceding the purchase are classified as "direct", which basically means we don't know how the user arrived at the site. The next, earlier interaction was traced to "organic search" meaning someone came to the website from (most likely) Google, and (most likely) searching directly for our company name. Looking further back, we also see a social network generating leads, and six more unknown interactions.

Marketers stare at this diagram and impose a causal explanation on it. Every interaction is assumed to be a contributor to the final purchase, albeit with different levels of importance. (In Chapter 3 of Numbersense, I caution that such a path is not causal. To see this, think counterfactually. If for example you remove Social Network spending, so that the purple arrows disappear, would the purchase still have occurred? If yes, the Social Network interaction had zero causal effect.)

The concept of "attribution" is to distribute credit among one or more of these prior interactions. Kaushik walks through a bunch of models that can be used to divide credit.

Kaushik then points out that analyzing the above figure (especially when it has thousands of other rows) is a waste of time: "There are too many paths, and you can't actually control the path that a potential customer can take."

***

While Kaushik dives into the mechanics, here are some high-level takeaways from his post:

• The attribution problem has no intrinsic answer. There is no single correct answer. Everything is subjective.
• Many decisions affect the attribution outcomes. e.g. which sources are credited and which are not, which positions in the path are privileged and which are not, the time window for eligiblility, what counts as an "action" and what doesn't. Different decisions lead to different attribution schemes.
• Having more data creates more complexity but does not reduce subjectivity. On the contrary, more data creates more levers resulting in more assumptions.

See Kaushik's full post here.

 

PS. Let me make a small clarification. I'm not against subjectivity. I'm saying every attribution scheme is subjective. One can spend time arguing over the details of one scheme versus another but the new scheme is also subjective. See the section on "Perversion of Measurement" in Chapter 2 of Numbersense for further discussion of this important point.

 

Baseball's Steroids Problem Won't Go Away

Steroids continue to plague professional sports. The latest name to fall is Alex Rodriguez, the shortstop/3rd baseball superstar who currently plays for the Yankees. It wasn't long ago that he was considered a "good guy" of the sport. Now, he's a pariah.

In the rush to make Rodriguez the villain, the media continues to miss these two important aspects to the steroids story:

• Anti-doping tests have a huge false-negative problem. I have been talking about this for years. This topic is  examined in Chapter 4 of Numbers Rule Your World. Rodriguez never failed any test; he got caught because one of his enablers ratted him (and others) out. His supplier betrayed him because he didn't pay up. All the other suspended players also were ensnared during the investigation of a Florida clinic.

• Statistics is powerful yet limited. Based on looking at the statistical properties of screening tests, I can be sure that lots of dopers are undetected. Under some assumptions, I estimate that 80% of dopers escape unscathed. However, statistics is hopeless at pinpointing which particular players are the dopers. A similar situation arises in Chapter 5 of Numbers Rule Your World (link) regarding cheaters in the Canadian lotteries; it's not hard to show that there are cheaters but it's hard to figure out who. The solution is to use the police or investigators to supplement the statistics. This is exactly how Rodriguez and the Balco crew (before him) got caught.

***

There is a third story that is completely ignored by the sports media. Ryan Braun was the 2011 MVP who also patronized the same clinic. Braun, unlike Rodriguez, failed a drug test in 2012. He vigorously defended his innocence (like others), and loudly attacked the messenger. Here is one of his now-infamous quotes:

"It hasn’t been easy. Lots of times I wanted to come out and tell the entire story, attack everybody like I’ve been attacked. My name was dragged through the mud. But at the end of the day I recognized what was best for the game of baseball.”

There are a bunch of other quotes here. Braun got off the same way Lance Armstrong got off years ago when he tested positive... by an expensive legal team and a minor technicality.

In Chapter 4 of NRYW (link), I argue that on chemistry alone, the anti-doping tests are configured such that the positive predictive value is very high, meaning if one tests positive, there is a high chance the subject is a doper. The reverse is not true: if someone is a doper, there is actually a low chance the test would come back positive. These two outcomes are linked and reflect the tradeoff of drug testing.

The Braun story proves again that the positive steroids finding is very informative. Yet again, we learn that drug testing doesn't have a false positive problem. There is a false negative problem.