Big Data, Plainly Spoken (aka Numbers Rule Your World)

Know your data 13: no-tracking technologies

I have been highlighting some cases of analysts doing a masterful job in taking apart claims made in the media. Here is another example from Bruce Schneier, the national security expert.

In June 2013, a Who’s Who list of telecom and technology companies, such as Verizon, AT&T, Google and Facebook, was busted for passing customer data to the National Security Agency in its domestic surveillance program. This episode invites scrutiny on one of the tenets of Big Data, the business practice of tracking customers. One of the articles I read at the time mentioned TrackMeNot as a way to foil the spies. TrackMeNot is a browser plug-in developed at New York University that sends randomly generated search queries from your browser. This strategy has a scientific basis—those who read Nate Silver’s book, The Signal and The Noise, can appreciate how adding noise can diffuse the signal.

Despite the association with a reputable institution and the scientific principle, one would be mistaken to think that TrackMeNot protects one from snooping!

A search of “TrackMeNot” brought me to a blog post from 2006 (!) by Bruce Schneier (link), whose valuable work on security technology I came across while researching Numbers Rule Your World. In fewer than 1,000 words, he turned what sounded like a promising concept into half-baked pseudoscience. The secret is read the fine print, something you should always do before believing statistical claims.

It turns out that the “random” search queries generated by the tool (circa 2006) consisted of two-word combinations, chosen from a dictionary of 1,673 words, sent to one of four search engines at regular twelve-second intervals. Imagine how easy it is for an analyst to remove such noise from the web logs. It is obvious from reading the comments that Schneier’s audience is well informed about the Achilles’ heel of all terrorist prediction methodologies, the problem of false positives. One reader simply said “WakeMeNot.”

Chapters 6 and 7 of Numbersense, on economic indicators, highlight the importance of learning the formulas behind metrics, and how data are processed. In short, details matter.

***

You have till Saturday to enter the book giveaway quiz. Click here to participate.

Some interviews, and an event

Here are two recent profiles of me:

Oracle's Profit: "Beyond Big Data" (link)

ASA News (link)

In the second one, they asked me to give advice to people who are interested in getting into data visualization. Here is my answer:

Keep sketching, and keep trashing.

Settle your story and then find the tools, never the other way around.

Hate the default.

Imagine your audience.

If you don’t have an audience, find one.

Inspect your data.

Have a point of view.

Keep a portfolio.

***

Next Tuesday, I will be speaking at BIG Frontier in Chicago on Big Data and marketing. Free admission. You can register here.

***

The second book giveaway is still open. Try your luck here.

Another chance to win a signed book

The winner of the Book Quiz is S. Vijay, from Austria. Congratulations!

The answers are: (C), (B), (D). No sumo wrestlers in Numbersense (that was Freakonomics). Fewer law schools reported employment rates after graduation. Fifty-five percent of the jobs were full-time, long-term positions requiring a JD.

I did a little geeking on the quiz data on the sister blog (here). But most importantly...

***

If you missed the book giveaway last week, here is another chance. McGraw-Hill is impressed by the turnout and has agreed to another book quiz.

This week's 3-question quiz is based on sample pages from Chapter 7, about inflation statistics. The sample pages are hosted on Slideshare and also as PDF.

To enter the quiz, click here.

Anyone can enter, including those who played last time. International readers are welcome, too.

Please remember to provide your email address so I can contact you for an address if you win.

Final day of Book Quiz

You still have until midnight Pacific Standard Time to enter the book quiz (link). The prize is a signed copy of Numbersense: How to use Big Data to Your Advantage.

The sample pages (from Chapter 1) have been placed on Slideshare in case you don't like or can't read PDFs.

***

Steroids are in the news again, with Tyson Gay, Asafa Powell, etc. testing positive recently. These are top names in track. Reminiscent of Ben Johnson, Marion Jones of earlier. This should come as no surprise; these anti-doping tests are easy to evade and have high false negatives. That's why everyone claims they passed hundreds of tests. Eventually, they run out of luck. Gay claimed he has no excuses then gave an excuse, "I trusted someone I shouldn't have." 

Until we start seeing lots of positive tests, the math is clear that lots of dopers are getting away with doping.

Some previous blogs about steroids testing: here, here and here

 

 

Causal thinking

Andrew Gelman, Columbia professor, wrote an important post about causal thinking (link) that I highly recommend reading. While he approaches the topic from a researcher's perspective, his framing of the issue is very practical, as I will demonstrate in this post.

Gelman's main point is the two modes of causal thinking:

• Forward causality is asking the question, if we change X, how does that change Y? This is typically answered by setting up a scientific experiment, varying X and observing Y while keeping other factors constant. When an experiment is not possible, we try to mimic it.
• Reverse causality is asking the question, if Y happens, what factor(s) X may have caused it? This is a very difficult problem to address. Gelman favors an approach where we first generate hypotheses of what X might be, then verify if the hypotheses hold.  

In the past, forward has been called "effect of causes" while reverse has been called "causes of effect." Those terms are cute but quite confusing so I prefer the forward/reverse language.

As Gelman explained, many statisticians consider reverse causality an unworthy question, unscientific, indeterminable, unserious. First issue is that even if one or more hypotheses hold, it is not clear that the real cause isn't still lurking. But Gelman is right in saying that a lot of real world problems are of the reverse causality type, and it's an embarrassment for us to ignore them.

In this post, I describe two examples of casual reasoning in the real world.

***

Most business problems are reverse causal. Take for example P&G who spends a huge amount of money on marketing and advertising activities. The money is spread out over many vehicles, such as television, radio, newspaper, supermarket coupons, events, emails, display ads, search engine, etc. The key performance metric is sales amount.

If sales amount suddenly drops, then the executives will want to know what caused the drop. This is the classic reverse causality question. Of course, lots of possible hypotheses could be generated... TV ad was not liked, coupons weren't distributed on time, emails suffered a deliverability problem, etc. By a process of elimination, one can drill down to a small set of plausible causes. This is all complex work that gives approximate answers.

The same question can be posed as a forward causal problem. We now start with a list of treatments. We will divide the country into regions, and vary the so-called marketing mix, that is, the distribution of spend across the many vehicles. This generates variations in the spend patterns by vehicle, which allows us to estimate the effect of each of the constituent vehicles on the overall sales performance.

***

Now, consider law enforcement or spying agencies building vast databases. The forward question is asking whether the habit of searching for the word "bomb" in a search engine increases the chance of someone later committing terrorist acts. This predictive question is highly challenging and the problem of false positives is very real.

The reverse problem is after a terrorist strikes, trying to understand why he/she did it. This question is aided by having a massive database of all interactions, and made much, much easier by knowing the identify of the terrorist, after the fact.

***

Something else is going on behind the scenes, which might explain why scientists are more interested in the forward question. It seems to me reverse questions are usually not "statistical" in nature; they are questions of trying to explain something that happened. This is the case in the example of the sales drop as well as the example of traceback investigations.

***

Chapters 2, 4 and 8 of Numbersense explore how data analysts think about causation although causal thinking is everywhere in the book. You can get the book here. Or try your luck in the Book Quiz (link).

How to Consume Big Data

Over at the McGraw-Hill blog, I wrote about how to consume Big Data (link), which is the core theme of my new book. In that piece, I highlight two recent instances in which bloggers demonstrated numbersense in vetting other people's data analyses. (Since the McGraw-Hill link is not working as I'm writing this, I placed a copy of the post here in case you need it.)

Below is a detailed dissection of Zoë Harcombe's work.

***

Eating red meat makes us die sooner! Zoë Harcombe didn’t think so.

In March, 2013, nutritional epidemiologists from Harvard University circulated new research linking red meat consumption with increased risk of death. All major mass media outlets ran the story, with headlines such as “Risks: More Red Meat, More Mortality.” (link) This high-class treatment is typical, given Harvard’s brand, the reputation of the research team, and the pending publication in a peer-reviewed journal. Readers are told that the finding came from large studies with hundreds of thousands of subjects, and that the researchers “controlled for” other potential causes of death

Zoë Harcombe, an author of books on obesity, was one of the readers who did not buy the story. She heard that noise in her head when she reviewed the Harvard study. In a blog post, titled “Red meat & Mortality & the Usual Bad Science,” (link) Harcombe outlined how she determined the research was junk science.

How did Harcombe do this?

Alarm bells rang in her head because she has seen similar studies in which researchers commit what I call “causation creep.”(link)

    

She then reviewed the two studies used by the Harvard researchers, looking especially for the precise definition of meat consumption, the key explanatory variable. She discovered that the data came from dietary questionnaires administered every four years (this meant subjects who didn’t answer this question would have been dropped from the analysis). All subjects were divided into five equal-sized groups (quintiles) based on the amount of red meat consumption. Surprisingly, “unprocessed red meat” included pork, hamburgers, beef wraps, lamb curry and so on. This part is checking off the box; it didn’t reveal anything too worrisome.

Harcombe suspected that the Harvard study does not prove causation but she needed more than just a hunch. She found plenty of ammunition in Table 1 of the paper. There, she learned that the cohort of people who report eating more red meat also report higher levels of unhealthy behaviors, including more smoking, more drinking, and less exercise. For example,

    

The researchers argue that their multivariate regression analysis “controlled for” these other known factors. But Harcombe understands that when effects are confounded, it is almost impossible to disentangle them. For instance, if you're comparing two school districts, and one is in a really rich neighborhood, and the other in a poor neighborhood, then race and income will be confounded and there is no way to know if the difference in educational outcomes is due to income or due to race.

Next, Harcombe looked for data to help her interpret the researchers’ central claim:

Unprocessed and processed red meat intakes were associated with an increased risk of total, CVD, and cancer mortality in men and women in the age-adjusted and fully adjusted models. When treating red meat intake as a continuous variable, the elevated risk of total mortality in the pooled analysis for a 1-serving-per-day increase was 12% for total red meat, 13% for unprocessed red meat, and 20% for processed red meat.

Her first inquiry was about the baseline mortality rate, which was 0.81%. Twenty percent of that is 0.16% so roughly speaking, if you decide to take an extra serving of processed red meat every day, you face a less-than-2-out-of-1000 chance of earlier death. (Whether the earlier death is due to the red meat or just more food consumed each day is another instance of confounding.)

This also raises the issue of error bars. As Gary Taubes explained in his response to the red-meat study (link), serious epidemiologists only pay attention to effects of 300% or higher, acknowledging the limitations of the types of data being analyzed. The 12- or 20-percent effect does not give much confidence.

The researchers are overly confident in the statistical models used to analyze the data, Harcombe soon learned. She was able to find the raw data, allowing her to compare them with the statistically adjusted data. Here is one of her calculations.

    

The five columns represent quintiles of red meat consumption from lowest (Q1) to highest (Q5). The last row (“Multivariate”) is the adjusted death rates with Q1 set to 1.00. The row labelled “Death Rate(Z)” is a simple calculation performed by Harcombe, without adjustment. The key insight is that the shape of Harcombe’s line is U-shaped while the shape of the multivariate line is monotonic increasing.

The purpose of this analysis is not to debunk the research. What Harcombe did here is delineating where the data end, and where the model assumptions take over. One of the themes in Numbersense is that every analysis combines data with theory. Knowing which is which is half the battle.

At the end of Harcombe’s piece, she checked the incentives of the researchers.

Harcombe did really impressive work here, and her blog post is highly instructive of how to analyze data analysis. Chapter 2 of Numbersense looks at the quality of data analyses of the obesity crisis.

***

Reminder: You can win a copy of my new book. See here for details.

 

How to Consume Big Data (from McGraw-Hill blog)

When you hear about Big Data, you almost always hear about the supply side: Behold the data in un-pronounceable units of bytes! Admire the new science inspired by all the data! Missing from this narrative is the consumption side. A direct consequence of Big Data will be the explosion of data analyses—there will be more people producing more data analyses more quickly. This will be a world of confusing and contradictory findings.

In my new book, Numbersense, I argue that the ability to analyze and interpret these data analyses will give one a competitive edge in this world of Big Data.

Numbersense is the noise you hear in your head when you see bad data or bad analysis. After years of managing teams of data analysts, I’ve learned that what distinguishes the best from the merely good is not math degrees or computer skills; it is numbersense.

Numbersense is an intangible quality that you can’t teach in a classroom. The best way to pick it up is by learning from people who have it. For this blog post, I selected two great analyses of data analyses that have impressed me recently. These are highly instructive examples.

***

Eating red meat makes us die sooner! Zoë Harcombe didn’t think so.

In March, 2013, nutritional epidemiologists from Harvard University circulated new research linking red meat consumption with increased risk of death. All major mass media outlets ran the story, with headlines such as “Risks: More Red Meat, More Mortality.” (link) This high-class treatment is typical, given Harvard’s brand, the reputation of the research team, and the pending publication in a peer-reviewed journal. Readers are told that the finding came from large studies with hundreds of thousands of subjects, and that the researchers “controlled for” other potential causes of death.

Zoë Harcombe, an author of books on obesity, was one of the readers who did not buy the story. She heard that noise in her head when she reviewed the Harvard study. In a blog post, titled “Red meat & Mortality & the Usual Bad Science,” (link) Harcombe outlined how she determined the research was junk science.

She knows this type of research methodology rarely if ever delivers conclusive evidence of causation. Then, she found support from a data table included in the research paper. The table shows that the cohort of people who report eating more red meat also report higher levels of unhealthy behaviors, including more smoking, more drinking, and less exercise. Thus, the increased risk of death observed in the study could have been explained by factors other than red meat consumption.

For a full dissection of Harcome’s amazing post, please click here. Chapter 2 of Numbersense looks at the quality of data analyses of the obesity crisis.

 ***

Netflix turns viewers into puppets! Felix Salmon declined to become one.

In February, 2013, Netflix, ever the media darling, premiered House of Cards, a re-make of the successful British television show, their second foray into producing original content for its tens of millions of subscribers. Netflix executives regaled the press with stories of how Big Data analysis took the risk out of their $100 million decision.

Andrew Leonard, the technology reporter for Salon.com, gobbled up the Netflix story, even interpreting it as a “symptom of a society-wide shift.” (link) Like other news analysts, Leonard was convinced by the “pure geek wizardry” used to analyze mountains of data collected from Netflix customers. The machine, we’re told, decided that David Fincher should be the director and Kevin Spacey, the star. From here, it is a short trip to the lala land of viewers as puppets with machines as the overlord.

This analysis aroused the skeptic in Felix Salmon, the finance blogger for Reuters. In his blog post, “Why the Quants Won’t Take Over Hollywood,” (link) Salmon raised other factors that affect the box office, including billions spent on marketing and publicity, the quality of the writing, the sociopolitical climate, the complex relationship between originals and remakes, and the poor track record of predictive modeling in Hollywood. On this last point, Salmon exhibits a keen sense of the limitations of science, speaking of “impossible-to-formulate cocktail of creativity, inspiration, teamwork, and luck.”

Chapters 4 and 5 of Numbersense explains how you should judge predictive models used by marketers.

***

When their respective blog posts surfaced, Harcombe and Salmon were lone voices vetting carefully the claims based on other people’s data analyses. Their well-honed numbersense allows them to stand firm in the face of mountains of data, worship of high science, formidable-sounding technical jargon, and academic reputations. The problems with the original research are far from obvious. The point is not to debunk these studies—no data analysis is ever infallible—but to figure out for yourself what is credible, and what is junk.

Win an Autographed Book

This is cross-posted on my two blogs.

For my fans on either of my two blogs, I'm giving away a free signed copy of my new book, Numbersense.

All you have to do is to answer 3 questions, based on a few sample pages (see the PDF here; also on Slideshare). Click on the quiz to enter.

The contest is open until Friday, July 19, 2013 (11:59 PM PST).

This is an open anything quiz, although as I like to tell my students, if you are working too hard, you're probably missing the point.

***

I will draw the winner among those who answer all 3 questions correctly.

Please provide a valid email address so I can get in touch with you to send you the book. 

 

Welcome Message

Big Data is a noisy marketplace of people peddling their data analyses. It’s a confusing world of claims and counterclaims. You need statistical thinking and numbersense to figure out what and who to believe. As a long-time data geek, I show you how I do it.

Follow me on Twitter. Subscribe to my RSS feeds (Big Data Plainly Spoken, Junk Charts). Learn more about my two books (Amazon, Barnes & Noble).

If you’re new here, you’ll find some recurring themes. Researchers trick us with “story time,” as they drift effortlessly from data to theory/assumptions. “Causation creep” is a kind of bait-and-switch, in which researchers first admit they have evidence of correlation only but later give advice based on a causal interpretation. “Know your data” is an ongoing series of posts about the tracking and profiling operations of businesses.

Some of my most popular posts are about the three hours in the life of a data scientist, Facebook and web analytics, Grouponomics, statistics of steroids testing, and my review of Nate Silver's book.

I’m a fan of the apprenticeship model of learning. On the left column, you will find other bloggers that I read regularly. Also check out the people I follow on Twitter.

You heard it here first: statistics is fun!

Announcing New Book: Numbersense

I have a new book arriving at stores this week. It’s titled Numbersense: How to Use Big Data to Your Advantage.

If you read this blog, you’d have a good idea what the book is about. I analyze claims made in the media that are supported by analyses of data. I show you how I dissect these claims to decide whether they are credible, or they are bogus.

The ability to analyze and interpret data analyses will be a critical skill in the world of Big Data. So far, the conversation around Big Data is focused around the collection and processing of mountains of data. The real challenge of Big Data is the proliferation of data analyses: it will be a confusing world of claims and counterclaims.

The book is arranged topically, with two chapters on social data, three on marketing use of data, two on economic indicators, and one on pop culture. I also include two vignettes on what “data scientists” do, one of which first appeared on this blog.

Numbersense is available on Kindle, and in stores. In addition to endorsements by Tom Davenport, Avanish Kaushik, John Sall, and Eric Siegel, a few early reviews have appeared: Mike Kruger... Franz Dill...  Andrew Gelman... Student of Parables... Eustacia Tan... Nick

Please tell your friends and colleagues about Numbersense. You can really help by writing a review on Amazon.

Enjoy!

Kaiser

 

PS. With the new book, I have relaunched the book blog, with the name "Big Data, Plainly Spoken." The links remain unchanged.