Big Data, Plainly Spoken (aka Numbers Rule Your World)

Know your data 14: they (may) know things about you

Note: Act quickly. Looks like you can still get a free book courtesy of SAS from here.

***

The New York Times features Acxiom, one of several data vendors that purportedly know a lot about you and me. Other key names in this sector include Experian and Equifax. What's new is that Acxiom will allow consumers to proactively "correct errors", or at least learn what is being bought and sold behind our backs.

I have dealt with all three of these companies. They are pretty much the same business. Here are some points to note about this news:

1. AboutTheData.com only deals with marketing data. Acxiom sells data to all kinds of parties, like credit card companies, insurance companies, and presumably government agencies. You cannot edit those data. The article gives examples of data that won't be made available: whether a person is a “potential inheritor” or an “adult with senior parent,” or whether a household has a “diabetic focus” or “senior needs.”

2. Marketing data are mostly harmless. When they get it wrong, you may have different kinds of offers or marketing materials sent to you, or you may receive a different price on a good than intended. The data that are sold to insurers and government entities are likely to impact your life more heavily but they are not being exposed in this program.

3. You are allowed to edit your data but the option to delete your profile is not available. Opting out of sharing is not the same as deleting the data.

4. Just like every report I have seen about "correcting data", the reporter just assumes that the only things being edited are "errors". Not true. Since the data represent you to external parties, there is an incentive to create a fictional self. This type of error correction measure is very likely to cause even more distortion in the data. It encourages gaming. I last wrote about this issue here.

5. At the individual level, much of this data is guesswork. In technical terms, they are "modeled". The data provider has actual data on a sample of people, and then try to guess what everyone is like. Even something as basic as gender might be guessed. Your name is used as a clue, so is your neighborhood and other things like religion. If you have an unusual name, the guess can be wrong. This type of data is valuable when used with some level of aggregation; it is not to be trusted at the individual level.

Lunch and talk on Wednesday

Note: This is a cross-post from Junk Charts.

I will be the luncheon speaker at INFORMS NYC on Wednesday in NYC. The talk will provide some context for my new book Numbersense (link), and discuss a few examples from the book. You can pre-register here.

INFORMS is the professional society for Operations Research and Management Science people. For some years, I have attended these regularly and learned a lot from other industry speakers.

If you decide at the last minute, you can pay the $5 extra fee on the day of the talk. Or register now.

***

Junk Charts is featured in an article in Harvard Business Review about data visualization. A few new reviews have appeared: CFA Institute,  Flagstaff Business News.

***

I maintain a list of future events on this blog. Look to the right column.

Numbersense Pros: Interview with Andrew Gelman

Professor Andrew Gelman is a pioneer in statistics blogging. His blog is one of my regular reads, a mixture of theoretical pieces, applied work, psychological musing, rants about unethical academics, advocacy of statistical graphics, and commentary on literature. He's one of the few statisticians who gets opinion pieces published in the New York Times. His expertise is statistics in politics, but I also enjoy his work on the stop-and-frisk policies of NYPD, debunking of evolutionary psychology and ESP reseaerch, etc. (I also recommend our collaborative piece on the Freakonomics franchise.) He is a co-author of Bayesian Data Analysis, one of the most influential textbooks on Bayesian statistics; the fourth third edition is out now soon (link).

Here is my interview with Andrew.

***

KF: Andrew, you have impeccable credentials, degrees from MIT and Harvard, Professor at Columbia, Fellow of ASA and IMS, etc., but in my experience, having degrees doesn't automatically prepare one to do great applied data work like you have done. What's your secret?

AG:

Here's one thing regarding "great applied work":  Ask yourself the question:  What makes a statistician look like a hero?  You might think that the answer would be, Extracting a small faint signal from noise.  But I don't think so.  I think that a statistician looks like a hero by studying large effects.  

Statisticians have been studying ESP for decades, trying to tease out tiny signals amid masses of noise, and they just look like chumps.  But in the projects I've worked on that have been successful, we've been aiming at big fat targets--things like incumbency in elections, or the effects of redistricting, or predicting home radon levels (not such a hard task; radon levels vary a lot by geography), or measuring the number of friends people have, etc etc etc.  In these problems, my statistical successes have often come from methods that have allowed the combination of information from different sources.  Often what is important about a statistical method work is not what it does with the data, but rather what data it uses.  Good methods have the flexibility to incorporate more information into the analysis.

I've picked up skills over the years, and I'd definitely say I'm better at data analysis and statistical reasoning than I used to be.  On the other hand, whenever I've tried to design an experiment for my substantive research, I've failed miserably.  I've had lots of success in my research in social science and public health, but almost all involving the analysis of existing data.  I attribute my inability to design an experiment to a combination of lack of practice and lack of natural talent.  Measurement is central to statistics and is a completely different thing than data analysis.

***

KF: What is your pet peeve about published data interpretations?

AG:

I've called it the lure of certainty.  It's a problem with researchers and with consumers of researchers as well: they don't want to acknowledge uncertainty and variation. There's lots of talk about the treatment effect without a sense that it can and will vary, that a small positive effect in one setting might be negative in another, and that existing data might not be enough to determine its sign, even in the context of the data collection.  I get so frustrated when people take a "p less then .05" statistically significant result as definitive evidence.

Here's an example.  Recently the journal Psychological Science published some papers claiming scientific results based on college students and participants in the online Mechanical Turk system.  I criticized these results for lack of generality and the response was that this is standard practice, that it's naive to criticize a psychology study for having a nonrepresentative sample.

In this case, though, I think the naive criticism is correct. If you're estimating an effect with large average size and small variation, then it's ok to use a nonrepresentative sample. But in these social psychology examples, we have every reason to believe that main effects are small and interactions are large--that is, effects could be positive in some groups and negative in others.  In such settings, a nonrepresentative sample can kill you. But people don't want to think about this, because it's more comfortable to think about effects as "real" or "not real" without acknowledging variation.

***

KF: What sources do you turn to for reliable data analyses?

AG:

I rely on various colleagues of mine such as Don Rubin, Jennifer Hill, Eric Loken, John Carlin, and others. These people are very busy, though, so I've set up Rubin and Hill emulators in my head to work on problems when the main CPU in my head gets stuck.

KF: Thank you.

***

Previous Numbersense Pros:

Felix Salmon

Avinash Kaushik

Dean Baker

Sports journalists reported lies and then lie some more, just like dopers

Note: This is part 2 of a three-part response to an important article that appeared in the New York Times in August. See Part 1 here.

***

Part 2 deals with a few disagreements I have with Tim Rohan, who is the author of the article.

The first sentence of Rohan's article took my breath away:

Doping experts have long known that drug tests catch only a tiny fraction of the athletes who use banned substances because athletes are constantly finding new drugs and techniques to evade detection.

It would be nice if he acknowledges the failure of sports journalists to inform the public of this piece of common knowledge ten or twenty years ago. If the false negative problem is "long known", why didn't the media report it?

Even when I was researching my first book five years ago, almost every story was about how athletes were wrongfully accused of doping, how only insufficiently talented athletes would need to dope, that a long string of negative tests threw doubt into the one positive result, how testers bungled the collection of samples thereby hurting an athlete's reputation, etc.

We also were led to believe that athletes were ingesting banned substances inadvertently because they took herbal supplements, that young, physically hyper-fit athletes had medical needs to patronize anti-aging clinics, that certain "good guys" were just curious, and somehow got ensnared on the very first and only time they tried steroids, that other "good guys" only took steroids when trying to recover from injuries, etc.

Some readers may recall the almost cultish defence of Floyd Landis after the Amercian cyclist won the Tour de France and then got stripped of the title. He made up some lame excuse which through the magic of sports journalism, developed into a kind of mass delusion, the evidence of which lives on at the Trust by Verify blog, long after Landis confessed, and took down the cycling profession while he sank. (See here for my previous coverage of the Landis circus.)

Shocker: all those stories were about "false positives".

***

The cause-effect link in Rohan's quotation above is misleading. I won't repeat all of the arguments here--they are carefully laid out in Chapter 4 of Numbers Rule Your World (link)--but false negatives arise for a multitude of reasons. One reason for false negatives is the emergence of new drugs and techniques to evade detection. But the most important reason is that the test themselves are rigged so that it has high positive predictive value--meaning, a positive finding is highly trustworthy; and because of the inevitable tradeoff between false positives and false negatives in any diagnostic system, by making positive findings more valuable, the side effect is to make negative findings less valuable.

Such reporting perpetuates the myth of predictive models. Whenever the media reports on statistical methods used to predict things, it never ever tells us about the accuracy of such models. These reporters act as if predictions are 100% accurate, or close to it. But statistics is a science of uncertainty. Our society is overconfident about any kind of predictive models, not only for detecting dopers but also for detecting terrorists.

See Chapter 2 of Numbers Rule Your World, and Chapter 5 of Numbersense for more coverage of predictive models.

***

The past year has finally seen a public recognition of the "false negative" problem in anti-doping testing, which I called attention to years ago. The downfall of Lance Armstrong and Alex Rodriguez, among others, confirms that elite athletes dope, and elite athletes can get tested hundreds of times and pass them all.

It would be appropriate if the sports journalist guild would issue a mea culpa: "Sorry readers, we have been duped."

For those who still haven't noticed, a drug test did not catch notorious superstar dopers Armstrong or Rodriguez.

 

Timid testers: evidence from a suppressed study

Note: This is part 1 of a three-part response to an important article that appeared in the New York Times this month. Doping in elite sports has become a taboo topic in sports circles as no one wants to kill the golden egg. This article will prove to be an important reference.

***

Tim Rohan's article is a must-read for anyone who cares about the sanctity of sports (link). I do have some gripes about certain parts of the article which I'll get to in a future post. But the main theme of the piece is absolutely on target.

Chapter 4 of Numbers Rule Your World (which appeared three years ago) is called "Timid Testers / Magic Lassos". In this chapter, I explore the statistical nature of diagnostic testing, using two examples--anti-doping tests, and lie detector tests (polygraphs). The "timid tester" in the chapter title refers to the anti-doping agency: I asserted that the anti-doping agency, in its unwillingness to face the negative publicity related to a false positive result, has rigged the tests so that the majority of dopers will test negative.

When I made that assertion three years ago, the false negative problem in anti-doping was unspoken by those in the know and oblivious to those who get their news from the sports pages. The popular view--now proven to be a delusion--was that the dopers and the anti-dopers were playing a cat-and-mouse game. It is this false premise that led many people, including statisticians, to focus on the non-existent false positive problem, while failing to notice the blatant false negative problem.

Rohan's story is about several (unnamed) researchers who have thus far been prevented from publishing the results of a 2011 poll of elite track and field athletes which showed that about 30 to 45 percent of them admitted to doping. (The poll uses a randomized response strategy designed to hide the identity of the responders so the athletes were in no danger of implicating themselves.)

It appears that the anti-doping agency required approval from the sports governing bodies to publish this result, and thus far approval has not been granted. There goes the myth that the anti-dopers are adversaries of dopers, especially the superstars protected by their respective sports governing bodies. This is exactly the kind of behavior that I expected when I labeled them "timid testers".

***

See this post for an abridged version of my argument. See also here for my proposal to increase the number of false positive results in anti-doping tests.

Statistics and Big Data: a pending marriage

The current issue of Significance includes an article by me on the "pending marriage between statistics and Big Data". If you are a member of either the American Statistical Association or the Royal Statistical Society, you should be able to access the article via this link. If you don't belong to one of those, and you have a smartphone or tablet, you can download the Significance app -- because 2013 is the International Year of Statistics, this app is free for all right now.

If you don't belong to ASA or RSS, and do not have a smartphone or tablet, then I am able to print an excerpt of my article below. The excerpt looks at online experimentation (aka A/B Testing), pointing out where current practice runs into trouble, and how statisticians can play a role. In the article, I identify several other areas in which statisticians have the potential to help move the Big Data field forward. It won't be easy because fundamentally, the way computer scientists approach data is at odds with the way statisticans approach data.

David Walker presents the other side of the debate in the same issue.

***

More effective experiments

In 2012, Wired magazine eulogised the “A/B test”, declaring it to be “the technology that’s changing the rules of business”. The A/B test is known to every introduction to statistics student as the t-test of two means. Yes, the t-test is traced back to Gosset who developed it for the Guinness brewery in the 1900s. In the contemporary setting, a website delivers at random one of two pages to visitors, and measures if one page performs better than the other page, typically in terms of clickthroughs.

Brian Christian, the author of the Wired article, asked: “Could the scientific rigor of Google’s A/B ethos start making waves outside the web? Is it possible to A/B the offline world?” Any statistician will answer that many industries long ago implemented randomised, controlled experiments, and did so before the web existed, and at a higher level of sophistication than at most web companies. For example, direct marketers routinely run statistical tests to optimise their marketing vehicles such as catalogues and direct mail.

One of Christian's talking points holds: the web is indeed a nice laboratory in which tests can be executed at scale, and relatively painlessly (though see the section on randomisation below) [Ed: not excerpted]. And yet, in the A/B testing universe, few people are aware of the huge literature on statistical testing, or of [Sir Ronald] Fisher’s monumental contributions. This field is ripe for collaboration between computer scientists and statisticians. A quick flip through the Wired article reveals numerous fallacies about t-tests: fallacies of certainty, of automation, and of false positives among them.

The fallacy of certainty. Again and again, Christian stresses the certainty of test results, using words such as “incontrovertible”. Data from tests end all subjective arguments, we are told. How is it possible to have such definitive results when, as these web businesses claim, they run thousands of tests per year? One expects that most tweaks, such as changing the width of a border on a web page, have inconclusive results. It turns out that most practitioners of A/B tests use point estimates. If the test fails to achieve significance, the variation with the best performance is declared the “directional” winner. Sometimes, a test is run for such a length of time that tiny effects display significance by virtue of sample size. 

The fallacy of automation. In Christian’s world, the summit of A/B testing is “automating the whole process of adjudicating the test, so that the software, when it finds statistical significance, simply diverts all traffic to the better-performing option – no human oversight necessary”. Twinned with this is the fallacy of real time. One of the deepest insights in statistics is the law of large numbers, which requires a sufficient sample size in order to detect a signal to a given precision. Real-time decisions imply undersized samples, and huge error bars. Furthermore, such decisions are biased, as Microsoft scientists explained in an important paper [PDF] on the “novelty effect” and the “primacy effect”, among other things. False positive results abound in small samples, turning statistical testing into witchcraft.

The plague of multiple comparisons. In the new world of “choose everything”, that is to say, “see what sticks”, Wired reports that “the percentage of users getting some kind of tweak may well approach 100 percent. Statisticians worry about false positive findings when so many tests are run at the same time. Given the complexity of correcting for multiple comparisons, it is not surprising that the software tools available to conduct A/B tests completely ignore this issue.

We should be excited that randomised, controlled tests have been embraced by the web community. Regrettably, only a few practitioners, such as Ron Kohavi’s team at Microsoft, and Randall Lewis and Justin Rao (in their work [PDF] at Yahoo!), have reflected on the practical challenges of this enterprise. Statisticians are well equipped to make important contributions to how experiments are designed, executed and analysed.

 

***

Cathy O'Neill at MathBabe has some related thoughts. In this post, she makes a distinction between business analytics (statistics) and Big Data. In this post, it's not clear that she thinks Big Data as defined in the other post makes sense.