Big Data, Plainly Spoken (aka Numbers Rule Your World)

Financial and statistical incentives to over-diagnose and over-treat

Nice article in the New York Times about the "overdiagnosis" problem in cancer screening. The particular case is thyroid cancer in South Korea.

There are a number of things about any form of screening tests that one should always bear in mind:

• Death rate is measured as the number of deaths divided by the number of people with the disease. The latter number increases with better diagnosis techniques.
• Better diagnosis techniques for cancers inevitably identify tiny tumors, almost all of which will never develop into cancer during anyone's lifetime. Peggy Orenstein had a fabulous article a year ago about the same issue in breast cancer. In that case, those tiny tumors weren't even called cancer until the diagnosis movement sprang alive.
• Once a tumor is labeled cancerous, patients will opt to fix it. Because these tumors would never have killed the patients, they inflate the number of diagnosed cases without increasing the number of deaths. So just by virtue of increased diagnoses, the death rate is brought down.
• The immediate outcome of a nationwide screening program is to dramatically increase cases. The article is a little unclear about whether it is the number or the rate of death that did not fall. It doesn't really matter; either case leads us to conclude that the screening has failed to improve health.
• One must not forget that screening tests, subsequent confirmatory tests, treatments, etc. all cost money, so there is a financial incentive to over-diagnose and over-treat.
• In addition to the financial incentive, there is the issue that I raised in Chapter 4 of Numbers Rule Your World (link). A false negative is a very public error on the part of the medical establishment while a false positive (followed by say removal of the thyroid) is an unobservable error. So there is a statistical incentive to over-diagnose and over-treat.

 

Round-up of coverage of the Big Miss of Big Data

There is now some serious soul-searching in the mainstream media about their (previously) breath-taking coverage of the Big Data revolution. I am collecting some useful links here for those interested in learning more.

Here's my Harvard Business Review article in which I discussed the Science paper disclosing that Google Flu Trends, that key exhibit of the Big Data lobby, has systematically over-estimated flu activity for 100 out of the last 108 weeks. I also wrote about the OCCAM framework, which I find useful to think about the "Big Data" datasets we analyze today versus more traditional datasets from the past.

The HBR article caught the attention of various outlets, including New York Times and ABC News.

Slate was probably the earliest to react, and noticed a post on this blog that was the precursor to the HBR article.

Readers who are specifically interested in GFT should read the source materials themselves, which are quite accessible. Start with the Science paper. After that, you can read the original research article by the Google team, hosted at google.org (click on the PDF link in the blue box at the bottom of the page). There are some bold claims in this paper, as well as caveats. They seemed to be concerned about "false alerts" at the time, such as news events rather than illness that drive certain searches. (For those statistically inclined, the underlying model involves only 1,152 points of data--128 weekly aggregates in nine regions--but a search through 450 million simple logistic models to not only define which search terms are important but also determine how many search terms to include in the final regression.)

Then read the article by Cook, et. al., which covers an update to the model made after the 2009 season when GFT totally missed the pH1N1 swine flu epidemic. Notice that this Big Miss is the opposite error to the "false alert" problem. (See Chapter 4 of Numbers Rule Your World for a thorough discussion of different types of prediction errors, and how to think about them.) From the charts in the Cook article, you can see that in the runup to the Big Miss, GFT systematically under-estimated flu activities for as many weeks as you can count.

The overhaul was drastic. The search term topics that accounted for 70% of the original model were reduced in importance to 6% while two other topics that counted for 8% originally were inflated to 69% in the updated model. This dramatically improved the "fit" statistic (RMSE) for the "first phase" of the Big Miss from 0.008 to 0.001.

Next, there is Butler's article for Nature, (Feb 2013) which precedes the Science article, but first pointed out the over-estimation problem for the 2012 flu season. One possibility is that the model update described above over-compensated for the Big Miss, making it more susceptible to the False Alert.

Other media coverage of Google Flu Trends include Guardian (which focuses on the need to understand causality), CRN, and the Economist (which talks mostly about Twitter data which is much more problematic than Google search data).

***

Tim Harford has probably the most educational of the pieces revisiting Big Data for the Financial Times. (When I wrote this line, the FT link wasn't working. The title of the article is "Big Data: Are we making a big mistake?" if you need to find a different link.) His is the longest and covers a lot of ground, and has great examples, including one of my own. Highly recommended.

***

One of the slogans of the Big Data industry (of which I'm a part) is the push toward "evidence-based" decision-making in place of "gut feelings" or "instincts". Until now, I'm afraid there has been plenty little "evidence" presented to support the assertions of universal, revolutionary goodness of Big Data (try searching for quantitative assessment of Big Data projects). I hope we are witnessing the birth of evidence-based decision-making inside the industry of dishing out evidence-based advice.

The reason I put out the OCCAM framework is to steer our community toward a more constructive approach to tackling "Big Data" problems. It requires a fundamental shift in how we define the problem. I have a moderately more technical take on some of the statistical challenges in an article published in Significance, earlier this year. This article discusses six technical challenges where we need substantial progress.

Statisticians sometimes dismiss these as "old news," claiming that the same problems exist in smaller datasets and the problems are well known. A recent example is Jon Schwabish's tweet saying that this discussion induces a "yawn". This reaction feels a bit like Fermat writing in the margin claiming he has a proof. The rest of the world don't want to wait 358 years to figure out what the goods are which these statisticians are hiding.

In my view, there has been some interesting work but nothing to settle debates. If we have great solutions, we won't be discussing these same problems today.

***

Back to flu prediction. It's really something that is well worth pursuing!

It's a nice-defined, self-contained problem that has social benefits and whose results can be easily measured. We should be grateful that the Googlers spent time working on it. It's a problem I'd love to work on if I have time and resources on my hands.

The researchers also have pioneered this type of research using search term data. This is highly significant, and the data represent a perfect example of what I call OCCAM data: the data is purely Observational (related to what Harford calls "found data" or what Dan Eckles calls "data exhaust"); it has no Controls; it is seemingly Complete; it wasn't collected for the purpose of predicting flu trends, that is, it is Adapted from other uses; and the search data was Merged with the CDC data (the matching of states and regions, and of weeks were not exact as you can tell from the original research article).

The several published versions of the predictive models are clearly failures but anyone who is in this business knows model building is an iteractive process. One can learn from these mistakes. I happen to think they need to wipe the slate clean and use an entirely different approach. It's a small price to pay if there is reward down the road.

I sincerely hope that this coverage will lead to improved modeling and analytical techniques rather than a retrenchment.

If you find other related links, please place them in the comments.

 

 

Sports journalists reported lies and then lie some more, just like dopers

Note: This is part 2 of a three-part response to an important article that appeared in the New York Times in August. See Part 1 here.

***

Part 2 deals with a few disagreements I have with Tim Rohan, who is the author of the article.

The first sentence of Rohan's article took my breath away:

Doping experts have long known that drug tests catch only a tiny fraction of the athletes who use banned substances because athletes are constantly finding new drugs and techniques to evade detection.

It would be nice if he acknowledges the failure of sports journalists to inform the public of this piece of common knowledge ten or twenty years ago. If the false negative problem is "long known", why didn't the media report it?

Even when I was researching my first book five years ago, almost every story was about how athletes were wrongfully accused of doping, how only insufficiently talented athletes would need to dope, that a long string of negative tests threw doubt into the one positive result, how testers bungled the collection of samples thereby hurting an athlete's reputation, etc.

We also were led to believe that athletes were ingesting banned substances inadvertently because they took herbal supplements, that young, physically hyper-fit athletes had medical needs to patronize anti-aging clinics, that certain "good guys" were just curious, and somehow got ensnared on the very first and only time they tried steroids, that other "good guys" only took steroids when trying to recover from injuries, etc.

Some readers may recall the almost cultish defence of Floyd Landis after the Amercian cyclist won the Tour de France and then got stripped of the title. He made up some lame excuse which through the magic of sports journalism, developed into a kind of mass delusion, the evidence of which lives on at the Trust by Verify blog, long after Landis confessed, and took down the cycling profession while he sank. (See here for my previous coverage of the Landis circus.)

Shocker: all those stories were about "false positives".

***

The cause-effect link in Rohan's quotation above is misleading. I won't repeat all of the arguments here--they are carefully laid out in Chapter 4 of Numbers Rule Your World (link)--but false negatives arise for a multitude of reasons. One reason for false negatives is the emergence of new drugs and techniques to evade detection. But the most important reason is that the test themselves are rigged so that it has high positive predictive value--meaning, a positive finding is highly trustworthy; and because of the inevitable tradeoff between false positives and false negatives in any diagnostic system, by making positive findings more valuable, the side effect is to make negative findings less valuable.

Such reporting perpetuates the myth of predictive models. Whenever the media reports on statistical methods used to predict things, it never ever tells us about the accuracy of such models. These reporters act as if predictions are 100% accurate, or close to it. But statistics is a science of uncertainty. Our society is overconfident about any kind of predictive models, not only for detecting dopers but also for detecting terrorists.

See Chapter 2 of Numbers Rule Your World, and Chapter 5 of Numbersense for more coverage of predictive models.

***

The past year has finally seen a public recognition of the "false negative" problem in anti-doping testing, which I called attention to years ago. The downfall of Lance Armstrong and Alex Rodriguez, among others, confirms that elite athletes dope, and elite athletes can get tested hundreds of times and pass them all.

It would be appropriate if the sports journalist guild would issue a mea culpa: "Sorry readers, we have been duped."

For those who still haven't noticed, a drug test did not catch notorious superstar dopers Armstrong or Rodriguez.

 

Timid testers: evidence from a suppressed study

Note: This is part 1 of a three-part response to an important article that appeared in the New York Times this month. Doping in elite sports has become a taboo topic in sports circles as no one wants to kill the golden egg. This article will prove to be an important reference.

***

Tim Rohan's article is a must-read for anyone who cares about the sanctity of sports (link). I do have some gripes about certain parts of the article which I'll get to in a future post. But the main theme of the piece is absolutely on target.

Chapter 4 of Numbers Rule Your World (which appeared three years ago) is called "Timid Testers / Magic Lassos". In this chapter, I explore the statistical nature of diagnostic testing, using two examples--anti-doping tests, and lie detector tests (polygraphs). The "timid tester" in the chapter title refers to the anti-doping agency: I asserted that the anti-doping agency, in its unwillingness to face the negative publicity related to a false positive result, has rigged the tests so that the majority of dopers will test negative.

When I made that assertion three years ago, the false negative problem in anti-doping was unspoken by those in the know and oblivious to those who get their news from the sports pages. The popular view--now proven to be a delusion--was that the dopers and the anti-dopers were playing a cat-and-mouse game. It is this false premise that led many people, including statisticians, to focus on the non-existent false positive problem, while failing to notice the blatant false negative problem.

Rohan's story is about several (unnamed) researchers who have thus far been prevented from publishing the results of a 2011 poll of elite track and field athletes which showed that about 30 to 45 percent of them admitted to doping. (The poll uses a randomized response strategy designed to hide the identity of the responders so the athletes were in no danger of implicating themselves.)

It appears that the anti-doping agency required approval from the sports governing bodies to publish this result, and thus far approval has not been granted. There goes the myth that the anti-dopers are adversaries of dopers, especially the superstars protected by their respective sports governing bodies. This is exactly the kind of behavior that I expected when I labeled them "timid testers".

***

See this post for an abridged version of my argument. See also here for my proposal to increase the number of false positive results in anti-doping tests.

Baseball's Steroids Problem Won't Go Away

Steroids continue to plague professional sports. The latest name to fall is Alex Rodriguez, the shortstop/3rd baseball superstar who currently plays for the Yankees. It wasn't long ago that he was considered a "good guy" of the sport. Now, he's a pariah.

In the rush to make Rodriguez the villain, the media continues to miss these two important aspects to the steroids story:

• Anti-doping tests have a huge false-negative problem. I have been talking about this for years. This topic is  examined in Chapter 4 of Numbers Rule Your World. Rodriguez never failed any test; he got caught because one of his enablers ratted him (and others) out. His supplier betrayed him because he didn't pay up. All the other suspended players also were ensnared during the investigation of a Florida clinic.

• Statistics is powerful yet limited. Based on looking at the statistical properties of screening tests, I can be sure that lots of dopers are undetected. Under some assumptions, I estimate that 80% of dopers escape unscathed. However, statistics is hopeless at pinpointing which particular players are the dopers. A similar situation arises in Chapter 5 of Numbers Rule Your World (link) regarding cheaters in the Canadian lotteries; it's not hard to show that there are cheaters but it's hard to figure out who. The solution is to use the police or investigators to supplement the statistics. This is exactly how Rodriguez and the Balco crew (before him) got caught.

***

There is a third story that is completely ignored by the sports media. Ryan Braun was the 2011 MVP who also patronized the same clinic. Braun, unlike Rodriguez, failed a drug test in 2012. He vigorously defended his innocence (like others), and loudly attacked the messenger. Here is one of his now-infamous quotes:

"It hasn’t been easy. Lots of times I wanted to come out and tell the entire story, attack everybody like I’ve been attacked. My name was dragged through the mud. But at the end of the day I recognized what was best for the game of baseball.”

There are a bunch of other quotes here. Braun got off the same way Lance Armstrong got off years ago when he tested positive... by an expensive legal team and a minor technicality.

In Chapter 4 of NRYW (link), I argue that on chemistry alone, the anti-doping tests are configured such that the positive predictive value is very high, meaning if one tests positive, there is a high chance the subject is a doper. The reverse is not true: if someone is a doper, there is actually a low chance the test would come back positive. These two outcomes are linked and reflect the tradeoff of drug testing.

The Braun story proves again that the positive steroids finding is very informative. Yet again, we learn that drug testing doesn't have a false positive problem. There is a false negative problem.

 

How to Consume Big Data (from McGraw-Hill blog)

When you hear about Big Data, you almost always hear about the supply side: Behold the data in un-pronounceable units of bytes! Admire the new science inspired by all the data! Missing from this narrative is the consumption side. A direct consequence of Big Data will be the explosion of data analyses—there will be more people producing more data analyses more quickly. This will be a world of confusing and contradictory findings.

In my new book, Numbersense, I argue that the ability to analyze and interpret these data analyses will give one a competitive edge in this world of Big Data.

Numbersense is the noise you hear in your head when you see bad data or bad analysis. After years of managing teams of data analysts, I’ve learned that what distinguishes the best from the merely good is not math degrees or computer skills; it is numbersense.

Numbersense is an intangible quality that you can’t teach in a classroom. The best way to pick it up is by learning from people who have it. For this blog post, I selected two great analyses of data analyses that have impressed me recently. These are highly instructive examples.

***

Eating red meat makes us die sooner! Zoë Harcombe didn’t think so.

In March, 2013, nutritional epidemiologists from Harvard University circulated new research linking red meat consumption with increased risk of death. All major mass media outlets ran the story, with headlines such as “Risks: More Red Meat, More Mortality.” (link) This high-class treatment is typical, given Harvard’s brand, the reputation of the research team, and the pending publication in a peer-reviewed journal. Readers are told that the finding came from large studies with hundreds of thousands of subjects, and that the researchers “controlled for” other potential causes of death.

Zoë Harcombe, an author of books on obesity, was one of the readers who did not buy the story. She heard that noise in her head when she reviewed the Harvard study. In a blog post, titled “Red meat & Mortality & the Usual Bad Science,” (link) Harcombe outlined how she determined the research was junk science.

She knows this type of research methodology rarely if ever delivers conclusive evidence of causation. Then, she found support from a data table included in the research paper. The table shows that the cohort of people who report eating more red meat also report higher levels of unhealthy behaviors, including more smoking, more drinking, and less exercise. Thus, the increased risk of death observed in the study could have been explained by factors other than red meat consumption.

For a full dissection of Harcome’s amazing post, please click here. Chapter 2 of Numbersense looks at the quality of data analyses of the obesity crisis.

 ***

Netflix turns viewers into puppets! Felix Salmon declined to become one.

In February, 2013, Netflix, ever the media darling, premiered House of Cards, a re-make of the successful British television show, their second foray into producing original content for its tens of millions of subscribers. Netflix executives regaled the press with stories of how Big Data analysis took the risk out of their $100 million decision.

Andrew Leonard, the technology reporter for Salon.com, gobbled up the Netflix story, even interpreting it as a “symptom of a society-wide shift.” (link) Like other news analysts, Leonard was convinced by the “pure geek wizardry” used to analyze mountains of data collected from Netflix customers. The machine, we’re told, decided that David Fincher should be the director and Kevin Spacey, the star. From here, it is a short trip to the lala land of viewers as puppets with machines as the overlord.

This analysis aroused the skeptic in Felix Salmon, the finance blogger for Reuters. In his blog post, “Why the Quants Won’t Take Over Hollywood,” (link) Salmon raised other factors that affect the box office, including billions spent on marketing and publicity, the quality of the writing, the sociopolitical climate, the complex relationship between originals and remakes, and the poor track record of predictive modeling in Hollywood. On this last point, Salmon exhibits a keen sense of the limitations of science, speaking of “impossible-to-formulate cocktail of creativity, inspiration, teamwork, and luck.”

Chapters 4 and 5 of Numbersense explains how you should judge predictive models used by marketers.

***

When their respective blog posts surfaced, Harcombe and Salmon were lone voices vetting carefully the claims based on other people’s data analyses. Their well-honed numbersense allows them to stand firm in the face of mountains of data, worship of high science, formidable-sounding technical jargon, and academic reputations. The problems with the original research are far from obvious. The point is not to debunk these studies—no data analysis is ever infallible—but to figure out for yourself what is credible, and what is junk.

Screening screening

Mammograms continue to be an emotional and controversial topic. I blogged about it some time ago. (link) Felix Salmon, whose blog should be daily reading, praises an article by Peggy Orenstein called "Our Feel Good War on Breast Cancer", NYT Magazine (link). Salmon's blog provides a quick summary; Orenstein's article is very long.

Orenstein's point of view has particular weight because she was diagnosed at an early age, and was an early advocate of breast cancer screening and now has a change of heart. She's worried about false positives, overdiagnosis, and over treatment.

You may recall that two or three years ago, there was a furor about new recommendations by experts that women reduce the frequency of screening. I used this example in a talk to illustrate the failure of these experts to communicate their statistical results, leading to a prompt, forced retreat. Immediately after the talk, several women raised objections--they believed this is such an emotional issue that numbers mean nothing. It was illuminating since I was at an analytics conference, and the women work with numbers on a daily basis. In the meantime, I was on shaky ground because of my gender. I'm glad that someone of Orenstein's stature has come out to try to explain the other side of this story. (See also my post on the Paradox of Screening here.)

***

Instead of rehashing the arguments around screening tests, today I just want to highlight some subtle aspects of measuring health outcomes.

One of these is described in Orenstein's article. One of the metrics used to measure health outcomes is the five-year survival rate... what proportion of patients are still alive five years after they are diagnosed with breast cancer? Said differently, we are interested in the length of time between the time of diagnosis and the time of death. There are two ways to improve this metric, by delaying the time of death, or by moving earlier the time of diagnosis. The former is difficult to achieve, and requires genuine progress in medication. The latter is very easy! If we screen younger people, we will move the time of diagnosis earlier. Without any medical progress, we will make the five-year survival rate fall. This phenomenon is known as lead-time bias.

Expanding screening to younger and less vulnerable populations has another effect, which I discussed in Chapter 4 of Numbers Rule Your World. The marginal person who gets diagnosed will be less ill than the average person - Orenstein tells us that the form of disease most commonly diagnosed in younger women (DCIS) ought not even to be called "cancer". Add less ill people to the population of ill people will automatically improve the survival rate, again without any medical progress. This phenomenon is known as length bias.

 

 

 

Timid testers, magic lassos

That is the title of Chapter 4 of Numbers Rule Your World. When the book came out two years ago, lots of steroids scandals have broken out, and yet many people were in deep denial. Recall the saga of Floyd Landis, the U.S. cyclist, who initially claimed the drug test was wrong, and that he was witch-hunted. There was even a website called Trust But Verify where all his supporters gathered. As we now know, he was guility as sin. He was also the chief whistle-blower that has brought down Lance Armstrong.

If you read the book then, or one of my blog posts here, you'd hopefully have realized the scale of the scandal before all of this transpired. Applying a bit of statistical reasoning to the numbers around us makes us smarter.

The most recent news coming out of that scandal is just maybe Lance will confess (link). What's taken so long? And after all this time, his PR people are still floating a trial balloon? 

At the Guardian, they asked if lie detectors should be used in sports (link). That is the "magic lassos" reference in Chapter 4. Lie detectors and steroids tests both use statistical detection but have different statistical properties. My readers will again have a head start on this next debate. The key point is that a "true negative" in a steroids test is not necessarily a "true negative" in reality because there are a million ways to hide one's doping. The real goal of any anti-doping problem is to catch the dopers, and having zero positive test results is only part way there.

Lance Armstrong beat statistical odds, but it's not enough

Readers of Chapter 4 of Numbers Rule Your World would not be surprised about the fall of Lance Armstrong. The big theme of the chapter is that any screening tests has to balance between false positives and false negatives. The anti-doping agencies are so concerned about not falsely accusing anyone that they leave a gigantic hole for dopers to walk through. Strings of negative tests, therefore, do not prove one innocent; however, even a single positive test could be quite telling.

While we think about Armstrong's plight, let's not forget about this fact: every one of those who now confessed  passed hundreds of tests in their careers, just like Armstrong did. In fact, fallen stars like Tyler Hamilton and Floyd Landis also passed lots of tests before they got caught. In effect, dopers face a lottery with high odds of winning and low odds of losing.

Sadly, the media covering this issue continues to ignore false negative errors in testing. I have yet to find one article covering the current situation that acknowledges that dopers pass tests all the time! This over-confidence in scientific models is a disease, not just in drug testing.

***

In the book, I also discussed why we need a lie detector test in athletics. So, we arrive now at a day when a good handful of big-name cyclists decided to shed their lies and open up to us.

***

Another myth shattered by this scandal is the idea that stars don't need to cheat. It is most likely the opposite. At the very top of any sport, especially a sport that pays, the difference between the number 1 and the number 2 is vast in terms of financial reward but infinitestimal in terms of physics. Every little advantage counts. Placebos count.

It's hard to imagine why someone who has no chance of winning anything would take drugs that might kill them. So, when they say everyone was cheating, I wonder if they meant everyone who was competitive was cheating.

***

Here're my previous articles calling for "more false positives" in drug testing (link, link).

There are many stories because so many cyclists are named. Some are already selling their stories. Here are some links:

New York Times on the big report (link)

Tyler Hamilton already has a book (link)

George Hincapie testified (link)

Levi Leipheimer statement (link)

Glossary of secret codewords (link)

Former Irish massage therapist told her story (link)

Former coach continues to deny, deny (link)

Tour de France organizers silent. Imagine not having anything to say about a 7-time champion. (link)

Also silent are his sponsors. Nike? (link)

 

 

 

Speaking analytics

I was a guest on the Analytically Speaking series, organized by JMP. In this webcast (link, registration required), I talk about the coexistence of data science and statistics, why my blog is called "Junk Charts", what I look for in an analytics team, the tension between visualization and machine algorithms, two modes of statistical modeling, and other things analytical.