Big Data, Plainly Spoken (aka Numbers Rule Your World)

The statistics of anti-doping: the entire book in one real-life example

A reader of my blog, Joran E., pointed me to this great article (by Ross Tucker) that covers one of the newer anti-doping measures (the biological passport), which links to this recent NYT article on two Italian cyclists found guilty of doping. So while I was researching this latest development, I came across the latest legal maneuvres in the case of Alberto Contador, the Spanish cyclist and multiple winner of Tour de France who tested positive after last year's victory, and subsequently blamed a contaminated steak (I mentioned his case here last year).

Anti-doping provides a perfect back drop to revisit all five statistical concepts that form the spine of my book, Numbers Rule Your World.

***

The most potent form of doping these days is human growth hormones (HGH), EPO and similar compounds that have the characteristic of occurring naturally so that labs must seek to separate dopers from people who have "natural highs". By contrast, for compounds that don't occur naturally, such as clenbuterol that ensnared Contador, even minute amounts can be proof of wrongdoing.

In order to know what level of a compound is "unnatural", statisticians need to establish what is natural. This is the concept behind Chapter 1: we calculate the "average" (natural) value, but focus on examining variations around the average.

Admitting that the natural value is not uniform across all people, statisticians determine different averages for different "types" of people; the simplest such subgroups would be male/female, and age groups. The biological passport takes this idea to the extreme: each individual athlete is tracked over time to establish his or her own average. We just put into practice the concept behind Chapter 3, which is to avoid lumping together things that are different.

The cases of the two Italian cyclists represent the first two in which athletes have been punished based on evidence from the biological passport. Previously, the enforcers need a failed drug test or a police bust to convict dopers.

Readers who like the materials in the Conclusion chapter (as related to Chapter 4) on the hematocrit test should definitely read Tucker's article for details on how the biological passport works.

***

The developments in the Contador case are very discouraging: the Spanish cycling federation showed an unwillingness to expose the biggest star of the sport, first by assessing a one-year ban (when the norm is two years), and most recently, overturning that shortened punishment. USADA, the US anti-doping body, expressed its concern here, and the reversal of the ban is under appeal.

The Spanish authority accepted the Contador camp's explanation of unintentional consumption of tainted beef as the reason for testing positive. Statisticians who believe in the logic of hypothesis testing will find such a conclusion absurd.

Let's walk through how we apply the logic as described in Chapter 5 of Numbers Rule Your World to this situation. Assuming that Contador did not dope, what is the chance that minute amounts of clenbuterol would be found in his body? Unfortunately for Contador and other athletes failing this drug test, the chance is vanishingly small.

Like most accused dopers, his camp did not challenge the presence of clenbuterol; they merely offered an alternative theory for why it was there. A large number of coincidences had to occur in order for their theory to be believed: beef had to be taken from Spain into France to serve Contador, and only Contador (not any of his teammates); a different source of beef must have been used on other days during the Tour on which Contador ate beef (since he tested negative on most other days of the tour); he was one unlucky fellow since anti-doping tests have high false negative rates in general, and he managed to test positive on that one time he ate the contaminated beef; he was also extremely unlucky since Europe banned the use of clenbuterol to raise cows in the 1990s, and the beef he ate on that one occasion had to have come from an unscrupulous farmer violating the ban.

Statisticans would politely listen to all that, and declare "rare is impossible". It's much easier to believe that he was doping. (We would admit that there is a miniscule chance that the conclusion is incorrect -- the chance is precisely that of those coincidences occurring.)

***

Why does the scientific process disintegrate into this sort of he-said-she-said argument?

The concept behind Chapter 2 proves useful here. The statistical model that links the biological passport and/or the drug test to doping is one based on correlation, not causation. The passport or drug test does not provide direct evidence of doping (unlike a police bust). But as I point out in the book, correlational evidence can be powerful, and has been profitably used in all kinds of decisions. Because clenbuterol is not produced naturally in the human body, this test result is very close to causal evidence; it's less secure for things like EPO and HGH.

It's just more complicated when causal evidence is unavailable because people can now advance all sorts of hypotheses to explain the correlation. We then get story time, a phenomenon I frequently discuss on my blog. I'm happy to hear the stories but one must seek evidence to support these stories.

In the Contador case, for example, I'd like to see evidence that the steak was eaten, receipts from the vendor who imported the beef, documentation of which farm raised the cow, inspection of the farm to confirm that it used clenbuterol, traceback of beef from that farm to find the presence of clenbuterol, etc. In none of the reports on this case have I seen any of this evidence, and more disturbingly, the supporters of Contador don't appear to be asking any such questions. (See, for instance, Christian Josi on Huffington Post.)

***

Why would statisticians accept the chance of falsely accusing a clean athlete, however small that chance is? This is because we know that there is no such thing as a perfect test. The only test that will never yield a false positive is the test that never issues any positive results!

We already accept this type of situation in the Western legal system. A criterion of "beyond reasonable doubt" in the courts does not guarantee no wrongful convictions. In fact, thanks to the work of groups such as the Innocence Project, we know that some unfortunate people are wrongfully convicted, sometimes for long sentences for grievous crimes they did not commit.

As explained in my book (which I won't repeat here), the real issue in anti-doping is not about false positives but about false negatives. I fear that the entire system is so lenient toward dopers that they would take the (small) risk of detection.  I'll make a case for this in a future post.

 

Major shocker: can't make something out of nothing

I point you to the latest Junk Charts post in which we unveil a shocker: a big lump of nothing is still nothing.

 

 

Know your data 5: Joel Stein says it's okay

Joel SteinKlein covers at length the silent tracking of on-line behavior, and linking of the collected data to individuals by businesses in a Time magazine piece.  I highly recommend it.

The first page is consistently amusing, as Stein describes the different profiles various companies have compiled about him. For example:

Google's Ads Preferences believes I'm a guy interested in politics, Asian food, perfume, celebrity gossip, animated movies and crime but who doesn't care about "books & literature" or "people & society." (So not true.) Yahoo! has me down as a 36-to-45-year-old male who uses a Mac computer and likes hockey, rap, rock, parenting, recipes, clothes and beauty products; it also thinks I live in New York, even though I moved to Los Angeles more than six years ago. Alliance Data, an enormous data-marketing firm in Texas, knows that I'm a 39-year-old college-educated Jewish male who takes in at least $125,000 a year, makes most of his purchases online and spends an average of only $25 per item. Specifically, it knows that on Jan. 24, 2004, I spent $46 on "low-ticket gifts and merchandise" and that on Oct. 10, 2010, I spent $180 on intimate apparel. It knows about more than 100 purchases in between. Alliance also knows I owe $854,000 on a house built in 1939 that — get this — it thinks has stucco walls. They're mostly wood siding with a little stucco on the bottom! Idiots.

These businesses now admit that they build individual profiles, which means that they are not just analyzing average behavior, not just looking at anonymous user behavior (for example, replacing names with numbers as tracking identities), but they are linking behavioral data to real names, addresses, phone numbers and more.

After some tortuous reasoning over 5 pages, Stein concludes:

the more I learned about data mining, the less concerned I was. Sure, I was surprised that all these companies are actually keeping permanent files on me. But I don't think they will do anything with them that does me any harm.

***

There are a few statements in Stein's article with which I don't agree in full.

• Data mining does no actual harm.

Stanford Professor Ryan Calo believes that privacy concerns are overblown because "data mining does no actual damage". And, that's because "the real problem with data mining arises when the data is wrong".  I read this to mean that Calo (and others in this camp) believes that data mining does "actual damage" when the data is wrong. News for them: data can never be accurate, it is always wrong.

• The problem of data errors is solved by letting consumers inspect and edit their own profiles.

Allowing individuals to access and "correct" their data is not a real solution. "Correcting" the data will render the data even less informative (more bias) as we can create "personas" to fool the algorithms. Say, the profile marks you as a BT (bad tipper), and indeed you are a bad tipper but nobody would want to be labeled as such. I discussed this issue in the context of "correcting" credit scores in Chapter 2 of Numbers Rule Your World. (More on BT here.)

• Data mining for ad targeting is a low-stakes activity. 

Stein is right that the stakes are low -- the cost of showing poorly-targeted ads is indeed low.  However, precisely because false positives are cheap, designers of data-mining algorithms will tune them to tolerate more such errors (in order to reduce false negatives, which cause lost sales opportunities). This means the data is guaranteed to be inaccurate, which undermines Stein's belief of no harm done. (See Chapter 4 of Numbers Rule Your World for more on the trade-off between types of errors.)

• Companies are not doing anything harmful.

"Are not" does not imply "will not" unless we have "cannot".

• Asking consumers for prior consent solves the problem of secretive tracking.

This type of solution is both deceitful and impractical. Anyone who has been asked to agree to the 50-page Terms and Conditions for the iPhone (or similar) understands that one is not supposed to read the agreement, just to categorically accept it. Besides, it is difficult, if not impossible, to document exactly how each employee of a business is using the data. This gets even more complex if the business sells the data to third-parties, and/or exchanges the data with other data collectors.

***

The CEO of Bizo, one of the companies that build profiles, claims that "not many people seem to be creeped out by all the junk mail they still get from direct-marketing campaigns, which buy the same information from data-mining companies". He clearly does not understand the other side of the argument.

The differences between traditional "junk mail" and today's "targeted ads" are speed and scale. Direct mail campaigns typically take at least 2 weeks to plan and execute (and that is extremely fast) while an on-line targeted ad appears possibly within seconds of someone visiting a web page. Needless to say, the creepiness factor is multiples higher. Stein has a nice anecdote about Zappos's experience.

The other key difference is scale. The extent to which data is being collected, exchanged, aggregated, analyzed today is also multiples that of yesterday.

***

If, indeed consumers do not care, as Stein seems to believe, then tracking should always be opt-in. Today, it is with few exceptions opt-out, and in many cases, it is undisclosed.

As I mentioned when launching the "Know your data" series of posts, I think these types of technology have both benefits and problems, and I don't oppose all such efforts. After all, I am a part of the direct marketing industry. Nevertheless, I think some of the current data-collection activities are too intrusive in addition to being misguided. I'll explain what this means in a future post.

 

The sway of being asymmetric, Google edition

That's the title of Chapter 4 of Numbers Rule Your World. A great real-life example of those principles at work is Google's attempt to cleanse its search results. A nice write-up is here.

Google engineers want to separate the useful websites from the spam websites ("content farms"). Unfortunately, no diagnosis system is perfect. They will make false-positive and false-negative errors. The article focuses on the false-positive errors: site owners who claim their websites have been wrongly classified as spam. In passing, the journalist also tells us that there are spam websites that have escaped the penalty. Google faces a trade-off: reducing false positives leads to more false negatives; and vice versa.

What's interesting about this is to figure out Google's incentive. The cost of each type of error is different; the level of public outcry in reaction to each type of error is different. It took years for search users to make enough noise for Google to implement this tweak; it took days for the false-positive victims to get mass-media coverage. Google has many constituents to please: search users, advertisers, site publishers, to name a few. It's a pretty complex web.

The blind spot in technology journalism

The New York Times covers several software companies whose business is to analyze databases of documents and emails in support of litigation. Large corporations are often sued and they have to submit troves of documents and emails to lawyers who charge hourly rates to inspect and summarize the information. Bills could run up to millions in large lawsuits. These technology upstarts claim to be able to replace humans with computers.

I recommend reading this article... with a critical eye. Don't expect help from the journalist (John Markoff) who seems blind to, or incapable of, evaluating the limitations of the technology. The tone and content of this article is typical of anything in the technology pages of our news media: one senses awe, and unbounded optimism, as if the articles were press releases issued by the companies making the technologies.

Here are a few points to ponder:

• Computers apparently don't make any mistakes. People can get tired and overlook things, we are told. But when a computer tells us that the "sentiment" of someone's email is "positive", it never makes a mistake. The truth is all statistical models are imperfect, and mistakes range from acceptable to numerous. Without proper studies of the false positive and false negative rates of these computer algorithms, there is no way for readers to judge whether these technologies represent progress or not.
• Errors are particularly frequent in analyzing "unstructured" data such as text. Time of day is a "structured" type of data while the sentiment expressed in an email is "unstructured". If the computer tells you that the CEO never sends emails after 9 pm at night, there is little reason to doubt the accuracy of that computation. If the computer tells you that emails about the CEO became increasingly "negative", how much confidence can one have in the accuracy of this statement? What is a negative email? If the same email contains both positive and negative comments, how is it determined if it is positive overall? How does the computer recognize sarcasm or irony? A simple comment "good job" can be positive; it can also be negative (if the job is accounting tricks); it can be ironic; it can be completely irrelevant.
• Much noise is made about technology displacing workers, and how this might affect the economy by producing lower-skill jobs. Two things are curiously absent from the article. First, the new technology generates new jobs in new companies and many of these are high-skill jobs like software development, product management, algorithm design, and so on. Second, the computers can surface information but this software does not replace the need to have human beings (analysts) to look at the output, and interpret the information. This key point is often lost: reports and dashboards are useless until someone looks at them. Business analysts also are high-skill jobs.
• Finally, I have direct experience with this sort of legal discovery processes. A giant amount of billings involves people manually scanning all printed documents collected from your office, including every copy of identical documents you might possess. If you distribute copies of a presentation to 20 people in a meeting, and these 20 people are all considered people of interest, all 20 copies will be scanned eventually. These billings I suspect won't be replacable by computers.

I am not saying these companies are hawking vaporware. For some tasks, computers clearly can do a much better job than humans. I just think that our technology reporters can serve us better by covering both the promise and the limitation of new tools. They can start by interviewing users of such software, both satisifed users and unhappy users.

 

Reading Proofiness 2 plus The Opinion Makers

Having picked up and breezed through David Moore's book The Opinion Makers, I am reminded of the promised second installment of my "close reading" of Charles Seife's Proofiness. Previously, I covered the first part of the book; the second part is the first part applied to political issues. (My summary review of Proofiness is here.)

Moore was a former Gallup pollster; perhaps he would consider himself a "reformed" pollster. He shares the same view with Seife as it relates to political polling: that polling has been usurped by the political apparatus, and abused by the mass media; that polls do not reflect the so-called "public opinion"; that politicians are foolhardy to rule by poll results; and that journalists (and also politicians) create polls to validate their agendas or preconceived stories. In short, they think polls usurp democracy, an ironic conclusion as their inventors had promised the opposite.

***

Moore's book is short (about 150 pages) and feels even shorter because he has one main thesis only but what a powerful thesis -- all modern poll results are misleading because they include the views of large numbers of people who have no knowledge or interest or conviction on the topic at hand; when a category of "don't know" is reported, it is almost always a criminal underestimate of the real portion of people who don't know or care. In poll after poll, the real insight is that most Americans do not follow major political events and thus do not have the knowledge or have any stable opinion of pretty much anything. But the pollsters don't tell you that.

This trickery is accomplished in two ways. First, the so-called "forced choice" design states all questions as yes/no, and the reported "don't know" comes about only if the respondent refuses to go with the two choices. In the rare poll in which an explicit "don't know" is available as a choice, the number selecting "don't knows" is manyfold higher.

Second, to forestall "don't know" type answers, the pollsters often feed information to the respondents, such as "This morning, Egyptian strongman resigned. What do you think ...?" Those people who are not aware of this development become aware of it by answering the poll! As Moore rightfully asserts, "Once respondents have been fed any information, they no longer represent the general public (p.146)

One measure of the failure of polling is that the respondents are found to have very low conviction on their expressed opinions. When asked whether they would be "upset" if the policy they oppose gets implemented, a surprisingly large number of respondents claim they don't care.

For the statistically-minded, I'd suggest jumping from Chapter 1 directly to Chapters 7-8 where Moore discusses some statistical topics. Chapter 7 contains some materials on low and declining response rates (80% in the 1960s to below 25% today), and an unresolved question of whether nonresponders are actually different from responders. Chapter 8 covers a variety of poll design issues, like how wording and order of questions affect response dramatically, and the rise of cell phones and the Internet. Seife, in his Chapter 4, has a nice treatment of the wording issue.

The other Chapters in The Opinion Makers details a sequence of examples illustrating Moore's main thesis. It forms a kind of chronology of major polling efforts in the recent past, including the controversies.

***

Both Seife and Moore look at pre-election polls that are used to "call" elections. They accept the "conventional wisdom" among pollsters that past errors are primarily due to "poor timing": voters change their minds between the time they were polled and when they cast their ballot. This is certainly plausible but in my view, they could have taken this line of thinking to its logical conclusion: that the average person doesn't treat our politics seriously enough to have any real opinions; positions they have are subject to change; thus, pre-election polls are a waste of time, possibly excepting the ones done within days of the elections. On exit polls, Seife does ask this question: whether our life would be diminished if exit polls didn't exist.

Both Seife and Moore lament that polls have become a "journalistic invention" used to create "pseudo-events" which allow journalists to write what they would have written without the polls. Moore believes the watershed was when the major polling companies started pairing up with national media outlets, losing their independent status.

***

Both Seife (Chapter 4) and Moore (Chapter 3) recount the lesson of the Literary Digest straw poll, which failed spectacularly in predicting Alf Landon to win the 1936 presidential election against Roosevelt by a landslide.  The lesson is that a large sample is necessary but not sufficient for an accurate prediction; because Literary Digest drew its sample from automobile and telephone lists – neither good was as ubiquitous then as they are now – the sample was biased against lower-income people who were more likely to vote Democrat.

Seife took this as an opportunity to explain two types of polling errors, technically known as bias and variance. Variance is the variability of results from sample to sample, and is captured by the margin of error of the poll. Bias is a systematic error caused by selecting a sample that does not adequately represent the population. This is really a lovely real-life example illustrating the concepts.

The narrative then took a wrong turn when Seife stated: "[Literary Digest] thought their sample size made their poll accurate to within a tiny fraction of a percent, when in fact it couldn’t be trusted within ten or fifteen points." (p.107)

Unfortunately, bias is not measured as plus/minus points. A good analogy is an archer. Archer V(ariance) sprays his arrows around the bull’s eye, frequently missing the target but on average, his aim is at the bull’s eye. Archer B(ias) hits the same spot most of the time but because of a slight tilt in his motion, his arrows consistently hit a spot away from the bull’s eye. We can estimate V’s error using a concentric circle around the bull’s eye but to describe B’s error, we use the distance between the bull’s eye and the spot that B typically hits.

In the case of the straw poll, the concentric circle is negligibly small because of the enormous sample size; the bias should be measured as a shift downward of the percent voting for Landon – it is an error in one direction.

 ***

Seife’s coverage is broader than polls. He argues that our politics has been corrupted by bad statistics in a variety of other areas, including vote counts, voting schemes, gerry-mandering, census politics, ignorance of statistics by Supreme Court justices, and government propaganda, (Chapters 4 to 8). Both Seife and Moore stress that both political parties are guilty, and give ample examples to demonstrate their wayward behavior.

The second part of Proofiness covers materials that are less trodden by others, and is well worth the read. 

***

A rich open question in polling is whether nonresponse constitutes bias. Moore seems undecided on this point as he cites nonresponse as a source of bias when explaining the spectacular failure of the Literary Digest 1936 straw poll while also describing research by Pew claiming that nonresponders are not meaningfully different from responders. This subject is, unfortunately, almost impossible to study because by definition, nonresponders don’t want to talk to us.