Never a failed test fails the statistical test

Lance Armstrong is being accused of doping by former teammates, many of whom directly helped him to win 7 Tour de France championships. According to this Yahoo! report, Armstrong's response, via Twitter, was:

20+ year career. 500 drug controls worldwide, in and out of competition. Never a failed test. I rest my case.

***

I don't know if Armstrong doped or not. Given that his racing days are years in the rear-view mirror, there is little chance we will ever have direct evidence either way.

However, as I pointed out in Chapter 4 in Numbers Rule Your World, "never a failed test" is not a great basis on which to rest one's case!

We have quite a few examples of athletes who never failed any drug test during their competitive careers but later confessed to doping. Marion Jones and Bjarne Riis are two examples I used in the book. Why is this the case?

The sad truth of steroid testing is that most dopers do not test positive. A recent example (discussed here) illustrates that about 50% of dopers would pass the test -- and that was measured in a controlled laboratory experiment. The reason for such high false negative rates is that the anti-doping labs want to minimize the chance of a false positive error. The underlying statistics dictate a trade-off between false positives and false negatives; the harder one tries to eliminate false positives, the more false negative results will be produced!

I call for more false positives in drug testing in this post.

***

The media has gotten the statistics totally backwards.

On the one hand, they faithfully report the colorful stories of athletes who fail drug tests pleading their innocence. (I have written about the Spanish cyclist Alberto Contador here.) On the other hand, they unquestioningly report athletes who claim "hundreds of negative tests" prove their honesty. Putting these two together implies that the media believes that negative test results are highly reliable while positive test results are unreliable.

The reality is just the opposite. When an athlete tests positive, it's almost sure that he/she has doped. Sure, most of the clean athletes will test negative but what is often missed is that the majority of dopers will also test negative.

We don't need to do any computation to see that this is true. In most major sports competitions, the  proportion of tests declared positive is typically below 1%. If you believe that the proportion of dopers is higher than 1%, then it is 100% certain that some dopers got away. If you believe 10% are dopers, then at least 9 out of 10 dopers will test negative!

***

While researching the book, I learned that trying to catch dopers is extraordinarily hard. Here are some reasons why a doping athlete could get a negative test result:

• he's using a drug that is also produced naturally by the body, which means that the test needs to detect "unnatural" levels of the chemical, rather than the presence of a foreign substance
• he's using a new drug that has no test yet
• he's using "masking agents" that hide the performance enhancing drugs
• he's used steroids during training but not during competition (many sports don't conduct out-of-season testing, and even if they do, you can't possibly test all athletes all the time)
• he's received a "therapeutic use exemption" (I'm not sure why the sports bodies have never disclosed which athletes have been allowed to use which drugs based on TUE)
• he's following a drug schedule that attempts to evade testing

Those are not the only reasons. Notice that all of these tactics are not replicable in a lab so the accuracy rates reported by labs are almost for sure overly optimistic.

***

Armstrong's latest accuser is Tyler Hamilton, who features in my book. When I first started writing, he had failed one test, got banned, came back, and failed another test. Even after the second failed test, he had maintained his innocence. By the time I finished writing, he had come back yet again and failed a third test, upon which he retired.

One other statistical point of note: test results for a given individual are not necessarily "independent"! It's not too surprising that people like Hamilton kept failing tests while other dopers like Marion Jones kept passing tests. A failed test indicates that the doping program of that athlete isn't foolproof, and we should expect that athlete to have a higher chance of failing again in the future. A negative test, by contrast, may indicate that the doping program is robust against the testing regime. (In Jones's case, she was using a new designer steroid that took years for the test labs to notice.)

 

 

Comments

[Ken]

The problem with professional cycling is that so many riders have used drugs that for someone to win consistently they either have to take drugs or be so much better than anyone else that they can win without drugs. A low dose of EPO will put someone just that bit ahead of the pack.

[Tord Steiro]

I wonder what is the chance of a false positive?

If you take 500 tests, shouldn't the chance for at least one false positive be quite large?

So when there are no false positives, perhaps that indicates tampering with the results, or something similar?

I am just curious. Statistically, you should get false positives, one or the other, if you test enough times. Regardless of how strict you are exchanging the risk for false positives with the risk for false negatives.

I am simply considering statistics here as I have little interest in cycling in general.

[Travis]

Lets set up a toy situation to see how strong Armstrong's argument is, based on the fact that he "never failed a test" of 500.

Suppose he /was a doper/ and that each test is independent of the next, conditioned on the fact that we know he is a doper (not an unreasonable assumption). Then, suppose that the test only correctly identifies a doper 1% of the time. That is, 99% of dopers go uncaught. Then, since each test is independent, the probability of Armstrong passing every test is (0.99) * (0.99) * ... 500 times. (0.99)^500 = 0.00657. So, even if there is only a 1% chance of identifying him as a doper in a single test, after performing 500 of them there is only a 0.6% chance that he passed all the tests.

Your comments about the tests not being independent don't make very much sense. Clearly, even if the tests are independent but identically distributed, information about some will tell you something about new unseen tests. Imagine tossing an unfair coin 500 times. The fact that you can learn the weighting of the coin from the first 499 tosses does not make the 500th toss dependent.

Anyways, it seems like a pretty strong argument to me. If you were a betting man and you bet that he was a doper then you'd have a 99.4% chance of losing.

[Jason]

@Travis: But never failing any number of tests is no kind of exoneration, as Kaiser points out, if any the bullet points he lists are true of a situation. E.g. in the case where you're using a drug with no test, your equation is not (0.99)^500 = .6%, but (1.00)^500 = 100%. If you're using a measure that has no validity, it tells you nothing.

[Travis]

@Jason: Of course, if your tests are unable to detect whatever drugs he is using then it is clear that the chance of him failing even one of any number test trials is zero (barring false-positives).

The point I was trying to make is that, even if your tests are highly ineffective-- with a success rate of only 1% (but not zero)-- not failing any of five hundred tests is a strong sign that you do not use what the test is testing for.

The bulk of Kaiser's article is about how tests are designed to have fewer false positives and that this fact makes Armstrong's 500 negative tests meaningless. And that is garbage.

I can't stop you from believing that he has magical, undetectable drugs.

[Kaiser]

Travis: That's why the independence assumption is the key here. If I were a doper, and I pass the test, this tells me that my doping regimen is pretty good; if I pass two tests, it increases my confidence that my doping regimen is good; the more tests I pass, the more I feel good about the expertise of my doping advisors.

Another way to think about this is the fact that every athlete who have confessed and/or failed a positive test will have had a long string of negative tests prior to failing. Unless one believes these athletes (like Andy Pettite) who claim that the only time they took steroids was the time they got caught, it is very difficult to make the case that a string of negatives means much.

[Splint]

Well, if someone is accusing you of doping, and you haven't and you have passed 500 some doping tests, exactly what else is left to you to prove your innocence? It's not possible.

I sort of understand your point about passing a drug test isn't proof you don't take drugs, but really, what else do you want from the guy?

[Charles]

I stumbled upon this blog today while looking for other things. I'm loving the lucid discussion of statistics, but want to comment on something else about this post.

Athletes (cyclists, tennis players, golfers, footballers, etc) are entertainers. They are entertainers in exactly the same sense as rock musicians, classical musicians, actors, models, dancers, etc.

We expect our entertainers to do lots of "unnatural" things to their bodies in order to enhance the entertainment value of their performances - botox, breast enhancement, tooth whitening, constant dieting, etc. And our entertainers do these things to enhance their performances.

Cyclists doping are enhancing their performances. It's all entertainment, folks. They're entertainers. If doped cyclists make for a more entertaining race, then it's their prerogative to dope all day long for all I care.

[Tom]

A recent article in Science Daily on testing baseball balls and bats also makes this point:

"The issue of juiced balls emerged in 2000, when the first two months of the major league baseball season saw substantially more home runs than the same time the previous year.
"...
"They found the balls' coefficients of restitution -- their ability to bounce -- were nearly identical. In retrospect, Smith speculates that it may have been the players, not the balls, that were juiced."

http://is.gd/M5q3lp

[Tom]

This post stuck in my head and bubbled to the top again yesterday with the Casey Anthony verdict. We'll never know the truth about that situation either. But the outrage over the Anthony verdict is an important reminder that we don't live in a binary world where "found not guilty" automatically translates into "must be innocent."