Trust no one when it comes to the numbers

I keep saying we need to develop some statistical sense because every day, other people are pulling wool over our eyes. Here's another example.

In short, if you are taking Avandia (diabetes drug), stop right now. The FDA will never get its act together; it is too timid to shoot down a billion-dollar product.

***

Here, the Washington Post reports on two new studies showing that 1 out of about 50 people taking Avandia will get a heart attack (solely due to taking the drug). This is known as the "number needed to harm", and measures the level of side effects, in this case, potentially fatal, of a given drug.

When I saw this, I immediately went in search of the other important number... the number needed to treat, i.e., if 100 people take Avandia, how many will receive the advertised benefits from that drug. Shockingly, NNT for Avandia is known to be 1000+, meaning that only 1 out of 1000 patients taking the drug would see a benefit. (In this article, they say it is "close to infinite".)

So among those 1000 who take the drug, we can expect the drug to induce 20 additional heart attacks. Or put differently, if you take Avandia, you have a 0.1% chance of getting better but a 2% chance of getting a heart attack. Or put differently, the chance of having the heart attack is 20 times higher than the chance of getting better. There you go, make a wise choice.

***

There is some handwaving denial from a GlaxoSmithKline spokesman telling us that "many other, more reliable studies have found no evidence the drug is unsafe". I'm assuming that he/she is referring to clinical studies that were done years ago to gain approval for the drug.

The two new studies are (1) a meta-analysis that combines 56 prior studies, and (2) a comparative study that shows Avandia to be much more likely to cause heart attacks than a similar drug called Actos. (I haven't read these in detail so I can only talk to the general statistical principles.)

There is no doubt that the clinical trials were "better designed"; however, the new studies have total sample sizes of 35,000 and 200,000+. The sample size in the clinical trials must have been only a fraction of that.

Any post-marketing trial cannot be well designed because you just can't force someone not to take Avandia once it's on the market. However, this is not a black and white issue. We can still learn things from so-called "observational studies" (non-randomized). In particular, having very large sample sizes make the results a lot more reliable. I do not share the worries of the GSK spokesman.

Comments

[Jay]

Please clarify your last sentence. It would seem that GSK is pretty hosed.

[Kaiser]

Jay: GSK's spokesman is "worried" that the new studies do not have the same credibility as their own clinical trials. I'm sure he/she is very worried about the prospect of Avandia! Sorry - the tongue in cheek doesn't translate well on line.

[J]

You should probably qualify your statement about sample sizes; a meta-analysis across studies involving 200k people total is not the same as one study of 200k people (I guess in theory they could be equivalent, if each study was conducted exactly the same - but that's not really a "meta-analysis").

The takeaway is that large sample sizes doesn't (always) trump design considerations.

[Kaiser]

J: agree that larger sample size is not always better. However, in this case, I believe that the spokesman is setting up a straw man. There are practical limitations to how large the sample size could be for a clinical trial. If multiple trials are available, a meta-analysis can help pool the information. It's by no means foolproof but I'd believe the pooled analysis of 56 trials more than the one trial GSK deems to be "well-designed".