Pills: victims of sloppy statistics?

According to the Daily Mail, a Canadian study proves "the peril of the sleeping pill: users a third more likely to die early". How should we read such an article? What should we look for to assess its credibility?

***

Start with these sentences in the middle of the article:

Dr Belleville analysed 12 years of data on more than 12,000 Canadians.

When all other factors were equal, death rates were found to be significantly higher among sleeping pill users and those taking tablets to ease anxiety.

And later, this sentence provides a bit more information:

The data includes information on people aged 18 to 102, surveyed every two years between 1994 and 2007.

So first thing is note is that this is not a randomized controlled trial but an observational study; also, the data came from self-reporting in surveys, and not direct measurements. Both these details should cause us to widen the margins of error. (In an RCT, we would divide people at random into two groups, one group will be instructed to take sleeping pills and the other group would take a placebo.)

Now, note the use of "when all other factors were equal". If they conducted an RCT, then this statement would be perfectly reasonable because random selection makes it believable that the test group and the control group had the same mix of males/females, age group mix, education level, income distribution, etc. etc. When this is an observational study, making such a statement is an overstretch. They would need to prove why we believe that people who choose to take sleeping pills are comparable to those who choose not to take sleeping pills. It is hard to believe that "all else is equal" between these two groups.

Now, notice the little bombs that drop along the way:

Crucially, the study did not distinguish between those who were heavy users and those who only took them occasionally.

Nor it seems they distinguished between dosage or brand of pills:

Pills used ranged from over-the-counter antihistamines to powerful prescription-only preparations such as Valium.

This is the issue of Chapter 3 of Numbers Rule Your World. They are lumping together into one average a wide variety of people with very different characteristics, and this may hide differences between subgroups. For example, the observed increase in risk may be fully explained by those who take powerful sleeping pills at high frequency but when lumped together with the light users, we can't tell.

***

This particular sentence raises more questions than it answers:

Pill takers were more likely to succumb to every type of illness, from parasites to cancer.

Frankly, reading this, I'd hazard a guess that people who take sleeping pills are less healthy than people who don't take sleeping pills. They are a self-selected group of people, and this explains why they succumb to every type of illness -- these diseases do not have to have anything to do with taking the pills themselves.

Analogously, if we compare the average income of graduates between a top public college and an Ivy League school, we might conclude that the Ivy League school provides a better education; however, most likely, the Ivy League school accepts students that are better prepared to begin with, and so we can't tell if the higher income is due to a better education or better preparation prior to college.

In other words, there is a high likelihood sleeping pills are not the killers they are after.

***

By the end of the article, the journalist has declared story time. How can this finding be rationalized so that readers can believe it? Here are some of the stories they told us in the article (strangely repeated)

Sleeping pills and anti-anxiety drugs affect reaction time, co-ordination and grogginess, which raises the odds of falls, she said.

Tablets might also suppress the respiratory system, which could aggravate breathing problems during sleep, particularly for those with heart problems.

In addition, effects on the brain could affect judgment and moods, increasing the risk of suicide.

Note that I am not against speculation per se. It could lead to real progress. I am against making such assertions without any attempt to test them. I am especially against elaborate multi-step assertions. For every story one can come up with, someone can concoct a different story with the opposite ending.

Comments

[Ken]

Most stats texts leave the impression that you can just throw a dependent and several independent variables into a regression program and then get a sensible result, ignoring the fact that the world can be fairly complex. Students who are interested in doing proper analysis seem very interested in discussing some of the stupidities that can happen, and it should be covered more.