According to the Daily Mail, a Canadian study proves "the peril of the sleeping pill: users a third more likely to die early". How should we read such an article? What should we look for to assess its credibility?
Start with these sentences in the middle of the article:
Dr Belleville analysed 12 years of data on more than 12,000 Canadians.
When all other factors were equal, death rates were found to be significantly higher among sleeping pill users and those taking tablets to ease anxiety.
And later, this sentence provides a bit more information:
The data includes information on people aged 18 to 102, surveyed every two years between 1994 and 2007.
So first thing is note is that this is not a randomized controlled trial but an observational study; also, the data came from self-reporting in surveys, and not direct measurements. Both these details should cause us to widen the margins of error. (In an RCT, we would divide people at random into two groups, one group will be instructed to take sleeping pills and the other group would take a placebo.)
Now, note the use of "when all other factors were equal". If they conducted an RCT, then this statement would be perfectly reasonable because random selection makes it believable that the test group and the control group had the same mix of males/females, age group mix, education level, income distribution, etc. etc. When this is an observational study, making such a statement is an overstretch. They would need to prove why we believe that people who choose to take sleeping pills are comparable to those who choose not to take sleeping pills. It is hard to believe that "all else is equal" between these two groups.
Now, notice the little bombs that drop along the way: