Numbers Rule Your World

So much medical research is pretend-science

Medical researchers are somehow allowed to get away with statistical murder. It upsets me to read the article in Forbes titled "Pet Owners May Have Lower Risk For Heart Disease." (link)

This article takes the form of many other similar articles that purport to find an association between some risk factor and a common disease. Note they always use the weasel word "may". If you see this word, and immediately trash the article, you won't be any worse off than the people who read the article.

The weasel word will inevitably lead to "causation creep". There it is in sentence 3:

Dr. Glenn Levine, a professor at Baylor College of Medicine, along with his colleagues recommend that people have a dog or cat as a way to lower the risk of heart disease.

If the researchers only discovered an association, how is it that owning a dog or cat will lower your heart disease risk? If that were true, then the researchers have discovered a cause, not a correlation.

Such drivel will inevitably lead to "story time", which is a string of make-believe theories that have no support whatsoever in the empirical research. There it is in sentence 4:

[Having a dog/cat lowers heart disease risk] occurs by people becoming more physically active, which ultimately can lower blood pressure, cholesterol, as well as lead to lower levels of stress.

While there are some studies (most of it of excretable execrable quality) that report correlation between pet ownership and heart disease risk, you will not find studies that link all of those items listed above.This means none of those items are proven. They are just theories.

What confuses people is that the transitive relationship does not usually hold in statistics: if you have A -> B in one study, and B -> C in another study, you do not know if A -> B -> C holds. In fact, A > B and B > C may not mean A > C, even in the same study (this is due to sampling variability).

***

After all of that, the last few paragraphs of the article serve to invalidate all that went before. Too bad the majority may have stopped reading by that point. (I encourage you to stop reading at the word "may" in the title.)

Dr. Levine came back to say this: "it may be simply that the healthier people are the ones that have pets, not that having a pet actually leads to or causes a reduction in cardiovascular risk." Then why did he and his colleagues tell us that having dogs or cats is a way to lower heart disease risk?

***

The American Heart Association has a hand in spreading this misinformation. In this paper, they just list all kinds of studies without regard to their quality.

Also notice that they never said owning pets will prevent heart disease. Reducing heart disease risk is not the same as reducing heart disease. In fact, the only two studies of "survival in people without established cardiovascular disease" both reported no statistically significant relationships between dog owners and non dog owners. I wouldn't trust any of these studies unless you pull out the study and carefully examine who the subjects were; almost all of these are observational studies with no control groups (non-pet-owners is not a control group).

***

What's wrong with "story time"?

Go to this blog where they have a laundry list of statistics about pet owners. I can make up a story or two based on any of the following data:

• Families with children are much more likely to have pets than families without children
• The age group most likely to own a dog is 18-24 year olds at 58%.
• 78% of dog owners own their own home. 18% rent.
• Blacks seem to be less likely to own pets because they are more likely to live in densely populated urban areas with less room for pets.

Occupational hazards in data science

An interesting episode is developing in econometrics over the very high profile Reinhart-Rogoff paper that was heavily cited as a source to "prove" that high levels of national debt impede growth. It appears that that result was based on a combination of spreadsheet errors, and bad assumptions.

1. Andrew Gelman has a great discussion here. His main concern is ethics of data analysts. This is a very important point - any experienced data analyst knows that it is extremely easy to make mistakes; in fact, if there is a data analyst who has not made errors that subsequently have to be corrected, or disseminated results that subsequently have to be revised or even retracted, one can be sure that the data analyst has not done a lot of analyses. (I'm excluding those who only work with simulated data.)

Errors come from many sources, and it appears that Reinhart-Rogoff managed to touch several important ones.

a. Processing errors - in the course of an analysis, data is often transported from system to system. Even on a spreadsheet, the data may be copied and pasted from one page to another, or formulas are set up to link from one area to the next. For a team project, the spreadsheet or dataset may pass from one person to another. A tiny moment of inattention could easily ruin the data. It appears that Reinhart-Rogoff dropped some data on the floor. I should note that "Big Data" makes this problem a hundred times harder to solve. If you only have one spreadsheet, you can in theory verify each number at different stages of the analysis. Not if you have millions of rows and thousands of columns in a dataset.

b. Bad assumptions - any analysis will contain assumptions, mostly because the data has problems or holes. Making assumptions is not itself bad. But some assumptions will turn out to be bad. In this case, the assumptions were badly explained. Critics noted that some data points were deliberately dropped from the analysis, and the conclusion was sensitive to those dropped data. Based on these reports, Reinhard-Rogoff have not convinced these critics why those data points should be dropped.

c. Bad data -the biggest reason for bad analysis is usually that the analyst does not know how the data came into being. Somebody else collected the data (or a machine did). The data was purchased from a third-party. All kinds of hidden assumptions will now affect the analysis. For example, you analyze credit-card transaction data to try to understand consumer spending patterns. You see that the billing decreased. If you're not careful, you'd conclude that consumers are spending less. But there are many other possibilities. For example, maybe credit-card companies increase their fees and then merchants set higher minimums, forcing consumers to pay cash. Frequently, you learn about this type of extraneous factors after you already disseminated the analysis.

 

2. Paul Krugman's response is also worth reading. I especially want to draw attention to his last point, which is what I have called the "causation creep" (link):

Finally, while they acknowledge the issue of reverse causation, they seem very much to be trying to have it both ways — saying yes, we know about the issue, but then immediately reverting to talking as if debt was necessarily causing slow growth rather than the other way around.

In their response to the critics, Reinhart-Rogoff's key point was "We were only arguing association, not causality." (see here and here.) I'm sure we'll find specific statements to that effect in the paper but all too often, the discussion of the implications of such research includes statements that assume the cause-effect relationship, exactly what Krugman was complaining about.

This problem is not just found in economics, although I see it a lot with the Freakonomics style of thinking. It is also very common in the medical literature, and other fields in which observational data is used. The usual ploy is first acknowledge that the data could not prove causality ("we found an association between sleeping less and snoring; our data does not allow us to prove causation."), then quietly assume that the causal link is there, and wax on the implications ("if you want to snore less, sleep less.")

 

Predictive modeling

Love this xkcd comic (link). Would have been even better cut to the final third. Great food for thought for predictive modelers.

A few years ago, some economists wanted to change their last name because economists with last names beginning close to the front of the alphabet have greater chance of winning the Nobel (according to their research). (link to PDF)

Predictive hiring models, and Chapter 2

This Wall Street Journal article about predictive modeling in the workplace is well worth reading. Your reaction may vary from excitement to stomach churning. (link)

As with most reporting on analytics, the author never addresses the issue of model errors. For example, this article started out with:

When looking for workers to staff its call centers, Xerox Corp. used to pay lots of attention to applicants who had done the job before. Then, a computer program told the printer and outsourcing company that experience doesn't matter.

The fun may be squeezed out if we want a more balanced story. I get that. But surely, someone should ask if this "computer program" makes no errors. Because if you read this style of reporting, you would think every prediction is correct.

The reporter does get into the creepiness and the legal murkiness of using such models but those discussions hover over the unspoken assumption that the predictions issued by these computer algorithms using "big data" are infallible. It's one thing if the model excludes, say, women from being hired because well, every woman fails in that particular job; it's another thing if the model excludes women erroneously! (I can already see the "I took steroids but look, it didn't help." excuse.) Later in the article, we are told that a startup company has created the model of "an ideal call-center worker". The fact that there could be an "ideal" reflects a level of over-confidence in models that bothers on insanity. (And I'm someone who builds models for a living.)

As you read the article, bear these points in mind:

• In Chapter 2 of Numbers Rule Your World, I talk about models of correlations versus models of causation. In my view, when people's livelihood is at stake, a correlational model is not good enough. All these models described in the article are correlation based. The factors they use are things like commuting distance, and social networks. There is a serious danger of "causation creep" here. 
• An analogy: if you look at crime statistics, it is certainly true that African Americans are more likely to be criminals. If you build a crime prediction model, race will be a strong predictor even if it's not the only predictor. A lot of us are uncomfortable with this kind of racial profiling.
• A technical note: this type of models suffers from "rejection inference" (terminology from the credit scoring world). I didn't find space to talk about rejection inference in Chapter 2 but I might as well say something here. The output of such predictive models is used to make a hiring decision. The candidate can either be hired or rejected. If the candidate is hired, the company can track his or her performance and make a determination as to whether, ex-post, the hiring decision was good or bad. If the candidate is rejected, the company has no way of knowing whether the rejection was just or not. So when any company collects data to build a predictive hiring model, the dataset is already biased, as it does not contain candidates who would have been rejected.

 

The causation creep

If you're reading this blog, you probably have heard that correlation does not imply causation. Apparently, health-beat reporters and medical researchers in peer-reviewed journals know this too. They often explicitly warn us that the studies only show correlation, and they couldn't explain why the results are as they are. All is well if they stop there. But they don't.

All too often, they succumb to "causation creep". All of a sudden, they interpret the results in a way that presumes causation.

Here are two particularly egregrious examples that appeared on the Vitals blog at MSNBC.

In the first example (link), we are told:

Researchers followed 397 children from pregnancy (sic) through their first year of life, and found that those living with dogs developed 31 percent fewer respiratory tract symptoms or infections, 44 percent fewer ear infections and received 29 percent fewer antibiotic prescriptions.

And then comes the disclaimer:

the researchers acknowledged that couldn't account for all such factors [other than living with dogs that can explain the finding], and noted that they found a correlation, not a cause-and-effect relationship.

Finally, causation creep happens:

For heathlier kids, get a cat or dog, study suggests.

This is a cause-effect statement, there is no getting around it. It's saying that if you get a cat or a dog, your kids will be healthier. If the researchers truly believe that they found a correlation, then drawing this causal conclusion is unconscionable. Chances are, they just thought this is a fun thing to say (or at least the headline writer thinks so), allowing causation to creep in.

PS. The first person who commented on the article suggests that perhaps families that own pets also spend more time outdoors (playing with pets) and it could be the outdoor exposure that causes the observed effect. We don't know why but one thing is for sure: families who keep pets are not at all the same as families who don't keep pets.

PPS. It may be cruel at this point to point out that the so-called "cat" effect is 2 percent fewer antibiotics, which amounts to nothing. Also, no urban families were present in this analysis.

***

The second example (link) is if anything worse.

We are told:

Katzmarzyk and colleagues analyzed information from five earlier studies involving more than 167,000 adults that looked at the link between sitting and risk of dying from any cause over the next four to 14 years... About 27 percent of deaths in the studies could be attributed to sitting, and 19 percent to television viewing, the researchers said.

The disclaimer:

The researchers noted their study assumed a cause-effect link between sedentary behavior and risk of dying, which further research should validate, they said.

Finally, a blossom of colorful statements all of which presume causation, which they assumed without proof:

Sit less than 3 hours a day, add 2 years to your life (sic)

Reducing the daily average time that people spend sitting to less than three hours would increase the U.S. life expectancy by two years (sic)

The study adds to a growing body of evidence suggesting that sitting itself is deadly. (sic)

***

These researchers should realize they are not doing Freakonomics. It's true Levitt and Dubner also sometimes succumb to causation creep, especially in the more casual pieces. But in the original book, where they described how abortion policy could have caused crime to fall, they went through a vigorous explanation to rule out many other possible explanations. None of the studies cited here (or many other similar ones) have the care that is required to claim causation. We should note that the abortion finding has been debunked - that just goes to show how hard it is to prove causation with data that is conveniently collected.