Big Data, Plainly Spoken (aka Numbers Rule Your World)

Losing sleep over a study of sleep deprivation

Economist Catherine Rampell who writes for NYT's Economix blog wondered why economists are rated as the #5 most sleep-deprived profession in the country. (link)

She noted:

Economists in academia, at least,  seem to have flexible schedules that should let them get lots of sleep. Maybe a lot of them are grad students scrambling to publish, publish, publish. Or maybe there are a lot of folks like Larry Summers who prefer allocating more hours for work.

Reader Jordan G. who alerted me to this article said what I think any reasonable data analyst would notice right away:

The authors have listed the "most well-rested" versus the "most sleep deprived" but an inspection shows that the most well-rested occupation gets only about 20 minutes more sleep on average than its sleep-deprived counterpart. That really isn't a whole lot of extra sleep - in fact, there's really not a whole lot of variation in the data presented. I can't help but think that this study is telling us nothing except that most people in these occupations have an average sleep time of about seven hours and some change. Using this chart, is it really meaningful to say that loggers will be more well-rested than home health-aides?

I'm not sure why Rampell would comment on what she called "a quirky little e-mail from Sleepy’s, the mattress chain". This sounds like one of the many emails bloggers get from all kinds of people, typically filled with infographics (see this war against infographics post) and inevitably commissioned by self-interestsed commercial interests, that show all manners of amateur analyses.

Or it shows an unhealthy interest in tiny effects, which I discussed further here.

Two black eyes for science, what a shame

Two cautionary tales appeared in press recently, serving notice to all "data scientists" (as statisticians are fancifully called these days). It's hard work to earn the status of a "science".

Via the New York Times comes the story of Dr. Robert Spitzer (link). As a young psychiatrist in the 1970s, he successfully pushed the profession to narrow the definition of homosexuality as a disorder. He observed wrily that many gay people are happy, and therefore only those who are depressed should be diagnosed.

In 2001, he presented new findings that show that homosexuality can be "cured" by reparative therapy. This was his method:

He recruited 200 men and women, from the centers that were performing the therapy, including Exodus International, based in Florida, and Narth. He interviewed each in depth over the phone, asking about their sexual urges, feelings and behaviors before and after having the therapy, rating the answers on a scale.

He then compared the scores on this questionnaire, before and after therapy. “The majority of participants gave reports of change from a predominantly or exclusively homosexual orientation before therapy to a predominantly or exclusively heterosexual orientation in the past year,” his paper concluded.

He strenuously defended the study for years, after it got published in his friend's journal without going through the typical peer-review process. (The article was published with commentaries by peers, which according to the NYT, were "merciless".)

At 80, he is coming forward to apologize and retract the study. Bravo to him for doing this. But one wonders how the industry of science failed to expose this failing much sooner. Is it because of the stature of the researcher? Is it conformance? Is it because he circumvented the usual peer-review process? ...

The reporter said the biggest problem with the study was self-interested subjects lying about sensitive issues like these. Actually, no. The biggest problem is the absence of a control group - gay men and women who did not receive such therapy. It boggles my mind that a study done in 2001 would have only cases and no controls. The case-control methodology has been in use since the 1950s/60s.

***

If you think that was bad, hold your nose before you read this Wall Street Journal article about cancer studies (link).

Here is a sample of the stinky sentences (my italics in all cases):

After publishing a paper on a rare head-and-neck cancer, [Dr. Mandic] learned the cells he had been studying were instead cervical cancer...

Dr. Mandic entered a largely secret fellowship of scientists whose work has been undermined by the contamination and misidentification of cancer cell lines...

Cell repositories in the U.S., U.K., Germany and Japan have estimated that 18% to 36% of cancer cell lines are incorrectly identified.

Dr. Tarin has spent 25 years working with that cell line--or so he thinks. A body of research suggests that MDA-MB-435 isn't breast cancer; many scientists now believe...[it's] melanoma... Dr. Tarin disagrees.

The prevailing attitude [among scientists] is that the other lab's cell line may be contaminated but not mine.

Nearly 40 years later, ... found 1,000 citations of the same contaminated cancer lines revealed in Dr. Gartler's 1966 findings, which have since been replicated many times using more advanced techniques. "They [the scientists] are either crooks or stupid."

As data scientists like to say, "garbage in, garbage out". But who among us is courageous enough to voluntarily consign decades of our own research to the dustbin?

 

How many millionaires?

From Reuters, via Yahoo!

Facebook's challenges ahead as it goes public

Given that Facebook is on the verge of going public -- the IPO supposedly over-subscribed, leading to an increase in the projected price of the offering, this article from Reuters is big news: GM, the third largest advertiser, will not spend anymore money on Facebook ads.

This should rightfully get Facebook's attention. We have previously discussed the incredibly low clickthrough rates of Facebook ads, and speculated as to why consumers don't respond to these ads. (link)

As marketers start to ask questions, Facebook is going to have to give some answers. Here is a starter's list:

• Why can't advertisers know which users viewed messages? Without views, advertisers are forced to look at "reach", which measures how many people are shown the message. Showing is a far cry from viewing.
• What is the decay factor that should be applied as one traverses down the social graph? The value of a friend of a friend of a friend should not be the same as the value of a friend.
• What is the point of collecting "Likes"? Do "Like" campaigns generate any new customers or increase brand awareness or does it just allow loyal customers to express their love of a brand they already know about?
• What is the incremental value of online advertising? The focus on the clickstream is a red herring (not just for Facebook). The clickstream is a description of the path taken by a customer to reach the buy event. For most companies, especially for large brands like GM, there are dozens of paths a customer could take to reach the buy event. The online channel must prove itself by showing that if the clickstream path is blocked, the sale is lost -- in many/most cases, the customer would just shift to a different path (e.g. click on an email, go to the store, call the 800 number). Said differently, the clickstream is not causal evidence.

Online marketing probably works but with the data and reporting currently available, we don't know if this is the case. I gave a talk recently making this point, that online marketers need to do a better job showing return on investment. The medium promises to be more measurable and more accountable but thus far, that is a promise and nothing more. GM's action pretty much speaks to this point.

***

Credit Facebook for extracting the highest price for its IPO - could it be downhill from here?

What is the (expensive) miracle drug that cut colon cancer deaths by 20%?

Andrew Gelman linked to this great reporting by Reuters on U.S. healthcare economics. It's a must-read. Be patient, and read through to the end even though it's a long piece.

Andrew cites statistician Don Berry who explains what "lead time bias" is, and why survival time is always the wrong metric to use in evaluating health outcomes. Survival time is the time from diagnosis to death. By doing more screening and diagnosing earlier, survival time will magically increase even if the patient's life expectancy stays put.

***

I ignored Andrew's warning and spent some time reading the Philipson, et. al. paper (link). Time that I want back but couldn't. To save you the trouble, I will discuss a few gaping holes other than the howler already identified by Berry - there are many other less significant issues.

The title of the paper purports to address the following "causal" relationship:

The reader immediately discovers that the authors analyzed a different "causal" relationship:

It may appear that the substitutions are harmless: spending on cancer care is a proxy for overall healthcare spending; survival gains for cancer patients is a proxy for overall health benefits. The authors hid the useful information in the Appendix (available online). In Table 3, we learn that spending on cancer care is only a single-digit percentage of total health care spending in almost every country. Besides, the total deaths by the 13 types of cancers counted in their study constitute only 31 percent of the total cancer deaths in the U.S. (using the 2011 statistics from this report - PDF). This list of included cancer types excludes the biggest killer (lung, over 150,000 deaths) while it includes testis which caused 350 deaths in 2011.

So, even if the analysis is correct, the result cannot be generalized to talk about cost and benefit of all health care spending. This is an instance of "availability bias": even though cancer makes a lot of news, most health care spending has nothing to do with cancer, and so we can't use cancer care spending as a proxy.

***

In assessing the value of cancer care spending, the authors decided to use a modeled change in death rates, rather than the actual observed data. Neither in the paper nor in the appendix is the actual model reported, nor is there any information on goodness of fit. However, we don't need to know the model to know it doesn't fit.

Take a look at the fourth column of Table 1 in the Appendix. This column shows the predicted deaths avoided or incurred in the U.S. (given the additional spending in the U.S. relative to "Europe").

Let's do a sanity check on these numbers. For colorectal cancer, the model claims that the extra spending has avoided 282,000 deaths over the 23 years (1982-2005), or roughly 12,300 deaths per year. According to the cancer death statistics, about 50,000 deaths from colon cancer actually occurred in the U.S. in 2011. That means the model claims that colon cancer deaths would have been 25% higher were it not for the extra spending. What is the miracle drug that caused this gigantic improvement? What prevents this amazing new treatment from crossing the Atlantic?

Maybe you believe in miracles. Then, take a look at stomach cancer. Here, the negative number seems to imply that the additional spending has induced 225,000 stomach cancer deaths over 23 years. That sounds really horrifying. Given that stomach cancer killed 10,300 Americans in 2011, the model claims that the extra spending has doubled the number of deaths from stomach cancer!

Simply put, their model makes no sense.

***

Now, go back to Table 3 in the Appendix and read the note. It says that missing data for percentage of health care spending that is cancer related is imputed as 6.5% (30% higher than the U.S. assumption of 5% which came from a totally different source), and we find that Iceland, Norway, Slovkia and Slovenia (40% of the countries) are all imputed. 

The problem here is that the authors are not consistent in their treatment of missing data. In the main paper, they explain again and again that their sample of data is restricted by data availability (i.e. they didn't impute values for missing data). For example, the choice of the 10 European countries is because "only ten reported data consistently over the 1983-99 period". This means no Italy, no Spain but you have Wales and Scotland (but no England), and also Slovakia and Slovenia (why are they comparable to U.S.?).

Why those particular 13 cancers? Because "data were consistently available from both the European and US survival databases". This means including testis cancer and excluding lung cancer. Insteading of imputing values for lung cancer, they just drop the cancer type that causes the most deaths.

Why look at survival differences only for patients diagnosed from 1995 through 1999? You guessed it. It's because only in those periods can they find consistent data.

Given that they use models throughout the research, and they imputed values for proportion of spending on cancer treatment, they could have tried to impute values in these other decisions, and then the result could perhaps be generalized. 

Dropping data because some variables are missing should be justified clearly. It's too easy to cherry-pick your dataset this way.

***

How about another non-sensical assumption? The average value of an additional year of life of someone who's dying from cancer is $150,000 to $360,000. They describe this as "standard figures for an extra year of life" and call the lower end of the range "conservative". Only 5% of Americans earn over $100,000 per year. The median personal income is less than $40,000. (From Wikipedia, for 2004, I think). Enough said.

***

It's sad that this paper gets publicity only because it makes a conclusion that is against "conventional wisdom". The clear evidence so far has been that while the U.S. spends twice as much on health care as other "wealthy" nations, our life expectancy is lower, and the bottom of the class. (See here, for example.)

The chart shown on the right is as clear as it can be. (I discussed this chart on Junk Charts.) The situation with science journalism is very dire, in my opinion, when outlets are chasing clicks and sales by publicizing bad studies that have eye-catching headlines.

 

 

Dropping out dropouts is a mischievous act

The following throw-away lines in a Wall Street Journal article about the "return on investment" of getting into college debt (what an idea) are the most important ones:

The report [by the College Board] also doesn't account for dropouts or extra college years. Only 56% of students who enroll in a four-year college earn a bachelor's degree within six years, according to a report last year by the Harvard Graduate School of Education...

PayScale, a Seattle data firm, examines the links between pay and variables like colleges and majors. Its analysis, which also ignores dropouts but accounts for students who take longer to complete their degrees, ...

I cut that off since I've heard enough. How can they get away with ignoring dropouts when they are assessing the return on investment of college debt?

Imagine a cohort of 10,000 students starting college on debt. By year 6, which apparently is when they stop counting, 4,400 have not graduated, either because they dropped out or they are still in school. Both of these groups are likely to have the lowest return on investment of those in the cohort. Most of the dropouts won't be getting college-graduate jobs which pay higher. Those still in school are probably troubled students who if they do graduate later, would also earn less - even if they are equally qualified, they would earn less by value of time.

Given this reality, the analyses by the College Board and by PayScale would "ignore dropouts" as if they didn't ever exist. In other words, they only look at the 5,600 not the 10,000. This means whatever return on investment they compute will be exaggerated.

***

Technically, this is an example of survivorship bias. The sample being studied does not contain "non-survivors", in this case dropouts, so it doesn't generalize properly.

Also, the data is censored in the sense that the observation window is not enough for us to know what would happen to those people who are in college longer than 6 years. This is a common feature of such data sets; you'd want to do something about it, not just ignore it.

There are in fact many other problems with this type of analysis. Here's another crucial one: the counterfactual for reasoning whether debt is the cause of higher future wellbeing is not having debt. In other words, any such analysis must tell us what would happen if the same students were able to complete college without having to incur debt. Based on what the WSJ reporter said, I don't think this is how they framed the problem.