Big Data, Plainly Spoken (aka Numbers Rule Your World)

Chance to ask me a question this Friday

I will be at Book Expo this Friday signing books at the McGraw-Hill booth. If you're in NYC, drop by and say hi between 11 and 12.

Yes, it's a new book!  The title is Numbersense: How to Use Big Data to Your Advantage (link). If you read my blogs, you already know where I'm going with this. How can we be smart consumers of data analyses in a world overflowing with data? It will be in stores in July. Between now and then, you can come back here to learn more.

Also, at 12:30, I'll be interviewed at the Shindig event by Peggy Sanservieri, who blogs at Huffington Post on book marketing. This is an online live chat event. Go to their site to register, and you'd have the opportunity to ask me questions.

(This is cross-posted on both blogs.)

Netflix adultery statistic debunked

Andrew Sullivan links to Maureen O'Connor (link) who picked up on Netflix's recent advertising pitch mischievously titled "Netflix adultery". Andrew highlighted this statistical result, with Maureen's interpretation, supposedly sourced from Netflix although I wasn't able to find the actual study. (I added the emphasis.)

In a study of 2000 American adults, 12 percent confessed to watching ahead on TV shows they were supposed to save to watch with their partners. Ten percent admitted to being the victim of Netflix adultery, which means either 2 percent are blissfully unaware of their partners’ indiscretions, or the cheaters are hitting multiple victims.

This last sentence is "story time". There is nothing in the study to prove or disprove this story. A third explanation -- which is more likely than the other two -- is that the 2 percent difference is pure noise. The margin of error of this study is about 1.3 percent plus or minus around each of those percentages.

The conclusion also contains a number of suspicious elements. First, it's unclear how exactly 2000 people responded. Second, not all adults have partners so either the 2000 people were pre-screened and not just any adult, or the percentages are biased by unattached people who could not have cheated.

Thirdly, Maureen made an assumption that both partners of each couple responded, which is highly unlikely. There are really eight types of couples: both cheated, which leads to four types depending on whether each partner confessed to the other; the first partner cheated or the second partner cheated, each of which leads to two types depending on whether there was a confession. Each couple may have returned one or two surveys. Given such complexity, the two percentages from the survey cannot be simply interpreted.

Finally, Netflix should disclose how the sample was selected.

I don't like 401(k) either

Felix Salmon hates the 401(k), and he explains his reasoning here. His strongest argument is the data, which shows that the first generation of retirees who grew up with these individual retirement savings accounts find themselves with meager retirement savings (average: $120,000, excluding those with zero).

I have always disliked 401(k), and here are some reasons:

I hate the myth of individual control. These accounts (just like health savings accounts and other similar things) are sold to us as an expression of individual freedom and choice. But it is a mirage.

First, it is never a good idea to chop up money into little pockets. The same banks who every day praise the benefit of diversifying our portfolios also lobby the government to establish these individual accounts. If you earn $60,000 a year, and get paid bi-weekly, then your paycheck is $2,500 less tax withholding. If you put 5% of your earnings into the 401(k), then every two weeks, you put away an extra $125. Imagine you then split the $125 into five or ten investments. It just doesn't make much sense.

And this doesn't even consider the per-trade fees that Charles Schwab, or E*Trade, or any of those firms charge us, as a percentage of the expected gains from your trade. Transaction costs are where individuals will always lose to funds.

Second, how many of us can truly beat investment professionals at the investment game? Most of us don't have the time or inclination to be monitoring our portfolio 24/7. The professionals have access to better information. They are able to react faster when something unexpected happens. In fact, our too-big-to-fail banks routinely have zero days of trading losses in a given quarter when they trade on their own accounts. (This does not mean these banks will perform the same when it comes to our money.) I'm aware that most professional fund managers are shown to do no better than some simulated simple-minded strategy. Is there a study that takes the investment performance of the average Joe and compare them to the average fund manager?

Third, moving from a defined benefit plan to a 401(k) is to shift risk from institutions to individuals. When we were sold the myth of individual control, we were also gifted the Trojan Horse of financial risk. Now, we are responsible for our own individual bad decisions. This is an important point. More people in the pool lowers the average risk.

Fourth, the 401(k) is regressive. The lower your salary, the smaller your biweekly contribution. The risk borne by the individual is higher when the principal is smaller.

If those are not enough, individual control is simply a lie, and becoming more so each year. Most of us find that the banks that run the 401(k) only allow us to put our money in one of the funds they administer. They charge a fee to put the money elsewhere. In one case, I was told if I really want to invest in funds not in the approved list, I must first buy one of the designated funds, and then transfer the money to the fund I really want. Needless to say, the in-and-out is not free. There is another company who matches my contribution but in company stock, which is the opposite of diversification; and if I want to transfer the money to a different asset, I have to pay fees.

It's a steep price to pay to enjoy that "individual control."

 

So much medical research is pretend-science

Medical researchers are somehow allowed to get away with statistical murder. It upsets me to read the article in Forbes titled "Pet Owners May Have Lower Risk For Heart Disease." (link)

This article takes the form of many other similar articles that purport to find an association between some risk factor and a common disease. Note they always use the weasel word "may". If you see this word, and immediately trash the article, you won't be any worse off than the people who read the article.

The weasel word will inevitably lead to "causation creep". There it is in sentence 3:

Dr. Glenn Levine, a professor at Baylor College of Medicine, along with his colleagues recommend that people have a dog or cat as a way to lower the risk of heart disease.

If the researchers only discovered an association, how is it that owning a dog or cat will lower your heart disease risk? If that were true, then the researchers have discovered a cause, not a correlation.

Such drivel will inevitably lead to "story time", which is a string of make-believe theories that have no support whatsoever in the empirical research. There it is in sentence 4:

[Having a dog/cat lowers heart disease risk] occurs by people becoming more physically active, which ultimately can lower blood pressure, cholesterol, as well as lead to lower levels of stress.

While there are some studies (most of it of excretable execrable quality) that report correlation between pet ownership and heart disease risk, you will not find studies that link all of those items listed above.This means none of those items are proven. They are just theories.

What confuses people is that the transitive relationship does not usually hold in statistics: if you have A -> B in one study, and B -> C in another study, you do not know if A -> B -> C holds. In fact, A > B and B > C may not mean A > C, even in the same study (this is due to sampling variability).

***

After all of that, the last few paragraphs of the article serve to invalidate all that went before. Too bad the majority may have stopped reading by that point. (I encourage you to stop reading at the word "may" in the title.)

Dr. Levine came back to say this: "it may be simply that the healthier people are the ones that have pets, not that having a pet actually leads to or causes a reduction in cardiovascular risk." Then why did he and his colleagues tell us that having dogs or cats is a way to lower heart disease risk?

***

The American Heart Association has a hand in spreading this misinformation. In this paper, they just list all kinds of studies without regard to their quality.

Also notice that they never said owning pets will prevent heart disease. Reducing heart disease risk is not the same as reducing heart disease. In fact, the only two studies of "survival in people without established cardiovascular disease" both reported no statistically significant relationships between dog owners and non dog owners. I wouldn't trust any of these studies unless you pull out the study and carefully examine who the subjects were; almost all of these are observational studies with no control groups (non-pet-owners is not a control group).

***

What's wrong with "story time"?

Go to this blog where they have a laundry list of statistics about pet owners. I can make up a story or two based on any of the following data:

• Families with children are much more likely to have pets than families without children
• The age group most likely to own a dog is 18-24 year olds at 58%.
• 78% of dog owners own their own home. 18% rent.
• Blacks seem to be less likely to own pets because they are more likely to live in densely populated urban areas with less room for pets.

Challenges with sports analytics

On the Junk Charts blog (link), I discussed some charts from the NYT graphics team (@nytgraphics) for a feature on the NFL draft.

In the second part of the behind-the-scenes blog post, they discussed how they visualized work by some economists. This is how the research was summarized: "across all players and positions, teams only picked a player better than the person who went next at that position 52 percent of the time."

That's an eye-popping statement, particularly when the use of 52 percent evokes the proverbial fair coin. It leads readers to think that draft picking must be almost as random as throwing a fair coin.

My reaction is this statement is disbelief. On the first order, as discussed in my other post (link), there is a clear and strong correlation between the round of the draft pick and average value. See the chart on the right which shows that the subsequent value of the players decline drastically as the order of the pick increases. When the first-order relationship is this strong, any result that contradicts it must be scrutinized.

***

The biggest issue in this analysis is the concept of "value". I'm speaking of value as defined in the ChartsNThings blog post: it's a number from a site called pro-football-reference, and a combination of games started and pro bowls, plus some adjustment for position played.

It's not hard to see that this metric is more discriminating for good players and we should expect the worse players to score essentially nothing. The graphics designers noted later in the post that it is not meaningful to talk about one benchwarmer being marginally better than another benchwarmer. (At this point, they stopped talking about the academic paper.)

In other words, the value gaps between the good players are large but the value gaps between middling and bad players will be small, or even nonexistent. The variance of value is small in the middle to lower rounds. This is a problem when you're comparing the value of consecutive picks in the draft: for the most part, you're just picking up noise.

***

Another potential hiccup are the outliers like Tom Brady who got picked in a later round. The criterion shown above would highlight one particular comparison: if you chose Spergon Wynn (Pick 183 in the 2000 draft) over Tom Brady (Pick 199), your guy is not better than the next guy in the draft. But there are five other QBs picked before Wynn in that draft; none of those teams picked Brady and yet this criterion did not penalize those teams.

***

Yet another concern is the cohort effect. The more seasons someone has played, the higher the total number of games started and pro bowls. The pro-football-reference site is very unclear as to what they do about this. It seems like they have a value for each season for each player but the NY Times team seemed to have picked out either one season or averaged all seasons or did something to end up with only one value per player. It's very easy to get cohort effect wrong.

Even if the adjustment for the player's tenure is made, we still cannot escape the high variance associated with younger players with short histories.

***

One last point: any proper analysis has to consider other factors that affect whether the draft pick could perform up to his potential. A quarterback playing behind a poor offensive line isn't going to do well. Injuries reduce playing time. Playing second-string behind an oft-injured player increases playing time, etc. While I haven't read the original paper, it doesn't sound like they accounted for these other factors.

 

Screening screening

Mammograms continue to be an emotional and controversial topic. I blogged about it some time ago. (link) Felix Salmon, whose blog should be daily reading, praises an article by Peggy Orenstein called "Our Feel Good War on Breast Cancer", NYT Magazine (link). Salmon's blog provides a quick summary; Orenstein's article is very long.

Orenstein's point of view has particular weight because she was diagnosed at an early age, and was an early advocate of breast cancer screening and now has a change of heart. She's worried about false positives, overdiagnosis, and over treatment.

You may recall that two or three years ago, there was a furor about new recommendations by experts that women reduce the frequency of screening. I used this example in a talk to illustrate the failure of these experts to communicate their statistical results, leading to a prompt, forced retreat. Immediately after the talk, several women raised objections--they believed this is such an emotional issue that numbers mean nothing. It was illuminating since I was at an analytics conference, and the women work with numbers on a daily basis. In the meantime, I was on shaky ground because of my gender. I'm glad that someone of Orenstein's stature has come out to try to explain the other side of this story. (See also my post on the Paradox of Screening here.)

***

Instead of rehashing the arguments around screening tests, today I just want to highlight some subtle aspects of measuring health outcomes.

One of these is described in Orenstein's article. One of the metrics used to measure health outcomes is the five-year survival rate... what proportion of patients are still alive five years after they are diagnosed with breast cancer? Said differently, we are interested in the length of time between the time of diagnosis and the time of death. There are two ways to improve this metric, by delaying the time of death, or by moving earlier the time of diagnosis. The former is difficult to achieve, and requires genuine progress in medication. The latter is very easy! If we screen younger people, we will move the time of diagnosis earlier. Without any medical progress, we will make the five-year survival rate fall. This phenomenon is known as lead-time bias.

Expanding screening to younger and less vulnerable populations has another effect, which I discussed in Chapter 4 of Numbers Rule Your World. The marginal person who gets diagnosed will be less ill than the average person - Orenstein tells us that the form of disease most commonly diagnosed in younger women (DCIS) ought not even to be called "cancer". Add less ill people to the population of ill people will automatically improve the survival rate, again without any medical progress. This phenomenon is known as length bias.