Big Data, Plainly Spoken (aka Numbers Rule Your World)

Gelman explains why massive sample sizes to chase after tiny effects is silly

What a lucky day I found time to catch up on some Gelman. He posted about the Facebook research ethics controversy, and I'm glad to see that he and I have pretty much the same attitude (my earlier post is here.). It's a storm in a teacup.

Gelman makes two other points about the Facebook study--unrelated to the ethics--which are very important.

First, he said:

if we happen to see an effect of +0.02 in one particular place at one particular time, it could well be -0.02 somewhere else. Don’t get me wrong—I’m not saying that this finding is empty, just that we have to be careful about out-of-sample generalization.

This statement is a reaction to learning that the measured response in the Facebook study, that is, the change in "emotions" of users due to the manipulation of positive/negative newsfeed items is tiny, on the order of hundredth of a standard deviation. Put differently, if the sample size of the study (~700K) were smaller, the effect would have been indistinguishable from background noise.

Sadly, this type of thing happens in A/B testing a lot. On a website, it seems as if there is an inexhaustible supply of experimental units. If the test has not "reached" significance, most analysts just keep it running. This is silly in many ways but the key issue is that if you need that many samples to reach significance, it is guaranteed that the measured effect size is tiny, which also means that the business impact is tiny.

For websites where the typical user does not sign on, tests are run based on using cookies to track unique users. It is often ignored how poorly cookies map to unique users (this ought to be a separate post). Let's say each user will show up at different times as different cookies. Then the iid assumption is violated; the correlation between units causes effect sizes to be over-estimated. The longer one runs a test, the more likely the same user shows up as multiple cookies.

***

Gelman's other observation is that studies with insignificant effects boostered by massive samples can get the p<0.05 and find their way to top journals. Add that to the pile of reasons why being published in a top journal is not necessarily a reason to trust a study.

***

On a separate note, I want to respond to a reader who asked me a question a while ago that I haven't answered. In one of my talks about the Netflix Prize, I remarked that the 10% targeted improvement was roughly equivalent to improving the accuracy of predicting the average rating by 1/10 of a star in the 5-star scale. Then, I forgot how I derived that number. Turns out it was in an earlier version of the talk. The accuracy metric used was RMSE which is on the same scale as the ratings data, i.e. the 5-star scale. The 10% improvement was roughly 0.1 on the RMSE scale, which is 1/10th of a star on the 5-star scale.

Pertinent to the tiny-effect issue discussed above, note that in the final phase of the Netflix Prize, several teams were just under the cusp of the 10-percent threshold. The eventual winners boosted their RMSE by 0.005 to get over the threshold. That is one half of 1/100th of a star on the 5-star scale. And it took 10 months.

Financial and statistical incentives to over-diagnose and over-treat

Nice article in the New York Times about the "overdiagnosis" problem in cancer screening. The particular case is thyroid cancer in South Korea.

There are a number of things about any form of screening tests that one should always bear in mind:

• Death rate is measured as the number of deaths divided by the number of people with the disease. The latter number increases with better diagnosis techniques.
• Better diagnosis techniques for cancers inevitably identify tiny tumors, almost all of which will never develop into cancer during anyone's lifetime. Peggy Orenstein had a fabulous article a year ago about the same issue in breast cancer. In that case, those tiny tumors weren't even called cancer until the diagnosis movement sprang alive.
• Once a tumor is labeled cancerous, patients will opt to fix it. Because these tumors would never have killed the patients, they inflate the number of diagnosed cases without increasing the number of deaths. So just by virtue of increased diagnoses, the death rate is brought down.
• The immediate outcome of a nationwide screening program is to dramatically increase cases. The article is a little unclear about whether it is the number or the rate of death that did not fall. It doesn't really matter; either case leads us to conclude that the screening has failed to improve health.
• One must not forget that screening tests, subsequent confirmatory tests, treatments, etc. all cost money, so there is a financial incentive to over-diagnose and over-treat.
• In addition to the financial incentive, there is the issue that I raised in Chapter 4 of Numbers Rule Your World (link). A false negative is a very public error on the part of the medical establishment while a false positive (followed by say removal of the thyroid) is an unobservable error. So there is a statistical incentive to over-diagnose and over-treat.

 

I can't believe I'm citing David Brooks on data

This is a first. I'm agreeing with David Brooks. Sort of.

In his new NYT column titled "Death by Data" (link), Brooks disparaged the recently celebrated practice of using machine learning in electoral politics, such as trying to win elections "Obama-style" by targeting investments on the people most likely to listen to his message, and trying to craft electoral messages by testing and measuring how people react to certain words and phrases.

Electoral politics is another success story often cited by Big Data people.

Brooks said a few things that pinpoint one consequence of how Big Data is being used today. Here are some nice quotes:

"As politics has gotten more scientific, the campaigns have gotten worse, especially for the candidates who overrely on these techniques."

"Data-driven politics is built on a philosophy you might call Impersonalism. This is the belief that what matters in politics is the reaction of populations and not the idiosyncratic judgment, moral character or creativity of individuals."

"Data-driven politics assumes that demography is destiny, that the electorate is ... a collection of demographic slices."

"... it is more important to target your likely supporters than to try to reframe debates or persuade the whole country."

"It puts the spotlight on messaging and takes the spotlight off product: actual policies."

The question in my mind is whether these issues are caused by the data-driven philosophy of the analysts, as Brooks asserts, or by the win-at-all-cost philosophy of the politicians.

By changing the context, these statements also apply to business use of data. A lot of the machine learning models improve the numbers but not necessarily the user experience or customer satisfaction. The prevalence of tricks used to promote unintended clicks of display ads is a powerful reminder of Brooks's "Impersonalism" idea.

I'm preparing my talk next week at the Business Intelligence Innovation Summit in Chicago, which is titled "The Accountability Paradox in Big Data Marketing". More data has not made us more accountable, so far.

***

I'm also unhappy about cookie-cutter campaign speeches that are but a string of buzzwords proven to appeal to the electorate by A/B testing results. But this is made worse by the politicians who are willing to utter these words brainlessly, and by the politicians who are willing to discard their own beliefs in order to win elections, and, also by the electorate who take these politicians at their words.

As Brooks correctly diagnosed, by using OCCAM data (link), particularly observational data without controls, the analysts surface correlations and have nothing to say about causation. This leads to a situation where the models provide little if any information to the politicians about the desires or wishes or expectations of the electorate. All they know is that if they include the word "family" and exclude the word "fear", they may get a higher rating, and if they get higher ratings among persuadable segments of the likely voters, they may win an election.

***

The second half of Brooks' article veers off course, displaying the arbitrary nature of the non-data-driven argument. He parades a list of names of past candidates, and claims that their failures are due to overly relying on data. Obama won the election but lacked any coherent agenda, for example. But there is no evidence to make the connection that subsequent failure is caused by bad use of data.

I last wrote about Brooks here.