Big Data, Plainly Spoken (aka Numbers Rule Your World)

More confusing statements of probability due to no controls

A recent article in USA Today is titled “Many with sudden cardiac arrest had early signs” (link). The signs include shortness of breath, faintness, chest pain, etc. Hold on to the headline because it’s the only thing believable in the entire article.

The words “early signs” imply to readers that were the men to heed these warnings, they could have prevented the cardiac arrests.

Think about the following two statements:

A1) Many with sudden cardiac arrest previously had symptoms.

B1) Many with symptoms subsequently had sudden cardiac arrest.

These two statements are far from equivalent, even though they describe the same sequence of events.

It’s easier to see the difference if we specify the symptoms:

A2) Many with sudden cardiac arrest had shortness of breath weeks before.

B2) Many with shortness of breath had sudden cardiac arrest weeks later.

It’s even easier to see if we include a number:

A3) 53% of those with sudden cardiac arrest had chest pain, shortness of breath, etc. (a direct quote from the article)

B3) 53% of those with chest pain, shortness of breath, etc. subsequently had sudden cardiac arrest.

B3) is clearly false. The universe of men who suffer from chest pain, shortness of breath, etc. is much larger than the population who have sudden cardiac arrest in any given week. B3) vastly exaggerates the number of sudden cardiac arrests.

***

How did the researchers come to make this type of claims? They were looking at a data set that had no control group.

“Clugh and colleagues studied medical records of 567 men from Portland, Ore., ages 35 to 65, who had out-of-hospital cardiac arrests between 2002 and 2012… 13% had [prior] shortness of breath… ” We have no way of interpreting whether 13% is a big or small number unless we know what proportion of middle-aged men with the same characteristics as those in the study but who did not have cardiac arrest suffered from shortness of breath.

One of the greatest challenges of the Big Data era is the absence of control groups. Without them, we don’t have a yardstick to judge.

For a change, the FDA earned my trust

I'm very excited to learn that the FDA has ordered 23andme to stop selling "personal genetic testing" to unsuspecting consumers. This is the company founded by the ex of Google's Sergei Brin (recent news here). So for a change, the FDA is doing something right for consumers, without waiting 10 or 15 years.

Here is a report on this action in San Jose Mercury (link). These tests have never been tested clinically and the company never applied for approval from the FDA. Given that it is not that hard to conduct proper testing of this product, especially if it is looking for diseases that have clear genetic markers, one suspects that the makers know that the tests would not pass rigorous testing.

There may be a day in which such tests are possible but the day hasn't arrived yet. I also think the chance that most diseases can be turned into equations that say if you have gene X, then you will develop disease Y is overly reductive.

***

One of the lesser studied effects of screening tests is the psychological impact of false results. If you tested positive for say breast cancer, and even if the biopsy showed the result was a false positive, there is always lagging doubt - every next time you feel something, part of your brain will make you worry. Then, there are those who take a better-safe-than-sorry attitude and elect to do procedures regardless - and some of these procedures have harmful side effects, which are frequently not properly explained.

 

 

PS. [12/3/2013] Andrew Gelman and readers join the discussion (link)

TSA gets into the randomization game

Surprise surprise. On my way to and from Raleigh last week, I became part of a randomized experiment! At the entrance to the security check, there was a tablet (iPad?) perched on a stand that randomized passengers into the left line or the right line. I encountered the same setup on the return leg.

On the outbound leg, I was assigned to the control group while on the trip back, I ended up in the test group, so I learned what the experiment was about.

In the test group, passengers followed the pre 9/11 rules. They called this "expedited": you don't have to take off your coats, not your shoes, not your scarves; and you don't need to take laptops out of your bags.

As usual, running real-life experiments is fraught with little annoyances. In the test group ("old style"), the agent has to explain to passengers that they don't need to take off shoes, etc. Some passengers are slightly confused and have started unloading coats, etc. Some probably ignored the agent's explanation, thinking that it is better not to risk being called back. If, say the throughput is one of the metrics being used to compare test to control, then this learning curve factor is surely going to disadvantage the "old style" protocol.

There was also an inconsistency in execution. On the away leg, the agent asked me to click on the screen while on the other leg, the agent clicked on my behalf and told me which line to enter.

I'n happy to see that TSA is getting serious about evaluating security policies using randomized tests. I wonder what metrics are being tracked. Any guesses?

Several things about education

First, I saw Andrew Gelman's rant about "big bad education" (link) which leads me to Mark Palko's rant about teaching "the Law of Large Numbers" in the new "Common Core" curriculum for New York schools. Mark's conclusion being:

If we start talking about setting aside significant time to cover probability and statistics accurately and in reasonable depth and put the ideas in proper context, you have my enthusiastic support, but until then maybe we should focus on the understanding, mastery, retention of the stuff that's already in the curriculum.

The Law of Large Numbers isn't even mastered by adults and college graduates and even PhDs. It's hard to imagine how we could do it justice in high schools. For example, there is mass confusion about what it is, as I have written about here.

***

On the sister blog Junk Charts, I mentioned the article I wrote for Imagine magazine, which is targeted at high school students. Hopefully, a few kids decide to take up statistics after reading my little contribution (link to PDF).

***

There are lots of bad things happening in education. To start with, the pay-for-performance concept being imported from the business world is singularly inappropriate when no one has been able to properly measure "performance". Even in the corporate world, I don't think I have come across a study that shows that CEOs, corporate board members or senior executives receive pay commensurate with their level of performance.

 

Hello North Carolina

This Wednesday, I'm giving the Big Data Seminar at NC State. Here is the announcement.

***

In his new book Numbersense: How to Use Big Data to Your Advantage, Kaiser Fung (NYU & Vimeo statistician) calls attention to one aspect of the Big Data phenomenon that has not received media attention: the consumers of Big Data analyses, i.e. everyone, will face more confusion and less clarity as the volume of data expands. As a result, consumers must develop “numbersense” to judge who or what to believe. 

In this talk, Fung will discuss the statistical challenges arising from Big Data studies, and explain what numbersense is. Some of the key conclusions are contrary to popular belief:

1. Big Data is subjective. 
2. Big Data does not end arguments.
3. Big Data makes mistakes.
4. “Moneyball” is not the rejection of intuition.

Numbersense is difficult to pick up inside a classroom. In Numbersense, Fung illustrates its applications through a series of recent stories on school rankings, the obesity epidemic, digital marketing, economic indicators, and fantasy football. 

A good paper on measuring digital marketing

In response to my post about the challenges of measuring digital marketing, Dean Eckles (Facebook) sent me a paper by him and his colleagues. The paper is titled "Social Influence in Social Advertising: Evidence from Field Experiments" by Bakshy, Eckles, et. al. (ACM 2012) and it's an impressive piece of work. In this post, I summarize the research for those who don't want to read an academic paper; and then present some of the feedback I provided to Dean, and Dean's response.

***

The fact that Bakshy, et. al. ran carefully designed randomized experiments is noteworthy in a field where statistical testing is much chattered but poorly practiced. The paper covered two sets of experiments. Facebook runs "sponsored stories" in the Newsfeed: these are advertising units in which a user is enticed to "like" a brand by being told the names of friends who have already liked the item.

Experiment 1 varies the number of friends shown in the ad, and concluded that listing more friends may increase the probability of the user clicking on the ad.

Experiment 2 compares the use of friends ("John Smith likes this") and the use of aggregate statistics ("14,892 people like this") in the ad copy, concluding that the inclusion of friends yields superior performance.

Experiment 3 (really just an interpretation of Experiment 2) considers the effect of stronger ties versus weaker ties on the chance of clicking. A stronger tie is someone with whom you exchange a relatively high proportion of communications. Not surprisingly, stronger ties are better influencers.

***

I sent Dean some detailed comments, from which I selected a few to discuss here.

While these experiments are useful, they also are limited in their interpretation. By design, each of these experiments measures whether one can improve response rates by varying the copy on the ad unit. It does not measure how much true benefit these sponsored stories are creating above and beyond other marketing activities. 

There are two obstacles to using these experiments to conclude that "sponsered stories produce incremental value for advertisers" (Admittedly, they don't make this general claim.) The first is that neither experiment has a "holdout" cell that did not receive any sponsored story ads. The "holdout" is a great way of measuring the baseline response, or the counterfactual (what if the advertiser did not buy these ads). 

The second obstacle is the choice of response. The metric used is clicks on the Facebook like button. This metric is so specific to Facebook that the only way to affect that metric is by marketing on Facebook. It would be great to see a more inclusive metric, like visits to the advertiser's home page or purchases, both of which can be affected through multiple marketing vehicles.

Dean generally agrees with these points. He said:

Yes, working with other outcomes makes a lot of sense. This paper predates some of the developments mentioned here and elsewhere re: Datalogix and increased use of "conversion tracking" for online purchases. Obviously, this doesn't remove the need to consider what the relevant counterfactuals are.

 

Many downstream outcomes are also substantially noisier. While I don't subscribe to all of their conclusions, you might find Lewis & Rao's "On the Near Impossibility of Measuring the Returns to Advertising" interesting, since it helps in considering under what circumstances precise estimation of ROI is possible.

 

I am a fan of the Lewis/Rao paper as well. It's one of the citations in my Significance piece on integrating Big Data and statistics.

***

Despite the experimental setting, interpreting results is still tricky. The simple conclusion that can be drawn from Experiment 2, for example, is that social cues are more powerful than statistics at influencing "like" behavior. We don't know for example that "liking" something is correlated with purchasing something. 

Another possiblity is "homophily" which the authors explained in the paper. Given that a social network is not a random group of people, one would expect that "birds of a feather flock together". So whether or not ad units are shown, it's quite possible that people within the same social network would like similar things. The fact that one member has indicated "like" on an item is predictive of another member of the same network also "liking" the item. And this latter action may not have been caused by advertising.

***

The paper is distinguished by the care taken to describe the many details of the experiment. As I pointed out in my Significance article, real-world online experiments are very complex and difficult to execute well. Let me describe several complications that the Facebook team has to navigate through.

Something like Experiment 1 can only be implemented on the subset of Facebook users who have one or more friends who have liked that brand. If someone has two friends who have previously liked the brand, and the experiment calls for showing one friend on the ad unit, then there is a question of which of the two is selected. As Experiment 3 shows, friends are not created equal; tie strength varies. Thus, the choice of who appears on the ad matters.

Facebook also runs "social marketing," which includes targeting ads based on characteristics of the users. This is a separate activity from the design of the ad copy of sponsored stories. A researcher would need to make sure that any such optimization does not introduce bias to the experiment.

Before ending, I'll address the trickiest but also most fundamental design issue... the unit of sampling. An uninformed view would think this is simple, just pick random users. In practice, this is not feasible. There are at any one time a variety of advertisers who are buying ad space on Facebook so you can't keep the ad units constant. In fact, many advertising campaigns are short-lived so you won't be able to collect enough samples if you stick to one or a few campaigns. So, the research is based on the (user, ad unit) pair as a sampling unit. 

This immediately presents problems of interpretation. For any given user, the ad units are not randomized -- in fact, the social marketing algorithm I mentioned above is likely to bias which ad units get served. Besides, the same user can appear with multiple ad units. This means that one must be careful trying to interpret the study at the user level.

What I like most about the paper is that the researchers have thought through these issues. Whether or not you agree with their design decisions, they deliberated on many of these complications. I'm looking forward to reading other studies by this research group.

Two videos of my recent talks

I'm hoping to speak at a location near you in the near future but in the meantime, here are two prior occasions you may have missed.

The first is the LISA conference. LISA is Leaders in Software and Art. You'd get a flavor of this fascinating group by viewing this 7-minute video (link). I gave a 5-minute lightning talk, which starts at 4:18 on the video. All the lightning talks have been condensed into a 7-minute clip, and a very impressive job indeed.

 

 

 

 

***

The second is a webinar on predictive modeling using social data. The most important sentence I said in this webcast is "We are spending too much time moving data around, and not enough time looking at the data."

This answer was in response to a question about how to deal with huge amounts of data -- a question that has become obligatory at any gathering related to "Big Data".

Be sure to click on the audio.

WSJ tells stories about working women

Over twitter, Jeff Harrison @mrjeffharrison shouted "Story time!" at this Wall Street Journal report on a Citibank/LinkedIn survey of working men and women.

"Story time" is the trick of reporting some statistics, then spinning a story that has little or nothing to do with the data just presented. This tactic is effective as some readers erroneously assume that the story is supported by the data.

A good illustration is this quote: "A Citi/LinkedIn survey earlier this year showed just one in four professional women has asked for a raise in the past year, a sign that women must build their negotiation skills, Descano said."

The reporter's conclusion is encapsulated by the headline of the entire article: "Job-Hoppers, Better Polish Those Negotiation Skills". However, the data concerned the proportion of women who asked for a raise. The rest is story time. Is it poor negotiation skills or poor self-confidence that causes women not to ask for a raise? Can't imagine it is the former.

Clicking through to the survey results leads to an infographic, confirming that the analysis is lacking. How is it decided that 25% asking for a raise is a bad number? What is the corresponding proportion for male workers? We aren't going to find out on this chart.

 ***

There is also confusion as to whether the survey involved only women. This is what the footnote to the infographics say but the Wall Street Journal reporter told us both men and women were included in the sample. Only involving women is a very bad research design, as there is no control group--this is one of the characteristic of so-called "Big Data" studies that often lead us down blind alleys.