Big Data, Plainly Spoken (aka Numbers Rule Your World)

Lining up at Disney during the holidays

The New York Times has an article out talking about Disney lines, a topic I also covered in Chapter 1 of Numbers Rule Your World.

If you are visiting Disney this holiday season, it's nice to acknowledge the work of lots of engineers trying to make your waits as painless as possible. By contrast, in the recent debacles of snow-induced chaos at various airports, one gets the feeling that the airline industry does not place customer comfort as its #1 priority.

(Thanks to Augustine F. for pointing me to this Endgadget post. Augustine blogs about digital media here.)

Know your data 3: Are you a BT or BB?

The New York Post tells us how restauranteurs profile clients. Here's something interesting about OpenTable, the on-line reservation website:

A client’s OpenTable profile could include his birthday, favorite cocktail or latest Yelp review. It might also list details about his personality, professional life and even his kids’ education.

“You can leave documentary notes about a customer — ‘This is the customer’s anniversary,’ ‘This is the customer’s favorite drink,’ ‘This is the customer’s favorite table.’ Or, we’d have little initials — B.T., B.B. [Ed: Bad Tipper. Big B*tch.]

Of course, we like restaurant owners to know us as individuals and treat us well. One of the coffee shop owners I frequent noticed that I like to put aside the tea bag before drinking tea, and soon enough, he started bringing me a small glass to hold the tea bag. In the past, owners observe customers; now, they can build detailed profiles using our data.

Better service or big brother?

The media stick to old labels in spite of 2010 Census

The 2011 Census result is out, and I've been listening to the news: the focus seems to be that the south and west grew faster than the northeast and midwest, and since Democratic-leaning states like New York, Massachusetts and Ohio will lose seats while states like Florida, Georgia and Arizona will gain seats, the Republican Party will be extremely pleased while the Democratic Party will be the big loser.

Be very careful before you buy this overly simplistic logic.

They use an undisclosed assumption, which is that the new Floridians are just like the old Floridians, the new Arizonans are just like the old Arizonans, etc., which in turn implies that the migrating New Yorkers are not like the average New Yorker, the migrating Illinoisan is not like the average Illinoisan, etc.

Put differently, they claim that the demographic shift does not affect preexisting "labels". Let's imagine a family living in Massachusetts packed their bags and moved to Arizona. The media seems to think that people vote based on where they live, maybe it's the water they are drinking, or the sun rays they are basking in, but the same family living in Mass. voted Democratic but having moved to Florida would vote Republican. The media tells us: the Party leaning of a state is an immutable truth unrelated to who lives in which state.

We don't know if this assumption is true or not. We do want to know the source of the change in demography. Is it migration from north to south as the media envisioned? Is it higher birth rates of current residents? Is it higher birth rates among certain demographic groups? Is it lower birth rates among certain subgroups?

***

I call this the "stranglehold of labels". States like Florida, Georgia and so on are given Republican labels; and Massachusetts and New York, Democratic labels. What's been forgotten is that voters determine these labels, and to understand the true impact of the 2010 Census, we have to trace the people.

This is the topic of Chapter 3 of Numbers Rule Your World. The media ignored an important source of variability; they have lumped together the type of people who chose to migrate with the type of people who chose not to migrate, but chances are these two subgroups have different voting behavior.

***

This type of thinking infects a lot of decision-making. For example, a business discovers that customers who pay by credit card are less likely to default than those who want an invoice billed later. The business knows if they have more customer paying by credit card (as opposed to invoice), it would be more profitable. So it decides to pay customers currently on invoice to switch their method of payment to credit card. Customers are very willing to take the payment to switch. The question is: if these customers would not pay their invoice in the past, would they be paying their credit card bills now?

What you can compute v. what you should compute

One of the themes that I cover in talks is the need for the analytics community to move beyond "insight" and shift attention to "impact". This requires a shift in mentality from "what I can compute" to "what I should compute".

I was looking at this analysis done by the Linkedin team:

They processed their user profiles and found the top 10 words or phrases people use to describe their work experience. Nice insights. Not something you and I could guess at with any accuracy without the data processing capability. However....

What should they be analyzing? Unfortunately, I don't care about "overuse"; in fact, this is an instance of "story time" as overuse is highest frequency with a negative value judgment, and I'm not sure from where the latter arises.

What should they be analyzing? What LinkedIn users care about, I suspect, is whether the use of particular words is correlated with a higher hit rate, more people looking up your profile. If they can show us this correlation, then they have successfully moved from insight to impact. And they are tantalizing close to this analysis, as they already have the word frequency data, and based on the statistics available on their website, they also track the frequency of profile views.

 

 

 

Know your data 2

The WSJ crew strikes again:

The iPhone version of the music app Pandora sent information to eight trackers. It sent location data to seven of these, a unique phone ID to three, and demographic data to two.

A few things to note when reading this piece: first, the suggestion that a "written privacy notice" is a solution is laughable; second, the notion of a company analyzing its own customers to provide better service versus a company selling our data to other companies for profit; third, "free apps" are not "free", and before you chastise app developers, think how advertising is the only source of revenues for them, and what that entails.

There is a constant refrain in the article, developers claiming that they "do not" tie the phone's unique, undeletable ID to a person, or they "are not" doing so. Nobody said they "cannot". That's because they can. I don't know about other platforms but on the iPhone, you must sign on in order to get any apps from the App Store, regardless of whether the app is free or not; but your App Store account is your credit card so your phone ID can definitely be tied to your credit card -- and sorry to say, the credit card leads very far indeed. Apple (nor Google) is not a neutral player in this game because Apple (and Google) runs large advertising businesses.

Thanks WSJ for running such a wonderful series. When are they giving out Pulitzers?

Efron on statisticians

A wonderful interview with Brad Efron (Stanford statistics professor) should be read in full at Significance magazine. The following quotes caught my attention.

First, he addresses the impact of "big data" on the scientific method:

scientists have misled themselves into thinking hat if you collect enormous amounts of data you are bound to get the right answer. You are not bound to get the right answer unless you are enormously smart.

they'll [cosmologists] will argue not about nature but about what they saw in the simulation.

Here, he explains what is a "second-level science":

the history of science is that we solve the easy problems first, the ones that were very hard-edged and that didn't need any statistics or probability, and one by one those fields were conquered and now they are leaning down on us [statisticians]. Very much more complicated things are being studied, including things that aren't in nature. So there is a science in nature and science beyond nature, and I think we are into the second.

And he thinks we're all cynics:

Statisicians are cynics, because you realise how much of the stuff that you are told is true in the world is actually just that month's accident that worked out, or that month's disaster that happened.

 

 

Know your data 1

I'm going to start posting links about how businesses use our personal data. This debate over privacy will become more intense over time. Whichever side you're on, you should know what is happening out there, especially at the big companies.

Today's link is about Facebook User IDs:

Armed with your e-mail address, data miners can hit Facebook and match it up with your user ID. That key unlocks a treasure trove of personal information.

Yet again, can't trust anyone

I keep saying we have to evaluate numerical claims, each and everyone of them. It's not just politicians who manipulate us; businesses are not beneath cheap tricks either.

Wired/Gizmodo helps us interpret TV specs here.

It's hard to judge who's right unless you're an expert but several general principles are well worth noting:

• Bigger numbers are not always better; the number may be meaningless. A great example of this is when the phone manufacturers pitch ever higher frequency bands (this is just like setting your radio dial in different positions; what's for sure is that higher frequency uses more power.)
• The limiting factor may not be the piece of equipment you're buying. Even if your TV can do something, you may be limited by the media you are watching, or by related equipment like your audio setup.
• The reason for a better experience may be something other than what the manufacturer claims, and much less convincing. For example, your higher-frequency-band cordless phone may indeed provide better reception, but that is likely because you are one of the first to shift to a new frequency band; wait until that band gets congested.
• The definition of a metric matters a lot. Having an industry standard helps.
• The specs are written to cater to every wish. Know what you care for, and ignore other factors. Others may want an ultra-thin TV but how much does thickness matter to you?

Aspirin: miracle drug?

Forgot where I first read about this but the Lancet reported that taking aspirin daily not only reduces heart disease risk but also reduces the risks of a plethora of cancers. The abstract can be read here.

What kinds of questions come to my mind when I read something like this?

First and foremost, I note that this is -- at this stage -- a purely statistical finding, not (yet) backed up by a biological mechanism that can explain why aspirin reduces cancer risk. One can be much more comfortable with the result if the biology were there to support the statistics. (This is a theme in Chapter 2 of Numbers Rule Your World.)

Secondly, one always should be wary when one drug is said to cure a dozen ailments. In such situations, one must ponder whether there is a hidden cause lurking behind all of this. For example, it could be that people who take aspirin daily are self-selected, and are more likely to take better care of their health.

The feel-good factor of this finding is that the words "randomized trials" in the title of the paper. It gives the impression that the aspirin test group and the placebo control group were randomly selected and thus self-selection is not an issue.

At this point, we need to delve into what kind of analysis was done. It is a "meta-analysis", meaning that they aggregated the results of "eight eligible trials", each of which was a randomized trial, and treated the pooled data as if it were a large randomized trial. Presumably each trial had different enrollment rules, and so it is not assured that the pooled groups are unbiased. Would like to see if they looked into this.

The key passage is this:

On analysis of individual patient data, which were available from seven trials (23 535 patients, 657 cancer deaths), benefit was apparent only after 5 years' follow-up (all cancers, hazard ratio [HR] 0·66, 0·50—0·87; gastrointestinal cancers, 0·46, 0·27—0·77; both p=0·003). The 20-year risk of cancer death (1634 deaths in 12 659 patients in three trials) remained lower in the aspirin groups than in the control groups (all solid cancers, HR 0·80, 0·72—0·88, p<0·0001; gastrointestinal cancers, 0·65, 0·54—0·78, p<0·0001), and benefit increased (interaction p=0·01) with scheduled duration of trial treatment (≥7·5 years: all solid cancers, 0·69, 0·54—0·88, p=0·003; gastrointestinal cancers, 0·41, 0·26—0·66, p=0·0001).

I'm surprised that they were able to keep the placebo/control groups in all of these 7 or 8 trials clean for 20 years. Is it really true that none of the patients in those groups took aspirin all those years? They would have a much more restricted result otherwise.

In addition, reading this passage (and other parts of the abstract), it is clear that the researchers conducted an exploratory data analysis. They told us it worked for this type of cancer but not another type, after this many years of delay but not fewer, etc. For this type of work, the conventional p< 0.05 criterion is inappropriate; the standard of significance at the level of individual tests must be much, much tighter. This is especially true of "meta-analysis": pooling together test populations will always reduce the margin of error but I'd like to see corrections for multiple comparisons (or, as Andrew Gelman likes to argue, a sophisticated modeling regime that shrinks the effects).  

If I were the authority, I would not recommend any treatment strategies based on this paper but I'd order up confirmatory tests to validate the hypothesis directly, and also order up research to uncover the biological mechanism that links aspirin to curing these cancers.

Neat reads 6

The illusion of control, Bad Science (link)

Must-read. This area of research sounds truly amusing. Points again to the importance of measuring the placebo effect.

Mystery: what happens when neuroscientists, psychiatrists, and business executives get together in a room? (link)

Someone at Wired is obsessed with why expensive medicines don't work. Last year, Steve Silberman proposed that placebo effect is getting stronger and in the way of meds. I made some dissenting comments here. Now, Jonah Lehrer tells us there is a New Yorker article claiming that "disturbing" false positive results threaten the "scientific process".  My frustration with this whole storm in a teacup is that the participants spend all their time on secondary issues (such as the placebo effect being "too strong", "our facts losing their truth", "claims enshrined in textbooks suddenly unprovable", too much natural variability) while ignoring the most obvious hypotheses (such as medicines not good enough, effect sizes too small to be measured properly, clinical trials standards generate too many false positives, clinical trials poorly designed, conflict of interest leading to insufficient attention to side effects, fast track and other politically-driven standards that usurp the scientific process, use of medicines beyond the originally targeted groups).

The quoted passage ends with this truism:

...experiments are essential because they "put nature to the question". But it appears that nature often gives us different answers.

This natural variability is posed as a challenge to the principle of "replication" in science. If this is the mainstream view, we are back in the Stone Age. It is precisely because of variability that replication is necessary. If every experiment would yield the same outcome, we only need one measurement. If pharmas spend billions of dollars to develop medicines that have effects so tiny they can't be measured without extraordinary efforts, then no amount of truth-finding can hide these bad investment decisions.

Statistics invade the humanities, according to the New York Times (link)

Can such analyses yield generalizable results?

getstats: a 10-year plan to improve statistical literacy (link)

I applaud.

 The Joy of Statistics, a BBC documentary (BBC link, Open University link)

Related to the getstats campaign. Hans Rosling, creator of Gapminder (absorbed by Google) is the star here. Looks like the documentary focuses on how to find insights from big databases of public data. Hopefully, it will cover how such insights can directly improve our lives. As I have been saying in my talks, it is not sufficient for statistics to be fun, it needs to be impactful.