Big Data, Plainly Spoken (aka Numbers Rule Your World)

Cycling in place

The Tour de France has been cleaned up in the past few years, right? Well, you decide.

Here comes news that this year's winner Alberto Contador tested positive during the race.

Here comes the claim that he ingested the stuff accidentally.

Here comes the spokesman who points to the "number of tests passed by Contador during the Tour" as evidence of no doping.

Go read Chapter 4 before making up your mind. 

Census 101 will help you avoid statistical fallacies

As an engineering undergrad, we had to take prerequisites in mathematics, physics, and chemistry. We are expected to possess a base level of scientific knowledge. It's like studying the classics for English majors. I'm surprised that it is not routine for social sciences majors to take a Census 101 course (tell me I'm wrong about this). Everyone should have a base level understanding of U.S. demography. Stuff like this:

These are population pyramids that show the distribution of population by age group and gender simultaneously. In each pyramid, the red and blue bars together sum up to 100%. The small-multiples format (see Junk Charts posts) brings out the comparison between racial groups: the top left chart shows whites, which essentially is the aggregate U.S. pyramid because the minority racial groups are still not large enough to matter (whites are about 70%, blacks and Hispanics each 12%, Asians 4%).

The key learning is that the minority racial groups all tend to skew young. In particular, focusing on the top row (whites v. blacks), we learn that about 34% of blacks are under 18 compared to 24% of whites while about 4% more whites than blacks belong to the 55-64 and 65+ age groups. (Note that annoyingly the scale chosen for the four pyramids are not the same. Despite its appearance, the Asian pyramid is quite similar to the one for whites.)

***

  What incites this post is the press release by Nielsen titled "African-Americans, Women, and Southerners Talk and Text the Most in the U.S.".  It has been all over the blogosphere (thanks to my friend Augustine F. for alerting me to it). For example, Gizmodo republished this as "Black people call and text way more than everyone else", with this chart shown on right. Curiously, the Nielsen article contains no graphics on either race or gender despite the prominence of these two factors in the headline.

The implication of such an analysis is that race is the main factor causing people to call and text more. This sort of thing I have called "true lies". The statistic is true but it creates a false impression.

Usefully, the Nielsen article contains a chart on differential usage by age group, which I reproduce here. Notice that the variance across age groups is much more pronounced than the variance across racial groups.

Teenagers text more than twice as many as 18-24-year-olds, who text more than twice as many as 25-34-year-olds, who text more than twice as many as 45-54-year-olds, who text three times as many as 56-65-year-olds, etc. In terms of voice minutes, 18-44-year-olds call 900 minutes or more compared to under 600 minutes for 55 and over.

If the analyst has taken Census 101, he or she might put two and two together and realize that minority racial groups skew young. So the question is whether blacks call and text more because they are black or because they are younger.

At this point, the analyst should refer back to the raw data, looking at the numbers at the race-age level. Not having the data, we make do with an estimate. By inspecting the pyramids, we compute that blacks have more 18-year-olds than whites by 10%, while whites have about 2% more 45-54-year-olds, 4% more 55-64 and 4% more 65+. So based on this age skew alone, we expect blacks to have an average of 270 more text messages! (Just do a weighted average of the excess proportions and the average text messages in each age group). The actual difference according to Gizmodo between blacks and whites was only 114 messages. So their headline of blacks texting more is misleading... the story is that blacks are disporportionately younger and youngsters text a lot more.

***

The story of voice minutes is different from that of text minutes--the opposite of what Gizmodo claimed. For voice minutes, the average for blacks (of all age groups) is almost double that of the white average. Referring back to the age groups for which the proportions of blacks are different from whites, the blue bars in the Nielsen chart tell us that the average minutes in those bars are not that different (631 for under 18, 587 for 55-64, etc.), and thus age-group variation cannot explain the large gap in voice minutes.

It is still premature to claim blacks call a lot more than whites because of their race. One should check other factors like income groups, education levels, etc. to make sure that something like the text message story is not repeating itself.

***

If you have the raw data, what you'd do is to look at black-white differences at each age group separately; if blacks do text a lot more than whites at most/all age groups, then that's evidence for the Gizmodo hypothesis. Of course, given the quick estimate above, we already know that is not the case with texting. For those who read Numbers Rule Your World, this is the key statistical concept of Chapter 3.

For comments on the graphic used in the press release, see the related entry on Junk Charts.

The population pyramids came from the report called "Youth Demographics" by Mark Hugo Lopez and Karlo Barrios Marcelo, published November 2006.

PS. For those paying attention, the reason why the quick estimate is not a perfect substitute for getting the raw data is that I have to assume there is no "interaction effect" between age and race, in other words, I assume that the average within an age group is the same for blacks as for whites but in reality, it is possible (though unlikely) that white teenagers are more like white adults than they are like black teenagers.

Bringing the bugs home

You may have heard of the bedbugs epidemic in New York. So the other day, a friend relayed the story at her workplace. Bedbugs were found. Dogs were sent in to sniff out the creepy insects. If the dogs detected something in an office, the room is closed off until cleaning is completed. The dogs did not find anything suspicious in her office.

We had a little dialogue. How safe should she feel about the situation? That's when a little statistical thinking could help. The dogs are essentially a detection system like a steroids test or a lie detector test (discussed in Chapter 4), and all such systems are subject to errors. There are two types of errors: false positives and false negatives; and any given system trades off the two errors. The designer of the system calibrates the sensitivity, which in effect is setting the trade-off level between the two errors.

To understand how such a system is calibrated, we need to know who makes the decision and what incentives do the decision maker has. My guess is that the dogs are sent in by the exterminators for a diagnosis of the problem. Exterminators make money if there is a problem. I wouldn't be surprised that these dogs generate a lot of false positives, which means not many false negatives.

In this case, if you are the employee, you would also like the test to generate false positives and not false negatives. The "cost" of the false positive to you is minimal but a false negative might mean you will bring some bugs home. The only people who might not like this calibration are the ones footing the bill for the cleanup, that is, the employer. It is apparently quite expensive. (But perhaps the employer too may favor false positives because there may be legal exposure if a worker thinking her room is clean brings some bedbugs home. If so, I'm not sure why the dogs are needed at all; if everyone agrees on the precautionary principle, then just send in the exterminators!)

On balance, I feel that those declared clean are pretty safe, and many if not most of those declared positive are probably not infected either. (Whether the bedbugs crawl from one room to another is another matter.)

PS. It is possible that for a detection system to generate a lot of errors of both types. Lie detectors is an example of such.

PS. This is a situation where the uncertainty / fear will drive a "market" to over-diagnose and over-spend on a problem. If you run an exterminator business, it is in your interest to drum up as much hysteria as possible over the problem, with the press feeding on sensationalist stories.

Pills: victims of sloppy statistics?

According to the Daily Mail, a Canadian study proves "the peril of the sleeping pill: users a third more likely to die early". How should we read such an article? What should we look for to assess its credibility?

***

Start with these sentences in the middle of the article:

Dr Belleville analysed 12 years of data on more than 12,000 Canadians.

When all other factors were equal, death rates were found to be significantly higher among sleeping pill users and those taking tablets to ease anxiety.

And later, this sentence provides a bit more information:

The data includes information on people aged 18 to 102, surveyed every two years between 1994 and 2007.

So first thing is note is that this is not a randomized controlled trial but an observational study; also, the data came from self-reporting in surveys, and not direct measurements. Both these details should cause us to widen the margins of error. (In an RCT, we would divide people at random into two groups, one group will be instructed to take sleeping pills and the other group would take a placebo.)

Now, note the use of "when all other factors were equal". If they conducted an RCT, then this statement would be perfectly reasonable because random selection makes it believable that the test group and the control group had the same mix of males/females, age group mix, education level, income distribution, etc. etc. When this is an observational study, making such a statement is an overstretch. They would need to prove why we believe that people who choose to take sleeping pills are comparable to those who choose not to take sleeping pills. It is hard to believe that "all else is equal" between these two groups.

Now, notice the little bombs that drop along the way:

Crucially, the study did not distinguish between those who were heavy users and those who only took them occasionally.

Nor it seems they distinguished between dosage or brand of pills:

Pills used ranged from over-the-counter antihistamines to powerful prescription-only preparations such as Valium.

This is the issue of Chapter 3 of Numbers Rule Your World. They are lumping together into one average a wide variety of people with very different characteristics, and this may hide differences between subgroups. For example, the observed increase in risk may be fully explained by those who take powerful sleeping pills at high frequency but when lumped together with the light users, we can't tell.

***

This particular sentence raises more questions than it answers:

Pill takers were more likely to succumb to every type of illness, from parasites to cancer.

Frankly, reading this, I'd hazard a guess that people who take sleeping pills are less healthy than people who don't take sleeping pills. They are a self-selected group of people, and this explains why they succumb to every type of illness -- these diseases do not have to have anything to do with taking the pills themselves.

Analogously, if we compare the average income of graduates between a top public college and an Ivy League school, we might conclude that the Ivy League school provides a better education; however, most likely, the Ivy League school accepts students that are better prepared to begin with, and so we can't tell if the higher income is due to a better education or better preparation prior to college.

In other words, there is a high likelihood sleeping pills are not the killers they are after.

***

By the end of the article, the journalist has declared story time. How can this finding be rationalized so that readers can believe it? Here are some of the stories they told us in the article (strangely repeated)

Sleeping pills and anti-anxiety drugs affect reaction time, co-ordination and grogginess, which raises the odds of falls, she said.

Tablets might also suppress the respiratory system, which could aggravate breathing problems during sleep, particularly for those with heart problems.

In addition, effects on the brain could affect judgment and moods, increasing the risk of suicide.

Note that I am not against speculation per se. It could lead to real progress. I am against making such assertions without any attempt to test them. I am especially against elaborate multi-step assertions. For every story one can come up with, someone can concoct a different story with the opposite ending.

Can a test prove two things are identical?

I have a (possibly irrational) fear of genetically modified/engineered foods. I thought I would disclose this before I plunge into this topic, agitated by this Washington Post article about the FDA stance that (1) does not require labeling of genetically modified foods; and (2) forbids labeling of non genetically modified foods as such.

***

How is this topic related to statistics? Consider how you might answer the question: is this new food going to harm us? Clearly, some form of testing is needed to compare the natural version and the genetically modified version. Are they the same or are they different?

According to the FDA, this is an open-and-shut case. Their scientists have determined that genetically modified salmon is not "materially" different from other salmon, by which they mean there are no "biologically relevant" variations between the two versions of salmon.

The level of confidence expressed in this conclusion is astounding and, I have to say, un-statistical. I seriously wonder if there are statisticians involved in this decision at all.

***

Notice that the scientists issued a qualified statement: no "biologically relevant" variations instead of no variations at all. What do they mean by "biologically relevant"? The article does not tell. 

In fact, "biologically relevant" is according to whatever metrics these scientists deem important to examine. Fat content, mercury content, etc. Presumably anything that they consider to have health implications. Therein lies the rub. This analysis presumes that we know exactly which parts of salmon are related to our health, and which parts aren't. In my opinion, there is no way to know because the genetically modified salmon is an innovation.

The general lesson is that tests won't tell us about things that they are not designed to measure. The design phase of testing is even more important than any subsequent analysis.

***

One wonders if a properly designed randomized controlled trial is being planned, and if not, why not. One branch would only eat natural salmon and the other branch would only eat modified salmon. Would we find differences in clinical outcomes?

The value of such trials is that if the clinical outcomes turn out to be different, we know something unexpected is happening, and we can go back and search for the biological cause of such a difference. Without such trials, our knowledge is limited by what dimensions the scientists attempted to test. But what we fear is not the usual suspects but the surprises.

***

Under the traditional statistical testing framework (discussed in Chapter 5 of my book), one would set up a hypothesis that the two versions are the same, and venture to seek significant evidence of difference to reject that so-called "null hypothesis". If insufficient evidence exists for rejection, the conclusion is that there is insufficient evidence to prove that the two versions are different. This statement is not the same as that the versions are identical.

There is a reason why we do not like to assert that two things are the same. The two things can be different along any number of dimensions -- a fish is a complex organism; one cannot possibly have exhausted all such comparisons.

One dimension was clearly not considered, namely, the genetic makeup of the salmons. Given that one version has been given a gene (which quickens its growth), it cannot possibly be the case that the two versions are identical. This difference is judged to be not biologically relevant by someone in a position of power.

***

Even if one trusts the FDA scientists, there is still absolutely no reason why marketers of natural salmon should be forbidden from advertising their fish as not having genetic modification. In the article, we were fed a bunch of legalistic, Orwellian mumbo jumbo about why this is so, none of which convincing. The last section of the article ("No 'Hormone free' either") is a must-read, and unintentionally hilarious.

If the scientific evidence is as clearcut as the FDA is saying, they should have no reason to fear a public rebuke unless they think the public is stupid. Withholding information and taking choice away from consumers are horrible ideas.

Happenings

Numbers Rule Your World was picked by Fareed Zakaria on CNN as Book of the Week on Sept 5. His verdict: "For those of you who hated statistics in college, this is actually an easy read with a big benefit." Thank you Fareed.

***

I gave a plenary talk on "What happens after the math is done?" at the JMP Discovery Summit in Cary, NC on Sep 15. (The abstract was posted here.) Dan Ariely (Predictably Irrational) also spoke at the conference about his new book.

Daniel at the Firefly Ecometrics blog summarized part of my talk and gave his reactions to it.

The transcript of a live blog is here.

One insight I gained from the conference is that even long-time Junk Charts readers are unaware of this sister blog! I think this is because the RSS feeds are separate. (The Twitter feed though is the same.) Will try to rectify this soon. As I mentioned in my talk, the two blogs are synergistic: one is about visual communications, the other written communications.

***

I noticed two new reviews of the book.

One is by Professor Gary Fetter at Western Carolina University's Business School, published by Asheville Citizen-Times (link here). He rated it 4/4, with the following summary comment: "In 'Numbers Rule Your World,' Fung makes statistics entertaining, accessible and relevant to our lives."

Another is on Martin's blog. He uses the occasion to point out the truism that "data is not the same as information", which is the first principle I teach to my students. The arrival of computer technology -- the plunging cost of storage and the fast growth in processing power in particular -- creates a huge amount of data, much of which were designed for engineering purposes. The existence of data does not per se make them useful. Martin gives the example of plotting the frequency of Facebook messages containing the word "break-up" and asks, essentially, "what happens after the math is done?"

I cannot agree more.

Innocence makes one more likely to make a false confession

That was one of the revelations while I was researching for Chapter 4 of the book, dealing with unavoidable errors in statistical decision making.

The New York Times took up this very important issue in a recent article. Typically, we think that if someone confesses to a crime, the confession is the strongest evidence possible, and guilt is  assured. However, criminologists who have studied this issue discover that innocent people do confess to crimes that they did not commit, and not just mentally challenged people. They believe that other evidence will surface to prove their innocence but juries tend to take confession evidence at face value.

The other shocking thing I learned was that it is legal for the police to tell you you failed a lie detector test even if you have passed it. The police also consider lie detector tests as a tactic to extract a confession (because in most courts, lie detector test results themselves are inadmissible.)

The trouble, as the article points out, is that very rarely are such "false positive" errors uncovered and only after extraordinary effort, and therefore the cost to the decision-maker (law enforcers) of this type of error is quite small. That's why the work of non-profit groups like the Innocence Project is absolutely invaluable.

(Thanks to reader Stephen N. for bringing this article to my attention.)

Bias, variance, random errors and honest mistakes

The website for financial skeptics, Zero Hedge, highlighted a chart showing the estimation errors committed by the statisticians at the Bureau of Labor Statistics relating to the weekly jobless claim number. A sobering sight:

Typically, the BLS publishes one number, and then subsequently, publishes revisions. The data being plotted are the differences between the revised estimates and the original estimates by week. They plotted this series all the way back to 2000 which allows us to observe the magnitude of these errors during both a boom period as well as a bust period.

What is striking is the clear positive bias, i.e. the revised estimates of the jobless claims were more likely to be higher than the original estimates. If the statisticians were making errors at random, we expect the area above the zero line to be about the same as that below. In an ideal model, the errors would be random but that is the ideal, not the reality.

What is even more disturbing is that the positive bias persists through the entire business cycle! I'd have guessed that a positive bias is likely during a bust period because the statistical procedure may "lag" (fail to catch up with) the actual growth in jobless claims but that theory went out the window when the same direction of bias remained the same even during the boom years.

***

"Tyler Durden" at Zero Hedge takes this as evidence of "propaganda":

The implication is that fraudulent (and we sure hope this is inadvertent, although a 90% error rate definitely would invite a criminal investigation into just who and how stands to benefit from such an manipulative upward bias) data reporting is responsible for a persistent upward bias in data, and that fundamentals have been disconnected from the "government's reality" for years.

He is right to draw our attention to these numbers, and we should take a few things away from this discussion:

• Statisticians pay a lot of attention to studying the so-called "error term" because they tell us a lot about the quality of our models.
• Notice the difference between "bias" and "variance". Bias is the red line in the graph that shows that on average, the revision is about 3,000 worse than the original estimate. Variance consists of the fluctuations around this red line.
• The performance of any estimation procedure will be more easily improved by reducing the large variance (the huge spikes) than by reducing the bias (average error). This is one way in which the main idea behind Chapter 1 enters.
• We should always pay attention to the incentives of the people working with numbers; this includes the statisticians compiling the data, the government officials reporting the data, and last but not least, the pundits commenting on the data (including Zero Hedge). This is one of the key messages in Numbers Rule Your World. This is also why the entire "official statistics" field, which was one of the earliest fields in statistics, has always emphasized the need to be politically independent, and an honest reporter of the numbers.

I do disagree with "Tyler"'s knee-jerk conclusion of malice. It is perfectly possible for statisticians to make "honest mistakes". Bias could result from incompetence rather than malice.

Ryanair CEO does not believe in investment science

The on-line business tabloid, Business Insider, just gave an airing to Michael O'Leary, the CEO of Ryanair, who lambasted the scientific community for saying there is man-made climate change. He made the following shockingly ignorant accusation:

It is absolutely bizarre that the people who can't tell us what the f***ing weather is next Tuesday can predict with absolute precision what the f***ing global temperatures will be in 100 years' time. It's horses***.

The original interview was carried on the Irish Independent newspaper.

According to Wikipedia, Mr. O'Leary has a business degree from Trinity College, Dublin. OMG, what do you teach business students in Dublin? I imagine something like:

Don't believe those people who tell you to buy stocks for the long run. If they can't tell you what the f***ing stock price is next Tuesday, it is absolutely bizarre that they claim they can predict with absolute precision what the f***ing return on equity will be in 100 years' time. It's horse***.

So, Mr. O'Leary just called out the research of Jeremy Siegel, Burton Malkiel, and mainstream economists and business school professors as well as the investment management community for peddling non-sense.

What happens after the math is done?

That is the title of my talk to be delivered next week at the JMP Discovery Summit (Sep 15 at 10:15 in Cary, NC). The other talks look great too, including Dan Ariely and Dick De Veaux.

Here is the abstract I submitted:

The phenomenal recent success of the Freakonomics and Gladwell franchises, and their offshoots, has sparked unprecedented curiosity in the analytics profession. Not long ago, business executives tended to see us as technocrats speaking an abstruse lingo engaged in niche activities with uncertain effects: smart, intimidating, hard to grasp, lost in details.

Times are changing. The famous authors have demonstrated how to use low-density, soft-touch narratives to convey analytical results to broad audiences. They imagine the analytics professional as a kind of intellectual hero, wielding data as weapons against conventional wisdom. The common structure of these stories follows the well-known analytical process, beginning with defining the research question, and ending with discovering the correct answer.

All this is masterfully told, but represents only half the story. In my experience, the analytics job is less a solitary struggle for the scientific truth than a managerial challenge in balancing objectives, facilitating collaboration, and creating consensus. In this talk, I will tell the second half of the story, using materials from my recently published book,  Numbers Rule Your World. The focus is on what happens after the math is done.