Big Data, Plainly Spoken (aka Numbers Rule Your World)

Story time

A new study released by the American Public Transportation Association (APTA) shows that people who live in communities with extensive public transportation networks exercise more, live longer, and are generally healthier than people in automobile-dependent communities. (Via Inhabitat blog)

So far so good. They found that the presence of public transport is correlated with public health.

***

Then the teacher declares story time, and then up is down, and left is right.

The headline writer tells us: REPORT SAYS PUBLIC TRANSPORTATION MAKES YOU SKINNY. It said nothing of this sort.

APTA president makes up this chain of cause-effect, with no supporting evidence:

Use of public transit simply means that you walk more which increases fitness levels and leads to healthier citizens. More importantly, increasing use of public transit may be the most effective traffic safety counter measure a community can employ.

Here are some other stories for your reading pleasure:

1. Public transport typically are better developed in cities. Younger people happen to prefer living in cities. Younger people exercise more and are typically healthier.

2. Cities that have extensive public transport typically also have an abundance of restaurants and food stores. Having lots of food around tempts people, and they tend to eat more than they need. They get fat and unhealthy.

3. Places with great public transport are mostly cities. Cities are crowded, polluted and have a dearth of outdoor space. City dwellers do not exercise as much, and they get unhealthy.

4. Places with great public transport are mostly cities. Cities are crowded, polluted and have a dearth of outdoor space. Because of scarcity, city dwellers value green space more, and therefore they tend to spend more time there and get healthier.

5. Large cities have plenty of distractions from healthy activities. Night clubs, movie theaters, Broadway shows, karaoke, shopping malls, etc. all take up valuable time that should otherwise be used for exercising. Therefore, people are less well in cities.

6. City dwellers are vain people who must look good. Looking good means being skinny, according to the fashion mavens. Thus, people exercise more and eat healthy.

***

Ok, you get the picture. There are no end to telling stories. The one thread linking all these stories is that there is not a shred of evidence to support any of the logic. The data that is cited merely states the correlation between A and Z, and tells us nothing about A causing B causing C causing... Z.

Taleb calls this the "narrative fallacy". Our minds are very active and very successful in making up stories. Unfortunately, most of them are junk, and nothing more.

Good news, bad news from the FDA

Good News

FDA is proposing to withdraw the "unproven" drug ProAmatine used to treat low blood pressure because the company producing this drug (Shire in Ireland) got it approved "fast-track" with some promising "early" results but never then confirmed the finding with a more rigorous trial. (AP report via Yahoo!)

ProAmatine has been on the market for 14 years, and Shire has made over $200 million in revenues from it. (This is not a "blockbuster" drug.)

Bad News

I may be wrong about this but the number 14 rings a bell. Isn't it the term of a patent? So the drug developer gets to enjoy monopoly rent during this term, and thereafter, generic drugs are allowed to compete in the market, and prices collapse. Cynical I say Shire is laughing all the way to the bank.

GAO said the FDA has never before forced any drug company to withdraw a drug due to missing or unimpressive data.

Summary of recent talks

For those who couldn't attend, I summarize the discussions that took place during the JSM session, and the talk at EdLab at Columbia's Teachers' College.

At JSM, the panel, led by Professor Mammo Woldie, focused on what business analytics is all about. Dan Coates, who heads up Youth Pulse and previously spent years at SPSS, gave an overview of the types of problems that businesses face, and the companies that have made key investments in developing advanced analytics teams. Nathaniel Derby, an independent consultant and entrepreneur, has lots of advice for statisticians who are interested in developing private practice. I made some comments about how to recruit and manage an analytics team.

Several themes emerged that elicited considerable discussion. One revolves around the analogy to cars. Should schools teach students how to drive a car or how to build a car? My view is that today, almost all statistics courses teach building, and if there is excess time (however scarce), may cover a tip or two on driving (like, pay attention to the cyclists). For most people in business it is not necessary to know how to build a car but essential to understand how to drive one.

This leads to a related point. The decision makers at most corporations are not quantitative thinkers. Is it better to give corporate executives technical training so they understand what statisticians are saying? Or is it better to train statisticians to speak to non-technical people instead? Readers of my blogs won't need to ask where I stand on this. Unfortunately, I sense that outside of this room, my opinion is not in the majority.

PS. An attendee who works at Capital One pointed out that their business leaders typically have quantitative training, and would demand technical details. Those are the cultures that would be most comfortable for statisticians but today are exceptions rather than the rule.

***

One of my slides received much attention. Nathaniel first asked for it to be re-posted, and it stayed up there for much of the session. The slide points out that analytics professionals do best if they can succeed on three types of skills: technical (obviously), business thinking and "intangibles". Business thinking is often the hardest to develop because quantitative training in college or grad school (outside of business schools) just do not teach it. For me, intangibles are the hardest to find, mainly because it is impossible to gauge in an interview.

***

Another theme concerns which types of companies are investing in analytics. In his career, Dan has sold analytical solutions to many companies and he told us that only large corporations have the resources to do so. I am sympathetic to this point of view because analytics has prerequisites, the most obvious being an accessible, somewhat accurate database. Nathaniel, however, has been working with many small companies, so what Dan and I see as a gap he sees as opportunities.

***

EdLab at Teachers' College does a lot of interesting things, like creating technologies for the classroom and libraries, blogging, and analyzing data sets in the education sector. I had a great time speaking to this group, which included summer interns (hello, the one who asked about the difference between observational study and experimental study, perhaps working on a problem set?).

The most intriguing and unexpected question was: to do well in this business, do you have to read a lot? This is where I stumbled into a spaghetti carbonara analogy while mixing metaphors with the gray flannel, with which I have already been associated. Basically, statistics is not pure mathematics, there is not one correct way of doing things, there are many different methodologies, like there are hundreds of recipes for making carbonara. What statisticians do is to try many different recipes (methods), and based on tasting the food (evaluating the outcomes), we determine which recipe to use. Because of this, statisticians need to be well-read, to keep up with what are the new methods being developed.

Another attendee -- a reader of Junk Charts -- asked about why I prefer graphics that are clearer but less "engaging" than the original. By "engaging", he meant telling stories. This gives me an opening to point out that while I have written extensively about the shortcomings of "infographics", one aspect I like a lot is the use of narrative in this type of charts, the use of multiple charts to illustrate complex concepts. Like going from photographs to a movie.

If in doubt, ask, ask and ask again 2

In a previous post, I indicated that the NYT article on the spinal-tap test for Alzheimer's contained insufficient data for readers to fully buy its supposedly super predictive ability. Reader Chris P. looked up the specificity, and found it rather lower than we would have thought. This doesn't mean the test is bad, it is just not "perfect".

Now, let us assume the test is good enough. Let's think about the implications.

There are two basic ways by which the spinal tap proteins could be related to Alzheimer's. Either that there is a direct causal path, or that some underlying process causes both the deposit of spinal tap proteins and Alzheimer's. The article mentions a "prevailing hypothesis" that amyloid and tau accumulation are necessary for Alzheimer's. That would be consistent with the first possibility.

There is a second possibility, which is that what we observe is only correlation, not causation. This could happen if some other mechanism causes Alzheimer's but that mechanism also creates spinal tap proteins.

***

The researchers reported that of all those known to have Alzheimer's, almost all of them have spinal-tap protein deposits. This raises the spectre of reversing the causal path. What if these protein deposits are a result of having Alzheimer's?

The point is that establishing the cause of a disease is no easy matter. In Chapter 2, I looked at tracing foodborne disease outbreaks; explaining other diseases are no less challenging.

Update: seminar at Columbia

For those in the New York area, I will be giving a talk tomorrow (Aug 11, Wed) at noon at Columbia's EdLab. The talk will cover a topic from the book, and what about it is not typically discussed in statistics courses. See here for an abstract.

If in doubt, ask, ask and ask again 1

Currently (Tuesday), the top story on New York Times's website is the one about spinal taps as a predictor of Alzheimer's.

In short, the researchers are making claims of "perfection" (or near-perfection), that the presence of certain proteins in one's spinal fluid is certain proof that one will eventually develop Alzheimer's.

If you've read my book, especially Chapter 4 (and the associated stuff in the Conclusion), you should be able to think statistically about what is being printed on the page.

While I am quite sure that this finding is important (at least in stimulating further research), I don't think there is enough information for readers to be fully convinced. Every time someone trumpets "perfection", and particularly in forecasting and prediction, we ought to start from a position of skepticism. So in this spirit, here I go.

***

In this post, I focus my attention on how the numbers were reported in this article, and these two sentences in particular (I've numbered them for convenience):

[1] The new study included more than 300 patients in their seventies, 114 with normal memories, 200 with memory problems, and 102 with Alzheimer's disease.

...

[2] Nearly every person with Alzheimer's had the characteristic spinal fluid protein levels. [3] Nearly three quarters of people with mild cognitive impairment, a memory impediment that can precede Alzheimer's, had Alzheimer's-like spinal fluid proteins. And every one of those patients developed Alzheimer's within five years. [4] And about a third of people with normal memories had spinal fluid indicating Alzheimer's. Researchers suspect that those people will develop memory problems.

[1] - the numbers don't add up, and I'm confused by the placement of the commas. Is it that the "more than 300" were composed of three subgroups? Is it that there were four subgroups in the experiment, one of which consisted of people in their 70s? This inconsistency, in itself not deadly, can easily be fixed but it does smack of carelessness.

But what are the not-to-be-missed words in [1]? It is the qualifier in their seventies. Blink and you may miss it. This is of crucial importance because all of the study's subjects were the elderly who are most at-risk of developing Alzheimer's soon. Why is this important?

Recall that in Chapter 4, I discussed trying to pick a criminal out of a police line-up with say 10 suspects, versus trying to pick a thief out of a large-scale screening of thousands of employees at a company. It turns out the latter is a much more difficult task (because the chance of being a criminal is much lower here than in the former task), and predictive accuracy for this task is definitely worse than that for the other task, all else being equal.

Applied here, trying to predict who will develop Alzheimer's among people in their 70s is much easier than predicting among people, say in their 40s. So, we are a long way from full success.

In reporting the result, the journalist started with [2]. Bad idea. I once took a class from an experienced journalist and learned that newspapers always print the most significant news first. Supposedly, if the editor then chops off the bottom of your article, it leaves the key points intact. The study is purportedly to find a test to predict Alzheimer's. [2] tells us that if we already know that the patient has Alzheimer's, then the test will be "nearly" always positive. This is a group of people that cannot benefit from this test, or any other diagnostic test for that matter.

[3] reads like the clinching argument for the lede ("a spinal-fluid test can be 100 percent accurate in identifying patients with significant memory loss who are on their way to developing Alzheimer's disease."). The second part of [3] can be made clearer, for example, by replacing the period with a semi-colon, or by stating that it's every one of the nearly three quarters.

Now recall, in Chapter 4, I discussed the inevitable trade-off between false positive errors and false negative errors in any diagnostic system. [3] tells us the positive predictive value (PPV) of this test is close to 100%, meaning that if someone tests positive, one almost certainly will develop Alzheimer's, or put differently, the false positive rate is low.

But this is not enough! One thing we learn from the above statement is that the proportion of 70-somethings with mild memory-loss conditions will develop Alzheimer's with at least 75 percent chance. The test will indeed be "perfect" if in fact the chance is 75 percent, that is, none of the people who tested negative eventually develop Alzheimer's.

What if the chance is 80 percent (this is known as the prevalence of the disease in the population)? That means 5 percent of those who will have Alzheimer's will not be detected, i.e. have a negative spinal-fluid test result. That's 5 percent among the 25 percent who test negative. Thus, the negative predictive value (NPV) would be 20/25 = 80 percent, which is good but not perfect. (Someone can look up the journal article when it is published and let us know what the actual NPV is.)

This then reinforces the earlier point... that when say 80 percent of the tested population will develop Alzheimer's, the prediction problem is not as challenging as one might think. Even if the test were to declare everyone positive, the error rate would be only 20 percent.

The bigger point is that prediction systems must be evaluated looking at three legs of a stool: PPV, NPV and the selectivity (how aggressively are people giving positive results?) I wrote about this before in the context of terrorist prediction, reacting to a section in SuperFreakonomics.

***

Finally, [4] leaves much to be desired. The group of most interest for the prediction problem is precisely this group of people who are not currently exhibiting anything unusual. I'd be interested in knowing the research design... did they decide before the experiment is conducted for how long they will track this group? Or are they tracking this group now waiting for the moment to declare victory/failure? In any case, the verdict is not in yet.

In a future article, I will make some comments about causation vs. correlation, as suggested by this research.

Reading media reports of medical findings

The headline writer tells us "Cold cuts could cause cancer: study". This is typical of all such headlines, and it always is frustrating because without fail, these studies make claims that are much more specific. Therefore, we must read the articles carefully, and also ask clarifying questions.

***

Start with: All cold cuts, or only certain types of cold cuts? Any kind of cancer or a specific type of cancer?

The reporter tells us the researchers found a "positive nonlinear association between red meat cold cuts and bladder cancer," which is a very different thing from "cold cuts cause cancer". Besides, they found no association between beef, hamburger, steak, sausage or bacon and bladder cancer.

Next: What patients? Male/female? Age?

Answer: men and women aged 50-71, in eight US states. The article did not state which states.

***

Statistical thinkers would also wonder about this statement:

The scientists also found that people who ate the most red meat were younger, less educated, less physically active, and had lower dietary intake of fruits, vegetable, and vitamins C and E than those consuming the least red meat.

Once the cause of a disease is found, it is natural to ask the question which types of people are most at-risk, and in this case, it is natural to ask who eats the most red meat cold cuts.

However, the fact that those who eat the most red meat cold cuts also eat less fruits, vegetables, etc., and are less educated, and so on, should raise profound questions about what is the real cause of the bladder cancer! Did the researchers rule out lower consumption of fruits and vegetables (say) as a cause for increased bladder-cancer risk? (The subject of causal models is covered in Chapter 2.)

(Also, we should not tolerate the sloppiness in the language used in this (and similar) article: every time they mention "red meat", we should be reading "red meat cold cuts", and they are not the same things!)

 

 

Update: events

I'm at JSM, will participate at a panel discussion at 10:30 on Thursday (Aug 5) on "The Rising Tide of Business Analytics". We will talk about what it is like to work in analytics for businesses.

Will be signing books at lunchtime today (Aug 4) at Seattle Borders (4th ave). Come by!

The old is the new new thing

Note: Sorry for the break in posting. I'm back. Also look out for book-related events coming this fall.

***

The Sunday New York Times carried an article about "econophysics", described as a new sub-field of economics (or physics) in which researchers analyze business cycles by analogy to seismic activity. The following excerpt explains the premise well:

Financial crises are difficult to predict... because markets are not, as some traditional economists believe, efficient, self-regulating, and self-correcting. The periodic upheavals are the result of a cascade of events and feedback loops, much like the tectonic rumblings beneath the Earth's surface.

One sees how these researchers usurp the orthodoxy in economics: "markets are not efficient!", "markets don't self-correct!" (i.e. don't tend to an equilibrium), etc. Not the stuff we were taught in Econ 101.

The author finds little room to provide some context -- not surprising for this kind of article. I think two additional points are salient: that a large body of work exists in related areas of statistics; and that models borrowed from the natural sciences are ill-suited to explaining human behavior.

***

The new theory relies on several advanced statistical concepts, things left out of my book. One is the so-called "power laws", an example of which is the 80-20 (Pareto) rule. In the 1900s, Pareto discovered that 20% of Italians owned 80% of the land. Power laws are related to the "long tail" or "heavy tail" or "fat tail" phenomenon, popularized by Chris Anderson and Nassim Taleb (his Extremistan). Even though the concept only went mainstream in the past decade, statisticians have studied it since the 1950s, even earlier.

The other important concept is "long-range dependence". This is the part about aftershocks, about how one big earthquake leaves the earth trembling long after. In elementary statistics, we make the assumption that the world operates according to a series of coin flips, that is to say, what happens next is a new coin flip (this is called "independence").

Later, students learn that some systems have memory, and then we assume that the present state of the world captures everything that has happened in the past (this is called the "Markov" property). Google's page rank algorithm is a famous application of this principle.

Neither of these assumptions prove useful when it comes to modeling financial markets. But this is not really new. Benoit Mandelbrot is usually given credit for recognizing this in the 1960s; see his "The (Mis)Behavior of Markets" for a gentle overview of his work. Another useful reference is Barabasi's "Linked". Nassim Taleb is its most recent, and vocal, evangelist.

These concepts have been actively studied by statisticians, most notably by those working in environmental statistics, the study of earthquakes, floods, etc., and also by actuaries in the catastrophe insurance sector. More recently, Internet systems are thought to exhibit similar behavior.

***
Speaking of Taleb, I find his quote in the NYT article intriguing. He said, "The physical world cannot surprise us massively. The financial world can." The reporter uses it in support of the emergence of econophysics. I interpret Taleb to be voicing caution about over-hyping this emerging field by pointing out the futility of thinking that human behavior can be fully characterized by physical equations.

What separates econophysics from earthquakes and floods is the human element. Volatility in the markets are hugely profitable for some people, and volatility in markets can be humanly induced (unlike earthquakes as far as we know). People's moods, greed, passions, fears, flakiness, irrationality are not to be contained by equations. This inherent "uncertainty" can be incorporated into a scientific model but the resulting forecasts will also be less certain, which does not lend itself to predicting future crises, let alone finding ways to prevent them.

***

A few months ago, I presented a summary of the materials about controlling traffic congestion on highways (in Chapter 1) to electrical engineers at Princeton. These engineers are experts in thinking about congestion in communications networks (like phone networks and the Internet), and they immediately recognized that highway congestion presents a whole new challenge. In the information highway, engineers can program the routers to send data packets here and there, and the packets will follow their instructions; on the highway highway, traffic engineers can "advise" drivers what to do but the drivers will do whatever they want.

So there you have it, some context for thinking about this new field. I hope they help break new ground.