Big Data, Plainly Spoken (aka Numbers Rule Your World)

The most cited and most butchered statistical law

The past week, you are unable to avoid, if you read the business news, mention of the "law of large numbers". We are led to believe that tech giants like Apple and Amazon are suffering from this statistical law. For example:

Apple: Newest Victim of the Law of Large Numbers (CNet)

New York Times Paywall Growth Slows (Columbia Journalism Review): "Much of this is due to the law of large numbers"

Amazon Growth Slows While Profit Margin Expands (Yahoo News/Reuters): "The law of large numbers is affecting Amazon, too."

Is WebRTC a Threat to Skype? (UC Strategies): "The law of large numbers provides protection [to Skype against the threat of a new technology]."

***

This is time to re-read my 2010 entry, called "The Many Laws of Large Numbers" (link). Also Chapter 3 of Numbers Rule Your World covers disaster insurance, and Chapter 5 covers lotteries, both of which are proper applications of the statistical Law of Large Numbers.

Statistics as inverse probability

Statistics is sometimes described as inverse probability.

In a typical probability problem, one starts by positing that a certain quantity has some given probability distribution, say the number of people entering a bank branch follows a Poisson distribution, and then goes on to compute probabilities such as the chance that more than 100 people (max capacity) require service at the same time. In a typical statistical problem, one observes the distribution, that is to say, the number of patrons over a period of time, and then finds a model to best represent the observed pattern.

***

In Charles Wheelan's Naked Statistics (I'll post a proper review when I finish the book), he offered readers the following:

If you flip a fair coin 1,000,000 times and get 1,000,000 heads in a row, the probability of getting a tail on the next flip is still 1/2. (p. 102)

I'm here to tell you why it isn't 1/2. And it has to do with the difference between how a probabilist thinks and how a statistician thinks.

A probabilist starts with a couple of truths:

• the coin is fair
• each coin toss is an independent event

and from there, he or she computes the probability of different outcomes: one such outcome is to obtain 1 million heads followed by 1 tail in 1 million and one tosses. The probabilist tells us that that outcome is extremely rare but possible.

Now, if I find that 1,000,000 coin tosses produced 1,000,000 heads, I reject the notion that the coin is fair! Based on the observed data, I am more comfortable believing that the coin is severely biased towards heads.  Therefore, my expectation of the next throw would be a very high chance of heads -- I'd certainly not conclude that the chance of heads is 50%, as Wheelan said there.

What Wheelan said is probably very commonly taught in statistics classes. This is unfortunate because in statistics, we start with the data, and figure out which probability distributions would be most consistent with the data. This is the inverse of probability modeling, in which one starts with the probability distribution.

It would be doubly unfortunate if that kind of statement shows up in a Bayesian textbook but I suspect you can find examples of that too in the section on probability.

***

If you want to understand how statisticans think, this is a great place to start.

FDA endorses masking placebos as proven drugs as a get-rich-quick scheme

That's not really what the FDA said but based on the shameful actions unearthed by ProPublica and reported by Scientific American here, one would think one can get away with this scam.

Three years ago, the FDA busted a Houston lab (based on a whistleblower report) that fabricated loads of research studies that were used by the FDA to approve about 100 drugs. Eighty-percent of those drugs were generic drugs that the lab supposedly proved as (bio)equivalent to the brand-name counterparts.

Three years later, the FDA has only reviewed new research on 21 of those drugs. The agency has refused to provide any information about the names of the drugs that have been fraudulently approved, nor provided any information about how they were reevaluating these drugs. The excuse is "confidential commercial information".

To date, the FDA has not pulled any of the drugs from the market. (Some are off the market because the pharma pulled them.) Even pharmacists do not know that those drugs are under a cloud of suspicion and continue to sell them. Some drugs still carry labels that reference the discredited studies. Some peer-reviewed articles continue to be available with no reference to the controversy (not too surprising).

The FDA claimed that so far, no harm has been reported from any of the suspicious drugs.

This leads me to my scam: if I create an injection of saline and market it as equivalent to a cancer-treating drug, I will do no harm; in fact, I may even do some good based on the placebo effect. I will make loads of money based on the belief (mislabeling) that my drug can treat cancer. The FDA seems to be saying I can do this so long as my "drug" does no harm.

The article makes no mention that anyone is being prosecuted for the fraud. The lab director does not appear to be in legal trouble. The pharmaceuticals who made a lot of money selling these drugs are portrayed as peripheral players. The chemists who faked data apparently have been fired. There is also no mention of whether the FDA personnel who approved the fraudulent research are being cited for professional negligence and failing their duty of care.

***

The biggest revelation is in the last page of this long article. Former FDA agent was cited:

There are thousands of bioequivalence studies done every year, he pointed out, with each study generating thousands of pages of paper records. "Do you really think we're going to look at 100 percent of them? We're going to look at maybe 5 percent if we're lucky," he said. "Sometimes 1 percent."

So, he's saying that most drugs are approved without any oversight.

This makes me angry, and it should make you angry, too.

 

Occupational hazards in data science

An interesting episode is developing in econometrics over the very high profile Reinhart-Rogoff paper that was heavily cited as a source to "prove" that high levels of national debt impede growth. It appears that that result was based on a combination of spreadsheet errors, and bad assumptions.

1. Andrew Gelman has a great discussion here. His main concern is ethics of data analysts. This is a very important point - any experienced data analyst knows that it is extremely easy to make mistakes; in fact, if there is a data analyst who has not made errors that subsequently have to be corrected, or disseminated results that subsequently have to be revised or even retracted, one can be sure that the data analyst has not done a lot of analyses. (I'm excluding those who only work with simulated data.)

Errors come from many sources, and it appears that Reinhart-Rogoff managed to touch several important ones.

a. Processing errors - in the course of an analysis, data is often transported from system to system. Even on a spreadsheet, the data may be copied and pasted from one page to another, or formulas are set up to link from one area to the next. For a team project, the spreadsheet or dataset may pass from one person to another. A tiny moment of inattention could easily ruin the data. It appears that Reinhart-Rogoff dropped some data on the floor. I should note that "Big Data" makes this problem a hundred times harder to solve. If you only have one spreadsheet, you can in theory verify each number at different stages of the analysis. Not if you have millions of rows and thousands of columns in a dataset.

b. Bad assumptions - any analysis will contain assumptions, mostly because the data has problems or holes. Making assumptions is not itself bad. But some assumptions will turn out to be bad. In this case, the assumptions were badly explained. Critics noted that some data points were deliberately dropped from the analysis, and the conclusion was sensitive to those dropped data. Based on these reports, Reinhard-Rogoff have not convinced these critics why those data points should be dropped.

c. Bad data -the biggest reason for bad analysis is usually that the analyst does not know how the data came into being. Somebody else collected the data (or a machine did). The data was purchased from a third-party. All kinds of hidden assumptions will now affect the analysis. For example, you analyze credit-card transaction data to try to understand consumer spending patterns. You see that the billing decreased. If you're not careful, you'd conclude that consumers are spending less. But there are many other possibilities. For example, maybe credit-card companies increase their fees and then merchants set higher minimums, forcing consumers to pay cash. Frequently, you learn about this type of extraneous factors after you already disseminated the analysis.

 

2. Paul Krugman's response is also worth reading. I especially want to draw attention to his last point, which is what I have called the "causation creep" (link):

Finally, while they acknowledge the issue of reverse causation, they seem very much to be trying to have it both ways — saying yes, we know about the issue, but then immediately reverting to talking as if debt was necessarily causing slow growth rather than the other way around.

In their response to the critics, Reinhart-Rogoff's key point was "We were only arguing association, not causality." (see here and here.) I'm sure we'll find specific statements to that effect in the paper but all too often, the discussion of the implications of such research includes statements that assume the cause-effect relationship, exactly what Krugman was complaining about.

This problem is not just found in economics, although I see it a lot with the Freakonomics style of thinking. It is also very common in the medical literature, and other fields in which observational data is used. The usual ploy is first acknowledge that the data could not prove causality ("we found an association between sleeping less and snoring; our data does not allow us to prove causation."), then quietly assume that the causal link is there, and wax on the implications ("if you want to snore less, sleep less.")