Big Data, Plainly Spoken (aka Numbers Rule Your World)

Having your cake and eating it: the numbers edition

First, Greece was involved in a web of lies relating to its financial situation. Then, it spread to the rest of Europe, trying hard to pretend that the center will hold. Now, the bystanders are partaking in the game of who's the most brazen when it comes to abusing numbers.

According to Bloomberg, the economist Ken Rogoff said this:

There’s at least as much as an 80 percent chance that Greece will leave the 17-nation common currency in the next 10 years.

How should we read this prognostication?

Does he mean

There’s at least an 80 percent chance that Greece will leave the 17-nation common currency in the next 10 years.

in which case, the probability is between 80 and 100 percent, or does he mean

There’s as much as an 80 percent chance that Greece will leave the 17-nation common currency in the next 10 years.

in which case the probability is between 0 and 80 percent,

or is he combining both cases, in effect, making a prediction of 0 to 100 percent?

Taking things further, the headline writer at Bloomberg puts his/her own spin on things. The headline tells us:

Greece will eventually leave the Euro, Rogoff says

Way to turn a probability into a certainty!

 

 

 

Tales of poor CEOs

Story time!

James Kwak (at Baseline Scenario) talks about the link between CEO pay and business performance here. He cites results from a paper by Cazier and McInnis that "people from successful companies don’t deserve the pay premium because the higher the premium they are able to command, the less well they are likely to do." So far so good.

Then James launched into the following:

This should not be too surprising. The more of a superstar someone is at Company A, the more likely the board of Company B is to overlook all the things that make her a bad fit for Company B—like not having experience in the industry, or with the new company’s customer base, or having led Company A through a different phase of its lifecycle than Company B, or not having the skills that Company B needs at that point in time, or any number of other things. The more reasons for concern that Board B overlooks, the more likely the new hire is to do badly. In the end, you get something vaguely like the Peter Principle: the more successful Company A is, the more market power its CEO has, and the more likely she is to be overpaid to be CEO of a company she is not qualified to lead.

This paragraph contains a dozen assertions that have no support from the Cazier-McInnis article. These may be interesting ideas but they are pure speculation on the part of James. The trouble with such ex-post explaining is the narrative fallacy that Nassim Taleb likes to talk about. If Cazier and McInnis were to discover that CEO pay is justified by future business performance, there would have been another dozen assertions one could come up with to "explain" the correlation.

 

 

 

 

 

How good is your data?

The following articles discuss the behind-the-scenes process of preparing data for analysis. It points to the "garbage in garbage out" problem. One should always be aware of the potential hazards.

"The murky world of student-loan statistics", Felix Salmon (link)

At the end of this post, Felix found it remarkable that the government would not have better access to the data. The same sentiment was expressed at a recent presentation by the data team at Bundle.com, in which they described the extraneous strenuous process by which they matched the names of merchants on credit card statements to a database of known merchants. One would think the credit card companies would be able to pass along the merchant identifiers but they don't or can't.

"European debt: the big picture", Simon Johnson (link)

Simon points out that while the New York Times did a fantastic job with this visualization of the European debt linkages, one should notice what wasn't present on the chart, namely, the murky world of derivatives and not knowing it denies us knowledge about the exposure of U.S. banks to this potentially devastating problem.

Musing over data science

You have to give it to the computer scientists. They are like the branding agencies of the engineering world. Everything they touch turns to PR gold. Steve Jobs, of course, was their standard bearer, with his infamous "reality distortion field". Now, they are invading the statistician's turf, and have already re-branded us as "data scientists". MIT Technology Review noted this event recently.

Data science

Machine learning

Knowledge discovery

Neural networks

Pattern recognition

Artificial intelligence

Statistics, statistical science, management science, decision science, operations research, even analytics, pale in comparison.

I'm impressed.

***

The blogger at Tibco Spotfire quoted me in this piece about "How to speak like a data scientist".

Perhaps I wasn't as clear in that source as I was in some of my other presentations. The biggest challenge for any analytics (oops, data science) role is how to communicate the statistical information to a non-technical audience. There is a language barrier. One side speaks mathematics, the other side speaks English. Two strategies are available. You could teach the one side English so both sides speak English; or you could teach the other side mathematics so both sides speak mathematics.

All too often, technical people use the second strategy. All too often, the second strategy fails. My biggest advice to you is to try the first strategy.

Omics, metrics, and tricks

The Deal Professor's take on how bankers conspired with Groupon and Zynga management teams to play with financial numbers is a must-read.

"Grouponomics and Zyngametrics, but Few Sound Numbers" NYT

***

Quite apart from the dubious business models of these companies, people who might consider buying into the IPOs should consider the signals from management: at Groupon, the investors who provided funds in 2010 have already cashed out previous investors (what would the motivation be for new investors to fund the exit of previous owners?); and at Zynga, despite wanting the public to buy ownership of the company, these new owners would have no effective voting rights since insiders will retain almost all of them (investment without representation!).

Get rid of those numbers people!

A "long-time retail industry insider" slams the numbers people:

I've never seen a numbers guy come in and be successful in a fashion company. I've never seen one example of it.

Wow, not one, ever.

Howard Davidowitz's talking about the CEO of the Gap, who is successful from the drugstore, and other kinds of businesses. The Gap just announced shrinking the number of stores by 20%. Probably a good move. According to Davidowitz, just get rid of this numbers guy, and everything will get fixed, pronto!

I'm not kidding. He said that and more in this report.

 

It takes (a lot) to build a village

The people involved in putting together the Cross Validated website contacted me recently for feedback. Cross Validated is a new site brought to you by the people who host the excellent Stack Overflow website.

I've been a regular user of Stack Overflow. It's a Q&A website where programmers ask and answer each other's questions. You want to find the best way to compute the union of two Python lists? You can find lots of people offering up their solutions; see here for instance. An initial question leads to a multi-thread answer in which people suggest code and others critique and vote on the suggestions. One learns different solutions, as well as different solution criteria (e.g. speed of the code, style of the programming, conciseness, elegance). Most importantly, one can lift code and place directly into one's work.

Cross Validated is their attempt to do this for the statistics/data mining community. It's in infancy stage as there are very few threads right now. Given my positive experience with Stack Overflow, I'd like to see this new site grow and prosper but I see some challenges ahead:

• For clearcut programming questions like "how do I turn the axis labels sideways in an R graphic?", this site could easily replicate the success of Stack Overflow. But lots of statistical questions are not so clearcut. Something like "What should k be in a k-fold cross validation?" is unlikely to generate anything like a majority opinion, and instead will likely degenerate into a skirmish over "Does cross-validation work?", which of course also does not have an answer
• For general programming questions, it is often possible to extract little pieces of a whole program at which to shine a light. For instance, the union-of-lists question could be a tiny part of a very large program; other users need not know anything about that program to contribute to the thread. Many statistical problems, including some programming related ones, do not have this modular property.
• Solutions to statistical problems are often data-specific. It is usually difficult to answer such questions without having a data set (at least a toy one) to play with. The same question may have different answers depending on which data is being analyzed. Is there a way to share data with other users?
• Let's say data sharing is solved. How do we encourage people to participate? It's easy to put up some sample Python/Java/C code but it's quite a bit of effort to write code and test it on a data set and then write up one's answers.
• Something like finding the union of Python lists will have the same solution for any user for any problem. But I don't think it helps me to know that someone else had success with some particular methodology with an unrelated dataset.

 

PS. See also this Gelman post about R-help, which is a mailing list for R users. As I said there in a comment, I think R-help should be thought of "tech support" where users can get answers from developers while Stack Overflow is a "user community" in which users help each other. At least that's how I feel when I interact with these two services.

 

Prostate cancer screening under scrutiny

Officials now recommend healthy men against routine screening for prostrate prostate cancer. See here. I discussed this issue before.

A few points of interest:

• The clinical trials have mixed findings on the benefits. The harm of unnecessary treatment is real. The harm of false positives outweigh the benefits

• Men at risk of getting prostrate prostate cancer are not part of this recommendation. It applies only to healthy men.

• "Of the men who had three or four rounds of screening, the false positive rate was 12 to 13 percent." I think this means that among men who don't have prostate cancer, 12 to 13 percent will be told they have cancer three or four consecutive times! This level of inaccuracy is quite astounding.

• Dr. Herbert Lepor (urology professor): "The notion that prostate cancer is not a threat to the well being of men is simply wrong." I'm shocked that an academic at NYU could equate a criticism of a detection method with a denial of an illness, without losing a beat.

• "Critics say the studies were not long enough to show a benefit." Let us do a thought experiment. We start screening men from 21 years old. Each year, we issue large numbers of false positive results. By the time these men become at risk for prostate cancer, a large proportion of them would have received at least one positive result. This screening policy will surely catch most of those who would develop the cancer. Now imagine, that we didn't have a screening technology, we just randomly tell a certain proportion of men they are positive. This "false screening" will also identify most of those afflicted.

• Further, unless prostate cancer has some kind of very long incubation period, waiting longer shouldn't improve the screening accuracy. The study referenced in the article also tracked patients for 10 years. This type of criticism is not serious unless they come prior to the test results being issued.

• Note also false positive errors are invisible. Once the little tumors are removed, it is not possible to know if they would have developed into anything malicious. Invisible errors mean the people committing them can't be called into account. By contrast, false negatives are definitely public. For more, see Chapter 4.

 

How not to critique models

I am following, with bemusement, the skirmish over the IS-LM macroeconomic model between Brad Delong and Tyler Cowen.

Here are three posts by Delong who likes the model:

Yet more rites of tribal solidarity among right-wing econ...

IS-LM

The tribal dislike of John Hicks and IS-LM

Here are three by Cowen, who dislikes the model:

Scott Sumner on IS-LM

Brad Delong on IS-LM

Why I do not like the IS-LM model

***

I will leave aside the macroeconomics (no expertise). What I care about is how one should, and should not, critique "models".

Since a model is an abstraction, a simplification of reality, no model is above critique.

I consider the following types of critique not deserving:

1) The critique that the modeler makes an assumption,

e.g. "it fudges the distinction between real and nominal interest rates". Making assumptions is not inherently bad, making bad assumptions is a problem but not all assumptions are bad.

2) The critique that the modeler makes an assumption for mathematical convenience,

e.g. "Don't assume they are the same, just to squash the two curves onto the same graph." Almost all assumptions are made for mathematical convenience. The inappropriate use of the Gaussian assumption, that most unpardonable of sins according to Taleb, is almost always invented to render the math tractable. But not all assumptions that simplify the math are bad assumptions.

3) The critique that the model omits some feature,

e.g. "those aggregate curves are not invariant to expectations", because this critique is no different from saying the modeler makes an assumption (see #1) More, what is a "bad" assumption? How does one determine which assumptions are bad among the set of all possible assumptions?

4) The critique that the model doesn't fit one's intuition,

e.g. "the model leads you to believe that interest rates are more important than they probably are". The model should fit reality (the data); it doesn't need to fit anyone's intuition.

5) The critique that the model fails to make a specific prediction,

e.g. "this distinction really matters when you're trying to predict the macro effects of 'window breaking'". No model, especially a macroeconomic model that can issue a large number of predictions, will ever predict everything. Not all predictions are equally important. One must agree on which predictions are the  most important to get right, and make judgment based on the entire list of predictions.

Above all, a serious critique must include an alternative model that is provably better than the one it criticises. It is not enough to show that the alternative solves the problems being pointed out; the alternative must do so while preserving the useful aspects of the model being criticized.

***

This whole debate reminds me of the climate change model controversies. I am not aware of an alternative model from those who dislike the consensus model. Until they offer such an alternative, their critique cannot be taken seriously, I'm afraid.

The underlying belief -- on both sides of the macroeconomic divide, it appears -- that someone's model can be proved "wrong" and thus discarded for all eternity is as dubious as the belief that a model can be proved "right".