Derman, Rodrik and the nature of statistical models

I just finished Emanuel Derman's new book, "Models Behaving Badly", which is a good introduction to the philosophy of statistical models. The topic has been swirling in my head after also having read this article by economist Dani Rodrik, who reflected on the recent walkout by some Harvard students of their introductory economics course.

***

In Rodrik's view, the students were right to protest the economics profession because the economic models being taught in the classroom are too simplistic. He paints a particularly eye-opening - and damning - scenario: in the undergrad classroom, as well as in public, the economist admits no doubts about his ideologies (such as "free trade", "free market") but in his "advanced graduate seminar on ... theory", the same professor would debate with skeptics, leading to a "heavily hedged statement" after "a long and tortured exegesis". The statement would begin with "if the long list of conditions I have just described are satisfied, .."

I could imagine Derman entering that graduate seminar, and declare everything as nonsense. (Derman currently teaches in the Financial Engineering program at Columbia, and previously worked on Wall Street as a "quant" building economic models, after spending his graduate career working with models of the physical world.) "Models Behaving Badly" is about how economic models can go off-track, how frequently they do, and why modelers must behave modestly. Derman would argue that Rodrik's "long list of conditions" are almost never satisfied.

Derman:

There is a crucial difference between the assumptions made by the Black-Scholes Model and the assumptions made by a souffle recipe. Our knowledge about the behavior of the stock markets is much sparser than our knowledge about how egg whites turn fluffy.

He goes on to argue, perhaps unexpectedly, that the Black-Scholes Model is "the best model in all of economics". He aims his criticism squarely at the sacred cow of financial economics, the "Efficient Market Hypothesis".

***

Rodrik does not believe that the economics profession needs better models. He claims "Macroeconomics and finance did not lack the tools needed to understand how the crisis arose and unfolded." The fault of the profession was to have trusted the wrong models (ones assuming efficient and self-correcting markets). He believes that this bad choice of models is facilitated by "excessive confidence in particular remedies - often those that best accord with their own personal ideologies."

It isn't clear to me how Rodrik proposes to resolve the ideology problem. In fact, his citation of another economist Carlos Diaz-Alejandro perfectly captures the heart of the issue: "by now [1970s] any bright graduate student, by choosing his assumptions... carefully, can produce a consistent model yielding just about any policy recommendation he favored at the start."

The diesease is more than ideological. Reading behind the lines, I think these models are far too complex for their own good. They cannot be falsified with observed data. They can be made to support any ideology. This leads me to two observations:

• The forecast is dire: So long as this type of modeling persists, the choice of models will be based on ideology only
• Even Rodrik's diagnosis is suspect: perhaps it is only in hindsight that one can determine which model out of that infinite universe of models is "a bad model".

***

Returning to the protesting Harvard students, Rodrik describes the discontent of the undergrad economics syllabus: "it is as if introductory physics courses assumed a world without gravity, because everything becomes so much simpler that way."

In making this analogy, Rodrik is giving economic models the status of models in physics. He's saying that there are simplified models in both disciplines which don't fit reality well, but there are complex models in both disciplines which work well.

Derman would beg to differ. Originally trained as a physicist, he now freely admits that "financial modeling is not the physics of markets". He spends a great portion of the book showing why economic models can never aspire to the status of physics models.

Reading Rodrik's analogy, one senses that he has yet to arrive at Derman's port. Rodrik continues to make parallels between physics and economics. But I know of no introductory physics course that assumes a world without gravity - the major omission is Einstein's relativity. There is, in fact, a huge difference between Newton's theory of mechanics and, say, the Capital Asset Pricing Model. Students who learn Newton's laws can explain how the world works without ever knowing any relativity theory. Newton's theory can stand on its own. Not so the simplistic economics models. As Derman points out, simple economics models are easily invalidated by observed data.

***

My own view, informed by years of building statistical models for businesses, is more sympathetic with Derman than Rodrik. There is no way that economic (by extension, social science) models can ever be similar to physics models. Derman draws the comparison in order to disparage economics models. I prefer to avoid the comparison entirely.

The insurmountable challenge of social science models, which constrains their effectiveness, is that the real drivers of human behavior are not measurable. What causes people to purchase goods, or vote for a particular candidate, or become obese, or trade stocks is some combination of desire, impulse, guilt, greed, gullibility, inattention, curiosity, etc. We can't measure any of those quantities accurately.

What modelers can measure are things like age, income, education, past purchases, objects owned, etc. Nowadays, we can log every keystroke you type on your smartphone (link). That models are even half-accurate is due to the correlation of these measured quantities with the hidden drivers of our behavior but this correlation is only partial.

Now add to that, the vagaries of human behavior.

 

 

 

Comments

[Michael Nute]

I think the models that Derman and Rodrik are talking about are actually quite different from each other, and that they're both right in their separate assertions. Statistical models of social behavior suffer from exactly the downfall you and Derman describe, but economic models are usuallly designed as theoretical setups between actors with the goal of predicting equilibrium behavior. Then statistics and historical data are used to show that this behavior has indeed resulted. The downfall of economic models is that they are hyper-dependent on assumptions and therefore lead to conclusions that border on the tautological.

The efficient market hypothesis is the classic example. Like all economic models it refers to equilibrium behavior only, and in the presence of an infinitely large market, though these provisos tend to be omited in popular discussion. It's meant to be illustrative only, showing the incentives and profit-seeking behavior of buyers and sellers and its effects on prices. Whether this accurately describes all markets for all securities at all times here on planet Earth is another matter and is beside the point. When I hear people discuss wehther the EMH is "True" or not, my head explodes. And when students take these messages blindly into the real world without the nuances of the argument from the graduate classroom, as Rodrik describes, tragedy ensues.

The dangers of elevating these models to the level of physics though are very real, and it happens all the time. Read a finance textbook's treatment of CAPM and you'd think it was cosmic truth. I think Rodrik's analogy was more of a clumsy as-if than that kind of elevation, but the point is valid.

[Luke Lea]

Adjusting the wooden head phones? Here's what Richard Feynman had to say about science in general: http://www.lhup.edu/~DSIMANEK/cargocul.htm

[Dean Eckles]

I wonder what you think of John Sutton's more academic (but short and readable) book Marshall's Tendencies: What Can Economists Know?.

It addresses the difference between having a model that is a rough approximation and one that isn't even that.

[John]

Students can describe the world with Newtonian mechanics, not explain it.

And you forgot the requisite "assume a con-opener" joke.

[ezra abrams]

Why does no one ever talk about money power and status when talking about economists
After all, when we talk about what economists do, we are talking about a small group of people at harvard, mit , uchicago etc who set the tone; James Galbraith has a lot on how the elites define the subject
and being a econ prof at harvard is a good gig; good pay, power over students grads and postdocs (and power is great in a job, better then pay for most people) the chance to work for hte president (who has a council of econ advisors; since almost all of us care more about our live lives and children then about money, why doesn't the POTUS have a council of sex adivsors...)
using math is like a union card, it keeps people from competing
and you can write texts and charge students an arm and a leg

[digital options]

I like your post. As an economics student I agree that the economic models being taught in the classroom are too simplistic. I think I am going to buy this book, it sounds very interesting.

[Audrey Lyn]

Thanks a lot for interesting article!