I just finished Emanuel Derman's new book, "Models Behaving Badly", which is a good introduction to the philosophy of statistical models. The topic has been swirling in my head after also having read this article by economist Dani Rodrik, who reflected on the recent walkout by some Harvard students of their introductory economics course.
In Rodrik's view, the students were right to protest the economics profession because the economic models being taught in the classroom are too simplistic. He paints a particularly eye-opening - and damning - scenario: in the undergrad classroom, as well as in public, the economist admits no doubts about his ideologies (such as "free trade", "free market") but in his "advanced graduate seminar on ... theory", the same professor would debate with skeptics, leading to a "heavily hedged statement" after "a long and tortured exegesis". The statement would begin with "if the long list of conditions I have just described are satisfied, .."
I could imagine Derman entering that graduate seminar, and declare everything as nonsense. (Derman currently teaches in the Financial Engineering program at Columbia, and previously worked on Wall Street as a "quant" building economic models, after spending his graduate career working with models of the physical world.) "Models Behaving Badly" is about how economic models can go off-track, how frequently they do, and why modelers must behave modestly. Derman would argue that Rodrik's "long list of conditions" are almost never satisfied.
There is a crucial difference between the assumptions made by the Black-Scholes Model and the assumptions made by a souffle recipe. Our knowledge about the behavior of the stock markets is much sparser than our knowledge about how egg whites turn fluffy.
He goes on to argue, perhaps unexpectedly, that the Black-Scholes Model is "the best model in all of economics". He aims his criticism squarely at the sacred cow of financial economics, the "Efficient Market Hypothesis".
Rodrik does not believe that the economics profession needs better models. He claims "Macroeconomics and finance did not lack the tools needed to understand how the crisis arose and unfolded." The fault of the profession was to have trusted the wrong models (ones assuming efficient and self-correcting markets). He believes that this bad choice of models is facilitated by "excessive confidence in particular remedies - often those that best accord with their own personal ideologies."
It isn't clear to me how Rodrik proposes to resolve the ideology problem. In fact, his citation of another economist Carlos Diaz-Alejandro perfectly captures the heart of the issue: "by now [1970s] any bright graduate student, by choosing his assumptions... carefully, can produce a consistent model yielding just about any policy recommendation he favored at the start."
The diesease is more than ideological. Reading behind the lines, I think these models are far too complex for their own good. They cannot be falsified with observed data. They can be made to support any ideology. This leads me to two observations:
- The forecast is dire: So long as this type of modeling persists, the choice of models will be based on ideology only
- Even Rodrik's diagnosis is suspect: perhaps it is only in hindsight that one can determine which model out of that infinite universe of models is "a bad model".
Returning to the protesting Harvard students, Rodrik describes the discontent of the undergrad economics syllabus: "it is as if introductory physics courses assumed a world without gravity, because everything becomes so much simpler that way."
In making this analogy, Rodrik is giving economic models the status of models in physics. He's saying that there are simplified models in both disciplines which don't fit reality well, but there are complex models in both disciplines which work well.
Derman would beg to differ. Originally trained as a physicist, he now freely admits that "financial modeling is not the physics of markets". He spends a great portion of the book showing why economic models can never aspire to the status of physics models.
Reading Rodrik's analogy, one senses that he has yet to arrive at Derman's port. Rodrik continues to make parallels between physics and economics. But I know of no introductory physics course that assumes a world without gravity - the major omission is Einstein's relativity. There is, in fact, a huge difference between Newton's theory of mechanics and, say, the Capital Asset Pricing Model. Students who learn Newton's laws can explain how the world works without ever knowing any relativity theory. Newton's theory can stand on its own. Not so the simplistic economics models. As Derman points out, simple economics models are easily invalidated by observed data.
My own view, informed by years of building statistical models for businesses, is more sympathetic with Derman than Rodrik. There is no way that economic (by extension, social science) models can ever be similar to physics models. Derman draws the comparison in order to disparage economics models. I prefer to avoid the comparison entirely.
The insurmountable challenge of social science models, which constrains their effectiveness, is that the real drivers of human behavior are not measurable. What causes people to purchase goods, or vote for a particular candidate, or become obese, or trade stocks is some combination of desire, impulse, guilt, greed, gullibility, inattention, curiosity, etc. We can't measure any of those quantities accurately.
What modelers can measure are things like age, income, education, past purchases, objects owned, etc. Nowadays, we can log every keystroke you type on your smartphone (link). That models are even half-accurate is due to the correlation of these measured quantities with the hidden drivers of our behavior but this correlation is only partial.
Now add to that, the vagaries of human behavior.