How not to critique models

I am following, with bemusement, the skirmish over the IS-LM macroeconomic model between Brad Delong and Tyler Cowen.

Here are three posts by Delong who likes the model:

Yet more rites of tribal solidarity among right-wing econ...

IS-LM

The tribal dislike of John Hicks and IS-LM

Here are three by Cowen, who dislikes the model:

Scott Sumner on IS-LM

Brad Delong on IS-LM

Why I do not like the IS-LM model

***

I will leave aside the macroeconomics (no expertise). What I care about is how one should, and should not, critique "models".

Since a model is an abstraction, a simplification of reality, no model is above critique.

I consider the following types of critique not deserving:

1) The critique that the modeler makes an assumption,

e.g. "it fudges the distinction between real and nominal interest rates". Making assumptions is not inherently bad, making bad assumptions is a problem but not all assumptions are bad.

2) The critique that the modeler makes an assumption for mathematical convenience,

e.g. "Don't assume they are the same, just to squash the two curves onto the same graph." Almost all assumptions are made for mathematical convenience. The inappropriate use of the Gaussian assumption, that most unpardonable of sins according to Taleb, is almost always invented to render the math tractable. But not all assumptions that simplify the math are bad assumptions.

3) The critique that the model omits some feature,

e.g. "those aggregate curves are not invariant to expectations", because this critique is no different from saying the modeler makes an assumption (see #1) More, what is a "bad" assumption? How does one determine which assumptions are bad among the set of all possible assumptions?

4) The critique that the model doesn't fit one's intuition,

e.g. "the model leads you to believe that interest rates are more important than they probably are". The model should fit reality (the data); it doesn't need to fit anyone's intuition.

5) The critique that the model fails to make a specific prediction,

e.g. "this distinction really matters when you're trying to predict the macro effects of 'window breaking'". No model, especially a macroeconomic model that can issue a large number of predictions, will ever predict everything. Not all predictions are equally important. One must agree on which predictions are the  most important to get right, and make judgment based on the entire list of predictions.

Above all, a serious critique must include an alternative model that is provably better than the one it criticises. It is not enough to show that the alternative solves the problems being pointed out; the alternative must do so while preserving the useful aspects of the model being criticized.

***

This whole debate reminds me of the climate change model controversies. I am not aware of an alternative model from those who dislike the consensus model. Until they offer such an alternative, their critique cannot be taken seriously, I'm afraid.

The underlying belief -- on both sides of the macroeconomic divide, it appears -- that someone's model can be proved "wrong" and thus discarded for all eternity is as dubious as the belief that a model can be proved "right".

 

 

Comments

[Thomas Colthurst]

I'm not sure why you say that Cowen and Sumner don't include any alternative models. Sumner states that level targeting "should be the standard model" in his post, and Cowen explicitly lists four alternatives here: http://marginalrevolution.com/marginalrevolution/2011/10/is-lm-keynesianism-why-not-and-which-alternatives.html

I also think that requiring that the alternative model be *provably better* than the critiqued model is an impossibly high standard outside of pure math. I think it is enough that the alternative be simply *more useful*. Sometimes this means preserving the useful parts of what is being criticized, sometimes it doesn't. The canonical example of the later is Copernican astronomy initially giving *less* accurate predictions than the Ptolemaic model.

Finally, if there is anyone who does not suffer from the belief that models can be proved wrong and discarded, it is Tyler Cowen. The man is methodologically pluralistic to a fault; his prose often reads like someone translating Bayesian model averaging into English. His critiques of IS-LM are best understood (well, by statisticians, at least :) ) as demonstrating a low value of P(model | evidence), particularly in the context of understanding contemporary problems.

[Kaiser]

Thomas: thanks for the comment. I did read the post on alternatives which came after my post. However, I don't see anything there. As some commenters on Tyler's blog point out, it's a hotchpotch of ideas that do not add up to a model. At least that is what it sounds like to me as an outsider (not an economist).
How do you show the alternative is "more useful" when you can't show it's "provably better"? I'm not sure I understand the difference.
The trouble I have is that whatever alternative Tyler is suggesting, a different researcher can go and list the assumptions that don't make sense, the features that are omitted, the predictions that fail, etc. That's because every model suffers from these types of issues.

[Vince]

Suppose a model, if applied, will kill a kitten you're fond of.
Suppose you can raise valid and persuasive criticisms this model but can't construct an alternative to it.
Why exactly is it the case that said criticisms can't be take seriously?

A pretense of knowledge can do real harm. It's better to be ignorant than to have faulty knowledge.