Illusion of success

A columnist in the NYT asserted that a "little-known market timing model" has had an "impressive track record" over the "long run".  The following graph was offered:

This is an illusion of success entwined in an illusion of charting. 

First on the charting: as discussed before, charts with double axes, each bearing different scales, leave the door wide open for manipulation.  I haven't studied their model and thus do not know if there is a one-to-one relationship between their "scale" and the return differential between the S&P and T-bills.  My suspicion is that someone superimposed the two time series, shifting them up and down to minimize the discrepancy.  How else does one decide how to align the two vertical scales?  (This was exactly the issue with Friedman's Petropolitics Law.)

Now, the smell of success.  Meticulous readers will notice large gaps between the two lines since roughly the late 80s.  Is it a surprise then that the model was constructed using data through 1989 and published in 1993?  The columnist noticed, noting that "the model has had a mixed record in the 13 years since their study appeared."

And yet, he proffered this defence:

Even after the incorrect forecasts in the 1990's are taken into account, the model's overall record is good enough to be statistically meaningful and not likely to be mere luck.

His idea of "overall record" was to study 1968 through 2006, which includes 22 years "covered in the professors' original study" and 17 years since.

To put it simply, he took the 13 years of poor performance, and mixed in 26 other years, of which 22 were used in constructing the original model.  This is equivalent to averaging training accuracy and validation accuracy.  (Training data is used to build a model; validation data is used to test its performance on previously unseen data.)  Training accuracy is always high or else the modeler would have rejected the model; thus, no honest one would make claims based on training accuracy.  Measured on validation data alone, this model is most definitely foul, as the right-hand-side of the graph vividly attests.

Here, we have seen an illusion of charting used to prop up an illusion of success.

And one more question, even if this model is acceptable, how is one to take advantage?  Such models have little utility unless one also has a sane investment strategy.  Having a model and having a useful model are worlds apart.

Reference: "An Old Formula That Points to New Worry", New York Times, June 18 2006.

Comments

[zbicyclist]

I love the fact that they say the model had a mixed record in recent years. It looks like a naive model (tomorrow = today) would beat it since 1995.

[Andreas]

Looks to me that the model is the one lagging. Getting a high corr is no problem, getting a good "actual" forecast is a lot worse.

Two axis graps might be great, depends on the purpose.

As allways, there is some magic in the maker!

[Todd]

How far ahead was the model making its predictions? Wondering if someone would have the time to use the data to make stock trades. Also, I would have preferred the author spend $100 and follow the model and see what they have left in 2006. My guess is they would have gotten wiped out in the 90s.