Keeping one's appetite after touring the sausage factory

For those wanting to a glimpse the sausage factory that manufactures unemployment statistics, this New York Times column paints a pretty good picture.

The article even comes with a picture:

If this post is on my sister site Junk Charts, I'll say something about the chart itself. But this isn't that place. (Ignore the darker section of the chart on the right side.)

First, look at the red columns in the bottom of the chart; these represent so-called "benchmark revisions" to the job gain/loss count. "Benchmark revisions" are, to put it politely, adjustments to prior adjustments to the job statistic. These adjustments are used to correct for things like seasonality and sample biases. (See my previous post on seasonal adjustments, and other adjustments.)

It turns out that certain adjustments -- typically these are based on the assumption that past experience approximates the future -- are unreliable, especially while we are experiencing a deep recession the magnitude of which we haven't seen for a long time. Thus, when the statisticians look themselves in the mirror at the end of the year, they see white hair. They find that their prior adjustments don't make sense in light of new data, and thus have to go back to restate the previous numbers.

In some instances, the errors were extreme. In the early part of 2008, for instance, the job losses were understated by roughly 200,000 per month but looking at the top chart, we learn that the (adjusted) job losses reported at the time were below 100,000 so the error rate exceeded 100%. Ouch.

For me, this suggests that such adjustments should be suspended in extraordinary times. Adjusting data poorly is worse than not adjusting at all. Notice that seasonality is much more predictable than sample biases (such as the so-called birth-death model that tries to predict small business growth), and thus does not contribute much to the error. So don't suspend every kind of adjustments.

***

Floyd Norris correctly identifies a key problem with public consumption of economics statistics:

But it is the first number for each month that people remember. The revisions over the next two months will be barely mentioned in news stories. When the final numbers are posted, they will seem like ancient history.

Given that Norris understands this problem, I find it amusing that he does not conclude that the media needs to do a better job of reporting. Perhaps they should warn readers that the first number is highly inaccurate; perhaps they should highlight the revised numbers in each report, rather than the most current, highly inaccurate number.

Presumably he wrote this story because he thinks this is a serious problem. But alas, the tour of the sausage factory has left his appetite for sausages intact.

Comments

[Belstaff Outlets UK]

Thanks, I'm going to have nightmares tonight.