The inevitable perversion of measurement

Supposedly one of the tactics in the fight against obesity is to change how we measure obesity (from BMI to DXA): that's the key message in an LA Times article (link).

This is a great read if only because it covers many common problems of measurement systems. In thinking about invented metrics, such as SAT scores, employee performance ratings and teacher ratings, bear in mind they only have names because we gave them names.

Measuring things always lead to perverse behavior. Here are some examples straight out of this article:

1. The metric, even if accurately measured, has no value

The goal of measuring obesity is to help reduce obesity-related illnesses but we are told that there is almost no evidence that the new metric (DXA) is linked to health risks. "For all its shortcomings, the link between the BMI and Type 2 diabetes, cardiovascular disease, certain cancers and other ills has been established by decades of research, Bergman said. But the precise level at which body fatness, as measured by DXA, contributes to such illnesses is not yet established." In other words, we have no evidence that DXA is related to illness.

2. Blame the failure of a program on the metric

According to the co-author of the study, the BMI metric is at least partially responsible for the failure of existing anti-obesity policies. The implication is that switching metrics would make these same policies more effective. One can dream. The fallacy of this argument is easily observed: focus on the segment of people who are deemed obese by both old and new metrics; do current policies succeed in reducing obesity among this segment?

This argument could work if we hypothesize that the BMI metric is so wrong that many people who should receive treatment are currently excluded while the people currently getting treatment are those who are unlikely to benefit. I just find that implausible.

3. A metric becomes more complicated over time

Since these are invented metrics, there is always a tendency to tinker with the formula. Tinkering is almost always an exercise in adding complexity. The more complicated is the metric, the harder it is for people to understand how to affect the metric, the less likely it is to improve. It is a simple matter to get on a scale to estimate one's BMI but it is not possible for individuals to measure DXA.

4. If in doubt, find a more expensive way to measure the same thing

Not surprisingly, measuring DXA is much more expensive than measuring BMI. Spending money gives us a sense of accomplishment. More expensive wine is deemed better.

5. Effort is spent negotiating how to measure the outcome, rather than how to change the outcome

This view is classic: "There's a casting about for better ways to address the public health problem of obesity. And to do so, you have to measure the problems and their public health consequences." It is hard to see how changing the obesity measure solves a public health problem.

6. If we don't like the outcomes, change how we measure the outcomes

"In the last two years, researchers have proposed a wide range of alternatives to the BMI; increasingly, they are using them to measure the effectiveness of interventions such as weight-loss counseling, exercise regimens and drug therapies." I don't know how to interpret these sentences. They imply that such interventions failed in the past but only because these interventions failed to affect BMI. Given better measures, these interventions would magically become effective. But look at point 1 above, which said that BMI has been linked to illnesses while other alternatives like DXA haven't.

7. More diagnosis leads to better outcomes

The new metric would classify more people as obese. Typically, the marginal person being considered obese is less obese than the average obese person. Thus, the marginal person has lower health risk than the average person. So, without doing anything, the outcome would improve just because the average risk in the population is reduced.

Unfortunately, the perversion of measurement is happening everywhere, not just in medicine.

Comments

[Joshua Jendza]

The problem with BMI is that it was never intended as a measure of Obesity, and definitely not intended for diagnosis of individuals.

An anecdotal story of how the scale can mislabel someone as obese includes my older brother. He was an athelete in High School, lettered in baseball, wrestling and soccer. He was in peak physical condition, but when he applied to the Naval Academy in Annapolis, they made him take a body fat test because his BMI said he was Obese. His test came back with a single digit body fat score if I remember correctly. All sorts of professional athelets are considered Obese based on BMI, and that decreases peoples faith in it as a metric, and as an extention their faith in recommendations made solely on how they rank on that scale.

Now, I know next to nothing about this proposed new metric, but it's newness and cost do not necessarily preclude it from being potentially more informative than BMI. That simply remains to be seen.

[Kaiser]

Joshua: thanks for your comment. Bear in mind that obesity is an invented idea for which there is no objective truth, and that any metric will have false positives and false negatives. Also see point 1, the proposed metric has not been linked to any health risks while the old metric has. I'm not advocating one or the other metric here; I'm just observing that the discussion points to the many social problems of defining metrics. You're certainly right about the problems with BMI.

[Cody L. Custis]

BMI is worthless for athletes, because muscle is more dense than fat.

The Navy uses a formula that takes the circumference of the neck to the circumference of the waist to estimate body fat, which is more accurate. The problem is that not everyone knows the circumference of their neck.

A simple estimate (especially for males) would be to compare waist length to pant length, which is well-known, easily chartable, and actually useful.

[Stuart Buck]

he proposed metric has not been linked to any health risks while the old metric has.

If BMI is linked to health risks only because it is a (rather poor) proxy for bodyfat -- and there's every reason to think that's the case -- then a better proxy for bodyfat should be more closely related to those same health risks.

[Tom West]

Joshua Jendza's comment indicates the Navy knows that BMI gives false postives for obesity, but use it because it gives few false negatives. So they run on the basis that low BMI = not obese, high BMI = obesity unknown.

[zbicyclist]

Kaiser, this is a great post.

Tom West's comment is a good one. Inevitably, in any simple measurement system there is a tradeoff between false positives and false negatives. The common objections about athletes' BMIs (including some above) miss the point that these false positives seem unlikely to lead to any negative outcomes -- it is vanishingly unlikely that anyone seeing Michael Jordan in his prime ever recommended he lose weight, even though his BMI was high (or, if they did, that he would do anything other than laugh).