Supposedly one of the tactics in the fight against obesity is to change how we measure obesity (from BMI to DXA): that's the key message in an LA Times article (link).
This is a great read if only because it covers many common problems of measurement systems. In thinking about invented metrics, such as SAT scores, employee performance ratings and teacher ratings, bear in mind they only have names because we gave them names.
Measuring things always lead to perverse behavior. Here are some examples straight out of this article:
1. The metric, even if accurately measured, has no value
The goal of measuring obesity is to help reduce obesity-related illnesses but we are told that there is almost no evidence that the new metric (DXA) is linked to health risks. "For all its shortcomings, the link between the BMI and Type 2 diabetes, cardiovascular disease, certain cancers and other ills has been established by decades of research, Bergman said. But the precise level at which body fatness, as measured by DXA, contributes to such illnesses is not yet established." In other words, we have no evidence that DXA is related to illness.
2. Blame the failure of a program on the metric
According to the co-author of the study, the BMI metric is at least partially responsible for the failure of existing anti-obesity policies. The implication is that switching metrics would make these same policies more effective. One can dream. The fallacy of this argument is easily observed: focus on the segment of people who are deemed obese by both old and new metrics; do current policies succeed in reducing obesity among this segment?
This argument could work if we hypothesize that the BMI metric is so wrong that many people who should receive treatment are currently excluded while the people currently getting treatment are those who are unlikely to benefit. I just find that implausible.
3. A metric becomes more complicated over time
Since these are invented metrics, there is always a tendency to tinker with the formula. Tinkering is almost always an exercise in adding complexity. The more complicated is the metric, the harder it is for people to understand how to affect the metric, the less likely it is to improve. It is a simple matter to get on a scale to estimate one's BMI but it is not possible for individuals to measure DXA.
4. If in doubt, find a more expensive way to measure the same thing
Not surprisingly, measuring DXA is much more expensive than measuring BMI. Spending money gives us a sense of accomplishment. More expensive wine is deemed better.
5. Effort is spent negotiating how to measure the outcome, rather than how to change the outcome
This view is classic: "There's a casting about for better ways to address the public health problem of obesity. And to do so, you have to measure the problems and their public health consequences." It is hard to see how changing the obesity measure solves a public health problem.
6. If we don't like the outcomes, change how we measure the outcomes
"In the last two years, researchers have proposed a wide range of alternatives to the BMI; increasingly, they are using them to measure the effectiveness of interventions such as weight-loss counseling, exercise regimens and drug therapies." I don't know how to interpret these sentences. They imply that such interventions failed in the past but only because these interventions failed to affect BMI. Given better measures, these interventions would magically become effective. But look at point 1 above, which said that BMI has been linked to illnesses while other alternatives like DXA haven't.
7. More diagnosis leads to better outcomes
The new metric would classify more people as obese. Typically, the marginal person being considered obese is less obese than the average obese person. Thus, the marginal person has lower health risk than the average person. So, without doing anything, the outcome would improve just because the average risk in the population is reduced.
Unfortunately, the perversion of measurement is happening everywhere, not just in medicine.