Detached in time and space
Good and simply better

Loss aversion and faux accuracy

Econ_geoengReader Bernie M. is not a fan of this Economist chart.

The chart was prepared by Aurora Flight Sciences, an aircraft manufacturer, commissioned by a professor who supports the concept of maintaining a fleet to pump sulphuric acid into the stratosphere as a way to induce artificial cooling to counteract human-induced global warming.

The chart appears to compare many different ways of shooting the acid into the skies along two dimensions: cost and altitude.

Bernie wrote:

I find the choice of axes extremely counterintuitive. Altitude one would expect on the y-axis. And mixing up the scatter chart elements with the connected line chart doesn' really help either.

The convention regarding axes is to put the outcome variable on the vertical, and the explanatory variable on the horizontal. Thus, in this case, if the cost of a particular solution is primarily determined by the "altitude" (presumably of where the acid would be released), then the designer has followed convention. It is unfortunate that "altitude" is more intuitively put on the vertical axis, but I suspect that defying convention might cause more confusion.

On the other hand, if altitude and cost are not related to each other but two different metrics to evaluate geoengineering concepts, then Bernie's point is right on - swap the axes!

***

The use of connected lines for two of the solutions but not the rest is a symptom of what I have called "loss aversion". The horror of leaving some of the data on the cutting floor.

The only mention of altitude in the article refers to Aurora's assertion that it is sufficient to use newly designed aircraft flying at 20-25 kilometers. If that is Aurora's preferred solution, there is little reason to show all the other altitude configurations that are suboptimal.

Perhaps the designer wants to make the point that the Boeing 747 solution is inferior to the Aurora solution because Aurora could design aircraft to fly at 10-15 km at a lower cost?  If so, then the chart is very misleading in not providing a comparable cost for Boeing's solution if required to fly at 20-25 km.

When comparing different entities, it is always a bad idea to treat the entities differently. Comparison is only possible on equal footing.

In fact, I think the chart would be a lot clearer if they dropped the altitude dimension on the floor. For each solution, plot the yearly cost at the optimal altitude selected by the respective engineers. Use a bar chart. With a single dimension, it is much easier to accommodate the very long data labels.

(Now, I'd defer to the geoengineers as to whether the altitude dimension is dispensable. I don't have any expertise in this science. Judging from the Aurora red line, I'm assuming that there can be feasible solutions at all altitudes, which leads me to conclude that altitude isn't all that.)

***

So what is the biggest problem with this chart? It is the faux accuracy.

Given the tremendous amount of uncertainty surrounding these projected costs, one would expect very big error bars around the cost estimates. Using single dots with no error bars is hard to stomach.

 

 

Comments

Aminpractice.blogspot.com

I couldn't agree more about the Faux accuracy!

Two other thoughts. The business about putting altitude or depth on the y-axis is common practice in many physical sciences fields like oceanography (see example at
http://www.windows2universe.org/earth/Water/images/temperature_depth_jpg_image.html), limnology and atmospheric science. I always find these charts hard to think about because they go against the standard practice (I'm an ecologist), but I suppose if you spend your days looking at them they start to make sense. The line joining a series of solutions reminds me of an "efficiency frontier" in economics, where you are trying to figure out how many different possible solutions compare. Good solutions are on the boundary of the cost-output space. This might be relevant if the report was arguing for an investment of say, 10^8 dollars in the Aurora design. Then the performance of Boeing's best alternative isn't relevant - only the performance of off the shelf solutions.

Laust Lund Kristensen

As you accurately pointed out, only the data in the 20 - 30 km range are viable alternatives to the hybrid aircraft. Since the closest datapoint in that range is a factor of 3 - 4 higher than the hybrid aircraft (we are looking at a log scale), surely error margins are not that big. So is faux accuracy really an issue here?

Kaiser

Laust: If the error bars are not plotted, then we don't know how big they are. Speculating on large-scale projects with new technology not yet implemented is a recipe for very wide prediction intervals. If they aren't large, they can't be believed. For example, Boston's Big Dig cost $15 billion with the original projection of $4 billion.
And remember, if Boeing gets its hands on this chart, I'm sure they won't feel fairly represented by that one point.

Aminpractice: I don't quite get how the line could be an "efficiency frontier" as I imagine that altitude is a driver of cost; there is no output measurement at all on this chart.

Laust Lund Kristensen

Kaiser: Thank you for taking the time to respond.

I agree with you that in principle all data should include some indication of error. But in some cases singificant digits is quite adequate as a measure of uncertainty. Here the datapoint for Hybrid Aircraft is at around 100 MUSD / year and the datapoint thethered pipe is at around 400 MUSD / year (the two closest datapoints in the range in question), so by the priciple of significant digits there is no overlap.

If you truly believe that the cost estimates can fluctuate by more than 100% then instead of demanding error bars, you should be demanding a clear overview of the model and its assumptions!

This comes back to your point about loss aversion as well. Since significant digits in this case likely understates the datas margin for error and this does not disturb the analysis of those data, then why should we demand error bars?

The comments to this entry are closed.