Junk Charts

Water and wine

Marketers have always argued that price signals quality; this leads to the startling idea that one should just set a high price. 

If you don't believe it, note how Coca Cola and Pepsi turned tap water into a premium-priced $1.7 billion market.  As we now know, Dasani and Aquafina are just bottled tap water.

Even if one can turn water to wine, now researchers discovered the same rule applies.  Unlike most scholarly articles, they actually published a well-made chart to illustrate their experiment.

Testers were given the same wine but told either it cost $10 or $90.  Their brain activity is measured.  The chart showed that those thinking it cost $90 (green line) had much better sensation about the wine than those thinking it cost $10 (blue).

A standard way to display this information is a data table that spells out every estimate and its standard error, plus some asterisk or bolding scheme to indicate statistical significance.  Visualization is far superior.

For more examples, see Gelman's paper or Kastellec and Leoni's paper.

Reference: "Study: $90 wine tastes better than the same wine at $10", News.com, Jan 14, 2008.

The eyeball test

This set of graphs was used by the NYT to discuss changes in U.S.  spending patterns over time.  For this post, I am focusing on the bottom left and bottom right graphs.  One shows spending on energy as a percent of GDP; the other, on "nonresidential structures" (aka, commercial buildings).

At first glance, spending on energy and that on commercial buildings look very similar in shape (see above or below left).  Alas, this "eyeball test" doesn't work very well with time series data.  Lets investigate further.

"Standardizing" the data (above right) tells us whether the swings are unusual or not in the history of the data.  So in the 1980s, commerical building spend spiked to more than three times the standard deviation above the historical average.  Generally speaking, the standardized unit of 3 is taken to mean highly unusual. 

Notice that the peaks of the left graph had equal heights but on the right graph, energy spending peaked only above two while commerical building spend rose above three.  This is because energy spending has been more volatile historically so it takes larger jumps (or plunges) to count as "unusual" movements.  This information is hidden in the unstandardized version.

Further, since we are concerned with long-term trends, lets take a look at five-year moving averages (below right): in other words, each time point is the average of the preceding five years worth of data. 

The fluctuations have been smoothed out and the peaks are no longer as high.  Glancing at this chart, we may still conclude that the spending patterns are quite similar -- especially in the period prior to 1995.

But is that really the case?  Zooming in on the 1980s, we may mistakenly think the two lines are "close together" if our eyes read the horizontal distance and/or area between the curves, rather than focusing on the vertical distance.  The arrows on the bottom left chart depict this difference.  To make things clearer, the bottom right chart plots the vertical distances between the two lines.

Observe that the difference expanded to above 1 unit in the late 1980s.  A difference of one unit is very large in the standardized scale (of "unusualness") since 0 is business as usual and 3 is "highly unusual".

Eyeballing the two time series would lead us to believe that the two series are similar but we run the risk of underestimating the differences as illustrated here.

Source: "Auto Sector's role Dwindles, and Spending Suffers", New York Times, Nov 3 2007.

Points of comparison

In light of the current housing crisis, arising from mortgage defaults, I pulled this graphic from a Jan 2007 opinion piece that plotted historical default rates of mortgages.  Notice the high degree of stretching on the vertical axis that exaggerates the volatility: essentially, the annual delinquency rate ranged from 1.75% to 2.65% during the last six years or so.  One might be forgiven to think that a 2% default rate is quite acceptable.

Compare the above chart to the pair that showed up in the NYT in Oct 2007 (see right).  The default rates here are in the 10-20% range, very alarming indeed.

The two graphics illustrate a key issue of "aggregation" in statistical analysis.  The first graphic is super-aggregated: all types of mortgages of all ages are put together to calculate each year's default rate.  The second graphic hones in on subprime mortgages only.

More importantly, the second graphic presents data in "vintages".  Each line represents loans originated during a particular year (a "vintage").  This establishes comparability.  On the first chart, each point in time represents the default rate of mortgages averaged over all ages (some loans may be only a few months old; others may be 15 years old).  Since the default rate is much higher for very young mortgages than for older mortgages, such averaging hides crucial information.

Overall, the NYT graphic very effectively conveys the alarming trend of new mortgages performing much worse, especially those originated in 2007.

It can benefit from two slight edits: adding a few more years, and using vertical lines (the most critical comparisons are default rates for loans of a given age!)  Something like this...

Sources: "As Defaults Rise, Washington Worries", New York Times, Oct 16 2007; "Mounting Mortgage Credit Problems", economy.com, Jan 23 2007.

Sense of proportion

[I'm back from vacation.  Will provide my reaction to the responses to the Gelman challenge, and for those who have sent me email, I will work through them soon.]

The NYT commented on a trend among marketers to shift their advertising spending from so-called "measured" media like print and TV to so-called "unmeasured" media like product placements, contests, etc.  The following chart accompanied the article:

This construct is akin to a population pyramid; it's great for comparing two groups along one metric, say age groups between males and females.  Here, the two halves aren't comparable groups but two different metrics.  The main metric, that is, the proportion of unmeasured, is not directly depicted: the reader must figure out mentally how much of each bar the black part covers.  Also, the companies are sorted by unmeasured media spending but this leaves the measured spending with a jagged profile, confusing matters.

As for the little white slits on the gray bars, they are admittedly cute but it is difficult to compare the detailed breakdown between print, TV and other media among companies.

The following dot plot gives the two halves equal weight.  (Pink dots are measured, blue unmeasured.) It's not a very interesting graphic though. The sense of proportion is still missing.

I settled on a scatter plot which relates the proportion spent on unmeasured to the total amount of spending.  It appears that the largest advertisers had the lowest proportional unmeasured spend while the smallest (among the majors) had the highest.  (It's only a weak correlation: a linear fit yields only 16% R-squared.)

Source: "The New Advertising Outlet: Your Life", New York Times, Oct 14, 2007.

Structuring a chart

This chart from the NYT was intended to show how the EPA has moved the bar on vehicle mileage ratings: 2008 estimates were lower than 2007 estimates across the board, regardless of manufacturer, model and city/highway.

The chart was built from one basic component, repeated for each model.  I like the discreet gridlines (the white ticks) which enable readers to count off the mileage ratings.

The data is rich: ratings were given along three dimensions (model, year of estimate and city/highway).  Readers can benefit from a stronger guidance in where to look for the most pertinent information.  As the chart stands, it is merely a container for the data.  It fails our self-sufficiency test: all the data were printed on the chart, and the bars add little.

In the junkart version, I use knowledge of the data to structure the chart. First, noting that sedans, hybrids and trucks/SUVs/minvans have different levels of mileage ratings, I clustered the models into three groups.  Secondly, the city and highway ratings were separated into two columns as I consider the between-model comparisons more important than city-highway comparisons.  The chart is a dot plot, with a vertical tick for 2007 estimates and a dot for 2008 estimates.  It's easy to see that all dots sit to the left of vertical ticks.

More subtly, we can also see that the hybrids appeared to have been penalized more.  Or perhaps, the higher the rating, the larger the downward adjustment...

Source: "Mileage Ratings Are Still Estimates, Though Closer to Reality", New York Times, Sept 16 2007.

Gauging the water level

This set of charts covered the back page of one of New York Times' sections this weekend.

Regular readers will share my enthusiasm for the top chart.  It makes a clear, cogent case to support the article's thesis concerning the rise of bottled water.  Various renditions of this type of chart have appeared here, for example.

Specifically, the smart use of color to cluster the line objects helps interpret the trends.  Blue sets out the two primary interests.  (It's a mystery to me why the gray lines were separated into darker and lighter hues.)

The twenty-year horizon used is another nice touch. I'd remove the gridlines although they aren't too distracting here.

Sadly, the second graphic does not meet the high standard of the first.  The biggest problem concerns the red rectangle, purportedly showing how much of the bottled water was imported.  The choice of differently-sized bottles as objects makes it impossible to gauge what proportion of the total was imported.  If the rectangle was placed over 1-litre bottles instead, it would look smaller.

Source: "A Battle Between the Bottle and the Faucet", New York Times, July 15, 2007.

Baby names and success

While we speak of baby names, David F. nominates this set of 6 charts from WSJ.  Compare this with Wattenberg's names voyager, and the benefit of interactive graphics is immediately evident.

In David's words:

They show graphs of six different names, but the two on the bottom use a dramatically different scale (from 1st to ~20th, instead of from 1st to 1000th). The introductory text notes the difference, but it is still a shock.

We like the use of "small multiples" but their impact is compromised if we don't keep the background material constant so that readers can compare between charts.  By having  different scales, the message was distorted: Mary has had a much larger drop than David, and it's easily missed in these charts.

Lines should take the place of areas which carry scant meaning in this context.

The use of blue and red is a nice touch but dovetailing the male and female charts strikes us as excessive fun.  It would have been clearer to give the sons and the daughters their own columns.

The article itself relates the anguish of modern parents in naming their babies.  Much of this angst can be traced to serious econometric studies that claim to have found cause-and-effect relationships between someone's name and their eventual success in life.  Some of this research was highlighted in Freakonomics, for example.  My stance is that all such studies are dubious, there being innumerable confounding factors (socio-economic, genetic, cultural, luck, etc. etc.).  In addition, the measured response can range from "happiness" to income to many other metrics.  The danger of finding something because one looks hard enough is very real.  We don't currently have tools powerful enough to substantiate this sort of studies.

Source: "The Baby-Name Business", Wall Street Journal, June 22, 2007

Losing the tune

Duncan C. nominates this Wall Street Journal chart.  A sure sign of trouble is when the accompanying article waxes about a new on-line music service, another one that practises "loss-based" pricing, i.e. priced to ensure a loss; the article does not mention anything about this chart.  It just stood on the side, like wallpaper.

Its key message does not seem to connect with the data.  "Growth in digital music downloads has not been rapid enough to offset declining CD sales": but in terms of total units, the chart shows a small dropoff in CD sales coupled with an explosion in digital songs sold.  Besides, the units of discs, songs and iPods are not directly comparable.

Source: "Listen to Music Free But Pay to Carry", Wall Street Journal, June 5, 2007.

Airline bumps and bump charts

The Harvard Social Science Statistics blog pointed to an NYT article about revenue optimization in the airline industry.  Huge props to the Times for explaining the science (and art and politics) of one of the most successful applications of operations research.

In short, valuable business travellers want refundable tickets.  Because of this and other reasons, about 10% of booked tickets become no shows.  Airlines recoup the loss by over-booking.  Implicitly, they trade off the potential for dissatifying a few unlucky passengers (who would be bumped from their flights) and the potential for flying with 10% empty seats (in addition to unsold seats).  Optimization algorithms (constantly tuned by entry-level staff) try to strike a balance.

Recently, because the average percentage of seats sold has been going up, the room for such maneuvreing has been squeezed, leading to higher bump rates, and more travellers being stranded.  There is some variation across airlines due to the level of sophistication of their revenue optimization algorithms, corporate strategy, etc.

The following charts present data by airline of the bump rates in 2005 and 2006.  One would be interested in answering questions such as:

• Which airlines have the best (or worst) bump rate?
• Are some airlines consistently better (or worse) at controlling the bump rate?
• Which airlines have improved (or worsened) from year to year?
• Are the differences of practical significance?

The original chart shown on the left does not reveal the answers readily.  My favourite bumps chart offers them up clearly (well, except on the question of significance).

The biggest problem, though, is the header: number of passengers per 10,000 bumped.  The data plotted appeared to be the reverse: the number of bumps per 10,000 passengers.  Otherwise, there would have been more bumped passengers than passengers!

Source: "Bumped Fliers and No Plan B", New York Times, May 30, 2007.

Visualizing web statistics

Tim inquired about:

how to create an elegant graph for Web visitor traffic statistics that shows both how many views a page gets and then how many people click that page to go further ("conversion rate"). Part of the problem is that conversion rates vary from, say, .3% to 50% (a wide range).

Lets work with this sample data set.  I ordered it from highest to lowest click rate, which is the primary metric of interest.  The number of page views is of interest too as sometimes rarely-visited pages may have high click rates.

At this point, it's important to know the context.  Specifically, who controls the allocation of pages? Did the data come from a randomized experiment? Or did they get a self-selected sample (e.g. web surfers deciding which section of the site to visit)?

The first construct I tried is the "lift curve" often used in marketing.  It's the same thing as the Lorenz curve used by demographers but interpreted differently.  Here, we see that Guitar pages accounted for 26% of the page views but 37% of the clicks; House pages accounted for an incremental 44% of the pages and 59% of the clicks; etc.  The relative click rates are immediately clear from the steepness of the line segments.  The lift curve is appropriate for the self-selected case, in which we can take the allocation of page views as fixed.

If the allocation of page views is a decision to be made, then it doesn't make much sense to accumulate page views.  The second construct is the "scatter plot" of % clicks versus % page views.  The steepness of the line through the origin helps us compare the click rates.  Bicycles is clearly inferior in generating clicks.

Both these constructs are highly efficient; adding new data does not expand the chart at all.

Keen readers will observe that the slope of the line is not the click rate but rather a click rate index (relative to the overall click rate).  This means that any data point above the diagonal has above-average click rate.