Junk Charts

Statistical literacy

I finally got around to reading "When Genius Failed", Roger Lowenstein's account of the spectacular collapse of LTCM, the hedge fund fronted by Scholes and Merton, Nobel laureates both.

It is a sobering read for anyone in the business of statistical prediction and modeling for sure.

What also caught my eye, and caused dismay, is how Lowenstein got basic statistical principles wrong in the book.   He used the bully pulpit to sound the usual alarm against the normality assumption and for fat tails.  He began by confusing LLN and CLT (central limit theorem):

Statisticians have long been aware of the "law of large numbers".  Roughly speaking, if you have enough samples of a random event, they will tend to distribute in the familiar bell curve ...

In the same breadth, he then equated two different probability distributions:

This is called the normal distribution, or in mathematical terms, the lognormal distribution.

Doesn't this say something about the state of statistical literacy?

PS. Here is a link to Dunbar's "Inventing Money" (thanks Marc).  It apparently came out before Lowenstein but didn't get as much press. 

Review: Curve Ball 3

Just want to highlight one more graphic from Curve Ball, one which I consider the most innovative, highly effective and powerful.  Without much ado:
This is one of those charts that paints a vivid story.  Any fan can mentally re-trace the baseball game by reading this chart, without having seen the game itself.  The horizontal axis traces the 9 innings of a baseball game while the vertical axis plots the probability of Toronto winning the game.  This probability is updated over the course of the game as we read from left to right.  (For those asking, this plots Game 6 of the 1993 World Series.)

To quote the authors:

" We see that Toronto's probability of winning rose from the start as they prevented the Phillies from scoring in the first inning.  This trend continued as the [Toronto] Blue Jays scored three times in the first inning... The low point in the [fifth] inning for Toronto occurred just after [Phillie] John Kruk walked to load the bases.  [Phillie Dave] Hollin's big out is shown by the rise ... in Toronto's victory probability from this low point ... in the seventh the Phillies turned the tables ... scoring five runs to take the lead.  The plot of Toronto's probability of winning looks like the Dow Jones Industrial Average in free-fall.  Toronto did not score in its half of the seventh, pushing its probability of winning even further down. ... the plot rises (and the plot thickens) in the eighth inning as a result of a threat with bases loaded and two outs.  In the ninth inning, the Phillies went down quickly.  Toronto came out storming, ... quickly putting runners on base.  The triumphant ... impact of [Toronto's Joe] Carter's home run is evident in the steep rise in the final markings of the plot."

This chart belongs to the same class as the Bumps chart.  In a previous post, I traced how one can re-imagine the Bumps race just by tracing the plot from left to right.

Reference: "Curve Ball", Albert & Bennett.

Narratives that create questions

Narrative charts, like this one shown on the right, are particularly difficult to master.  The temptation is strong on the part of the designer to mislead by inclusion/exclusion and on the part of the reader to misjudge by reading between the lines.

The accompanying article made the point that James Frey's fraudulent memoir experienced a severe drop in sales since Oprah "sacrificed" him.  Unfortunately, what the chart shows is how choppy sales have been for this book.  It experienced a similar drop after his first appearance on Oprah.

In fact, this chart raised a host of unanswered questions:

• How to explain the end-of-year rise in sales back to post-first-Oprah-appearance level?
• What is the purpose of the red line?  Is it fair to compare a hardcover with a paperback?
• How to explain the "turbulence" of Frey's memoir sales even before the scandal broke?  Why is it that his other book has a much smoother sales trend?  (Is it merely a scale effect hiding its ups and downs?)
• Oprah's influence appeared to be highly specific to the recommended book and time of the show.  There seemed to be zero impact on the sales of other books by the same author (scale effect?) and a rapidly diminishing impact even for the highlighted book.  Is this a general phenomenon?

This type of time-series chart does not provide direct evidence of cause and effect; however, they are commonly used in the media for that dubious purpose.  At the best, we can conclude that those factors are correlated, prompting hypotheses and further analyses.

Reference: "James Frey's Falsehoods Improved His Tale", New York Times, Feb 1 2006.

Review: Curve Ball 2

Continuing the book review.  The reader who sent me the book noted that the authors used a similar technique to the one I used to study whether suicide spots on the Golden Gate Bridge were random (see here, here, here, here and here).  They used it to study whether Todd Zeile was really a "streaky" hitter or not.

In the following set of charts, Zeile's batting average in the first half of a season was compared against eight fictional hitters.  These fictional hitters were simulated to have two hitting "states" (hot, cold).  At each fictitious game, the hitters were assigned one of the two states with some probability.  The authors asked whether Zeile's batting pattern was similar to those of streaky hitters.  (Conventional sportscaster wisdom says he is streaky.)

Here, we want to know whether the graphs are similar; in my graphs of suicide locations, I asked whether the actual data is different from random.  In general, I find it easier to see differences than similarities.  In both cases, it is not sufficient to visually inspect these charts.  We must use some tests (possibly statistical tests) to help confirm our intuition.  The authors picked these metrics: Max - Min, Number of long streaks (8 or longer), Number of runs, Number of 0-hit games and Number of 3+ hit games.  (A "run" is a string of consecutive 0-hit games or consecutive games with at least 1 hit.)  These are all measures of dispersion or extreme values.

My first review can be found here.

Review: Curve Ball

A kind reader sent me a Christmas gift, which accompanied me on my vacation.  The book is Curve Ball by Jim Albert and Jay Bennett, and I'm completely fascinated by it.  It presents a statistical perspective on baseball data, a soothing antidote to the nonsense spouted by the typical sportscaster.  Even more impressively, the book is liberally sprinkled with charts, and these charts are generally of a very high standard.

Their first feat was to debunk the myth of the batting average BA (hits divided by at-bats).  They accomplish this using this innovative chart. 
Each vertical bar is a range of estimate of the batter's BA after he has a given number of at-bats.  The bars get shorter as the number of at-bats increases because over the course of the season, we can be more and more certain of the batter's true hitting ability.

Notice that the bar is very tall in the first 100 at-bats, roughly ranging from 0.35 to 0.50.  This illustrates why statisticians love data quantity: without sufficient samples, any estimation is highly unreliable.

Also notice that the rate of shortening is very slow after say 250 at-bats and after 700 at-bats (roughly a full season), the bar is still about 0.06 tall, roughly between 0.385 and 0.459.  This shows why BA is not as definitive as usually thought.  Looking up 2005 batting statistics, one finds that Derek Lee, the top hitter, hit 0.335.  This means his true batting average is roughly between 0.305 and 0.365.  There were 20 other hitters who hit at least 0.305.

Further, because the 2005 league BA was 0.264, any player with BA between 0.234 and 0.294 may be a league-average hitter.  Looking up the statistics, one finds that this range includes hitters ranked 37 through 150 (which is the end of the list).

More to come...

Reference: Albert and Bennett, Curve Ball, pp. 67-8

Concordance, or tag clouds

I noticed that Amazon has adopted the tag cloud metaphor in its newest feature known as "concordance".  Clicking on concordance gives you a list of the top 100 most frequently occurring words in the book; mousing over each word provides the exact number of mentions in the book; clicking on the word brings up pages on which the word is mentioned.

They are using the simple and elegant presentation that I praised here, the same as Flickr.  Beautiful as it is, it took me a little while to come up with a use case for this feature.  But I did!

Imagine someone wanting to buy a book on probability for self-study.  It is a cardinal rule of book publishing that every text book must be labelled "introduction" or "elementary", regardless of content.  But Amazon's concordance is here to help.  Here are four books of increasing difficulty (Aczel's Chance, Ross' Probability Models, Resnick's Probability Path and Dudley's Real Analysis and Probability):

Looking at the tag clouds, one can roughly judge the level of sophistication of these books.  Below I present them in mixed order.

[1] appears to be an elementary book that emphasizes the key concepts ("probability", "random", "distribution", "independent") while "customers" is the most interesting word indicating it is perhaps an applied book.  [2] is even more novice as we don't find words like "suppose", "system" and "function" that showed up in [1].  Words like "martingale", "sequence", "convergence" give away [3] as reaching another level of sophistication.  I should click on "oc" and "oo" to find out what these mean.  [4] is the only book on probability where "probability" is not in the top 10; it is evidently entirely theoretical with oodles of measure theory.  (So, [1] Ross [2] Aczel [3] Resnick [4] Dudley.)

How else have you used this concordance feature?  Let us know!

How representative is your sample?

Taking a hint from Mahalanobis, I dug into Howard Wainer's other book  (Visual Revelations) to find the following gem.  Imagine you're an engineer working for the military.  You have the ingenious idea to inspect planes that returned home and plot the pattern of bullet holes.  The dark regions had high density of bullet holes.  Your task is to recommend where to put extra armour on the new planes.  What would you recommend?  (Note: the answer appears after the graphic!)
 

 

 
 
 
Howard credited Abraham Wald for his counter-intuitive insight.  We should put extra armour in the white regions, not the dark regions.  The inference is that the planes that got shot in the dark regions managed to return to the base while others got hit presumably in the white regions and never returned.

What has this to do with sampling?  If we forgot about the planes that never came back, we may jump to the conclusion that we should reinforce the dark regions.  The sample we didn't see is as important as the sample we observed.  To wit:

Statisticians call this "survivorship bias".  We only oberve survivors but we must not forget about the non-survivors!

A related page I found on the Web: Steve Simon