Junk Charts

Getting inside my head

[This is a cross-post from the sister blog, Numbers Rule Your World]

Some interviews with me or snippets of such have surfaced recently. Here is a list:

Kate Meersschaert interviewed me for New Learning Times (link; registration required). I talked about my teaching philosophy, and why I write books.

Jay Ulfelder, a political scientist who keeps an interesting blog, recommends Numbers Rule Your World, and a few other books for political scientists (link).

If you haven't heard already, 2013 is the International Year of Statistics. I was one of the talking heads here.

Here, I talk about the history of Junk Charts, and the new paradigm of interactively building graphs, as opposed to the template paradigm popularized by Excel.

 

Rank confusion

This chart, found in Princeton Alumni Weekly, only partially scanned here, supposedly gave reasons for "Princeton's top-rated [Ph.D.] programs" "to celebrate". My alma mater has outstanding academic departments, but it would be difficult to know from this chart!

Due to the color scheme, the numbers that jump out at you are the ones in the bright orange background, which refers to how many other departments are ranked equal to Princeton's in those subjects. It takes some effort to realize that the more zeroes there are in the top buckets (fading orange), the better.

The editor started with a nice idea, which is to convert raw rankings into clusters of rankings. She recognized that in this type of rankings (see a related post on my book blog here), it is meaningless to gloat about #1 versus #2 because they are probably statistically the same. For instance, in the ranking of Architecture departments (ARC), 37 schools (including Princeton) all belonged to the same cluster as Princeton, judged to be a statistical tie.

One of the main reasons why this chart looks so confusing is its failing the self-sufficiency test. It really is a disguised data table, with some colorful background and shadows; the graphical elements add nothing to the data at all. If one covered up all the data, there is nothing left to see!

In the following rework, I emphasize the cluster structure. Each subject has three possible clusters, schools ranked above, equal to, and below Princeton. Instead of plotting raw numbers, the chart shows proportions of schools in each category. The order is roughly such that the departments with the relatively higher standing float to the top. Because a bar chart is used, the department names could be spelt out in their entirety and placed horizontally.

If one has access to the raw data, it would be even better to reveal the entire cluster structure. It is quite possible that the clusters above and below Princeton can be further subdivided into more clusters. This will allow readers to understand better what the cluster ranks mean.

 

On the other blog

I posted summaries of my recent talks here.

Head-shaking at the deep hole

Continuing to work through the pile of submissions, here is Jeannie C. recommending one of my favorite economics charts. The economics blogs are generating lots of charts, many of which uninspiring and run-of-the-mill but this one about the jobs picture, relative to past recessions, truly paints a harrowing story. (Looks like TPM took the chart from Business Insider but this chart has appeared everywhere).

The little dotted extension at the end of the current curve (red) indicates the jobs picture after removing Census jobs. I have already explained why this adjustment is necessary (here, and here).

Two small improvements I'd make to the chart:

• Instead of a rainbow of colors, should use a foreground-background concept. Have all the past recessions in gray, and the current one in red. This change necessitates a change in curve labeling strategy: should affix the year labels directly on the 0% line above the curves. Doing so eliminates the head shakes needed to find the year of the curve.
• Smoothing out some of the curves will help remove clutter without harming the central message of the chart.

The scatter-plot matrix: a great tool

The scatter-plot matrix is one of the lesser known graphical tools beloved by statisticians. A scatter plot displays the correlation between a pair of variables. Given a set of n variables, there are n-choose-2 pairs of variables, and thus the same numbers of scatter plots. These scatter plots can be organized into a matrix, making it easy to look at all pairwise correlations in one place.

***

Since Nate Silver's feature article about New York neighborhoods came out, I have been working on capturing the data because so much was left unsaid in that article.  His ranking formula takes 12 factors (housing affordability, transit, green space, nightlife, etc.) and combines individual scores into an overall score based on chosen weights (e.g. housing affordability counted for 25%). Scores are then converted to ranks.

Silver's discussion focuses on explaining which factors caused which neighborhoods to be ranked high (or low). I'm interested in whether the individual factors are correlated. For example, do neighborhoods with more expensive housing also tend to have higher-quality housing? what about better schools? are more diverse neighborhoods also more creative? and so on. There is really a treasure trove of information locked up in this data.

***

A scatter-plot matrix neatly organizes all of the pairwise correlation information.  See below.

Each small chart shows the correlation between the given pair of variables (one listed on the right, the other listed below). The dots represent the neighborhoods. The pink patch contains the "middle 75%" of the nieghborhoods, and we can use the orientation of these patches to get a sense of whether the two variables are positively, negatively or not correlated.

There are lots to see in this chart. I just picked a random few things for illustration:

• In the top left corner, the slant shows that the more affordable the homes are, the worse is the transit.
• The better the shopping, the better the dining.
• Interestingly, more diversity seems to mean lower creative capital (also the correlation is only moderate).
• Wellness scores fall within a rather narrow range compared to other categories, and they seem to be almost completely unrelated to any of the other factors.

***

(Note: I used JMP to generate this matrix. Excel unfortunately does not make scatter-plot matrices natively. JMP is great for such exploration... if the developers are reading this, please make it easier to man-handle the category labels! I made a mess of rotating the text on the right.)

P.S. I had an adventure processing the data from New York magazine. There appears to have been quite a few typos. For more, see my writeup on the book blog.

Weekend cross-posting

On the book blog this past week, I discussed Steven Strogatz's column "Chances Are" in the New York Times. Professor Strogatz has been hired as a columnist to write about mathematical topics, and his most recent column concerns Bayes' Theorem, and its use in analyzing screening tests. In addition, a reader was curious about the (over-)reaction of  officials in Boston who advised residents to stop drinking tap water for fear of possible contamination which has not been verified.