Head-shaking at the deep hole

Continuing to work through the pile of submissions, here is Jeannie C. recommending one of my favorite economics charts. The economics blogs are generating lots of charts, many of which uninspiring and run-of-the-mill but this one about the jobs picture, relative to past recessions, truly paints a harrowing story. (Looks like TPM took the chart from Business Insider but this chart has appeared everywhere).


The little dotted extension at the end of the current curve (red) indicates the jobs picture after removing Census jobs. I have already explained why this adjustment is necessary (here, and here).

Two small improvements I'd make to the chart:

  • Instead of a rainbow of colors, should use a foreground-background concept. Have all the past recessions in gray, and the current one in red. This change necessitates a change in curve labeling strategy: should affix the year labels directly on the 0% line above the curves. Doing so eliminates the head shakes needed to find the year of the curve.
  • Smoothing out some of the curves will help remove clutter without harming the central message of the chart.

The scatter-plot matrix: a great tool

The scatter-plot matrix is one of the lesser known graphical tools beloved by statisticians. A scatter plot displays the correlation between a pair of variables. Given a set of n variables, there are n-choose-2 pairs of variables, and thus the same numbers of scatter plots. These scatter plots can be organized into a matrix, making it easy to look at all pairwise correlations in one place.


Since Nate Silver's feature article about New York neighborhoods came out, I have been working on capturing the data because so much was left unsaid in that article.  His ranking formula takes 12 factors (housing affordability, transit, green space, nightlife, etc.) and combines individual scores into an overall score based on chosen weights (e.g. housing affordability counted for 25%). Scores are then converted to ranks.

Silver's discussion focuses on explaining which factors caused which neighborhoods to be ranked high (or low). I'm interested in whether the individual factors are correlated. For example, do neighborhoods with more expensive housing also tend to have higher-quality housing? what about better schools? are more diverse neighborhoods also more creative? and so on. There is really a treasure trove of information locked up in this data.


A scatter-plot matrix neatly organizes all of the pairwise correlation information.  See below.


Each small chart shows the correlation between the given pair of variables (one listed on the right, the other listed below). The dots represent the neighborhoods. The pink patch contains the "middle 75%" of the nieghborhoods, and we can use the orientation of these patches to get a sense of whether the two variables are positively, negatively or not correlated.

There are lots to see in this chart. I just picked a random few things for illustration:

  • In the top left corner, the slant shows that the more affordable the homes are, the worse is the transit.
  • The better the shopping, the better the dining.
  • Interestingly, more diversity seems to mean lower creative capital (also the correlation is only moderate).
  • Wellness scores fall within a rather narrow range compared to other categories, and they seem to be almost completely unrelated to any of the other factors.


(Note: I used JMP to generate this matrix. Excel unfortunately does not make scatter-plot matrices natively. JMP is great for such exploration... if the developers are reading this, please make it easier to man-handle the category labels! I made a mess of rotating the text on the right.)

P.S. I had an adventure processing the data from New York magazine. There appears to have been quite a few typos. For more, see my writeup on the book blog.

Weekend cross-posting

On the book blog this past week, I discussed Steven Strogatz's column "Chances Are" in the New York Times. Professor Strogatz has been hired as a columnist to write about mathematical topics, and his most recent column concerns Bayes' Theorem, and its use in analyzing screening tests. In addition, a reader was curious about the (over-)reaction of  officials in Boston who advised residents to stop drinking tap water for fear of possible contamination which has not been verified.