The scatter-plot matrix is one of the lesser known graphical tools beloved by statisticians. A scatter plot displays the correlation between a pair of variables. Given a set of n variables, there are n-choose-2 pairs of variables, and thus the same numbers of scatter plots. These scatter plots can be organized into a matrix, making it easy to look at all pairwise correlations in one place.

***

Since Nate Silver's feature article about New York neighborhoods came out, I have been working on capturing the data because so much was left unsaid in that article. His ranking formula takes 12 factors (housing affordability, transit, green space, nightlife, etc.) and combines individual scores into an overall score based on chosen weights (e.g. housing affordability counted for 25%). Scores are then converted to ranks.

Silver's discussion focuses on explaining which factors caused which neighborhoods to be ranked high (or low). I'm interested in whether the individual factors are correlated. For example, do neighborhoods with more expensive housing also tend to have higher-quality housing? what about better schools? are more diverse neighborhoods also more creative? and so on. There is really a treasure trove of information locked up in this data.

***

A scatter-plot matrix neatly organizes all of the pairwise correlation information. See below.

Each small chart shows the correlation between the given pair of variables (one listed on the right, the other listed below). The dots represent the neighborhoods. The pink patch contains the "middle 75%" of the nieghborhoods, and we can use the orientation of these patches to get a sense of whether the two variables are positively, negatively or not correlated.

There are lots to see in this chart. I just picked a random few things for illustration:

- In the top left corner, the slant shows that the more affordable the homes are, the worse is the transit.
- The better the shopping, the better the dining.
- Interestingly, more diversity seems to mean lower creative capital (also the correlation is only moderate).
- Wellness scores fall within a rather narrow range compared to other categories, and they seem to be almost completely unrelated to any of the other factors.

***

(Note: I used JMP to generate this matrix. Excel unfortunately does not make scatter-plot matrices natively. JMP is great for such exploration... if the developers are reading this, please make it easier to man-handle the category labels! I made a mess of rotating the text on the right.)

P.S. I had an adventure processing the data from New York magazine. There appears to have been quite a few typos. For more, see my writeup on the book blog.

Sweet!

Posted by: Peter H | Jun 17, 2010 at 03:49 AM

Have you tried doing this in Protovis?

http://vis.stanford.edu/protovis/ex/brush.html

Posted by: C.vdBerge | Jun 17, 2010 at 10:26 AM

Nice post, some points to add:

- With "only" 11 variables and some dozens of observations the SPLOM does still work reasonable well. For 20 variables and some hundreds of cases this plot will fail.

- The ellipses help a lot in judging the correlations, but do we need a plot if this is essentially all we look at?

- Linking cases across the scatterplots will take us to even higher dimensional insights than just 2-d.

- Is the data available somewhere? I am keen on looking at the data in Mondrian.

- If there is a geographical reference in the data, i.e., the neighborhoods, we should link the map with the data. This will be far more powerful than any analysis which ignores this aspect.

But the important point is that you actually collected real data and addressed a real problem!Posted by: Martin | Jun 17, 2010 at 01:16 PM

Check out the "ezCor" function in the "ez" package for R. It plots something similar, but with additional features such as univariate densities, correlation coefficients, etc.

Posted by: Mike Lawrence | Jun 18, 2010 at 08:34 PM

regrg

Posted by: fefer | Jun 29, 2013 at 12:59 PM