Dispute over analysis of school quality and home prices shows social science is hard

Most of my friends with families fret over school quality when deciding where to buy their homes. It's well known that good school districts are also associated with expensive houses. A feedback cycle is at work here: home prices surge where there are good schools; only richer people can afford to buy such homes; wealth brings other advantages, and so the schools tend to have better students, which leads to higher test scores.

A few weeks ago, the New York Times Upshot team ran some visual analytics to illustrate the correlation between school quality and home prices (link). Good for them, they went beyond a simple bivariate analysis. They also added commute time to the mix. The primary concern appears to be comparing so-called central cities with their suburbs in their respective regions. Some of these suburbs have better schools but the commute times are on average higher. Is it worth it to move to such neighborhoods?

This article has attracted much controversy. Alexander Russo summarizes some of the complaints here. The key objections cited in his report are as follows:

1. using test scores as a proxy for school quality is simple-minded
2. no accounting for other important variables such as demographics, backgrounds and so on (i.e. no attempt to control for other variables)
3. not causal: no one believes that moving a randomly selected student from a "bad" district to a "good" district will cause his/her test scores to jump by say two grade levels
4. using grade equivalent units which no one can understand

There are also complaints about the visual design. I will have some comments in a separate post on the sister blog. (The post should be available soon after this post goes live.)

***

I think #2 and #3 above are very important caveats. Point #1 is definitely a concern but critics admit that there are no great proxies. I will get to Point #4 below.

For me, the biggest issue is something missing completely from the original analysis: what decision scientists call the "indifference curves". What underlies this analysis is the assumption that families are making trade-offs between commute times, school quality and home prices. An indifference curve makes explicit what these trade-offs are: for example, how many more dollars per square feet is someone willing to pay in order to send kids to a school that is one grade level above another school? These curves are generally not linear. The value of moving from 3.5 to 4 grade levels is not as high as that of going from 1.5 to 2 grade levels, even though both shifts are precisely half a grade.

Once these trade-offs are in focus, we see the relevance of complaint #2. In the New York area chart, shown above, almost all of the districts are in the lower right quadrant relative to NYC, which means that a lot of districts on average have higher test scores and lower home prices than NYC. So, if those two factors were the primary reasons for selecting where to live, then no one would rationally choose to live in NYC! The addition of the third factor indicates that commute time may be another key consideration. Is commute time able to explain all of the data? That's not something easy to see from this chart, especially when commute time is reduced to two levels (under/over 30 minutes) and encoded in the colors.

***

The test score data came from the Stanford Education Data Archive (link). After you give them your email address, you will find a note saying that one standard deviation is roughly three grade levels. This raises a different kind of issue! Are the differences plotted on the chart meaningful? Usually, meaningful differences are at least two standard deviations, which would be six or more grade levels.

"Grade level equivalent" is a complicated concept. The same test is administered to students of different grades, so for example, the 5th-grade test is given to students in 3rd, 5th and 7th grades. The average scores of 3rd, 5th and 7th graders are computed. These are then mapped back to the 5th-grade scale. Anyone in 5th grade who scored above the average 7th grader is said to be two grade levels "ahead".

The fact that one standard deviation equals three grade levels tells me that the score distributions of 5th graders and 7th graders taking the 5th grade test overlap substantially. It seems likely that the difference between the 5th and 7th grader is not statistically significant - which, if true, is a sad commentary on our education standards.

Back to the chart above. If my interpretation based on the note from the Stanford Eduaction Data Archive is correct, then almost all of the horizontal differences on the NYT charts are practically meaningless.

***

Now, click here to read my comments on the data visualization.

 

 

Comments

[Jeff]

"It seems likely that the difference between the 5th and 7th grader is not statistically significant - which, if true, is a sad commentary on our education standards."

Is it? I think it doesn't necessarily mean that kids aren't learning between fifth and seventh grades; it could just be that variation among individuals of a certain age is really wide--fifth graders don't all cluster tightly around "grade level" with a few gifted/delayed outliers in each direction. I was surprised by the three-standard-deviation comment, but when I thought about my schooling experience I realized that it made a lot of sense.

[Kaiser]

Hi Jeff: I'd think the variability among 5th graders should be similar to the variability among 7th graders but because of selection effects, that may be false. However, I still stand by the original statement - remember that the 7th grader here is taking the 5th grade test, not a 7th grade test. Also, in that last sentence, you mean three-grade-levels not three-standard-deviations; the researchers say one SD is roughly three grade levels.

[Jeff]

Thanks--yes, I understand the distinction; I just misphrased it.

I suppose I agree with you about the state of our educational standards if you are critiquing the way we are defining and measuring success. The expectation that everyone will be in the same place at the same age is unrealistic. What people usually mean when they say something like this, though, is that our expectations aren't high enough, or that the instruction isn't sufficiently rigorous, or some such. This may also be true, but is a wide range in ability relative to the pace of the curriculum evidence of it? Given all the factors in and outside of school that can affect student performance, it's not obvious to me that it is.

[Kaiser]

Jeff: I am also an educator so this is self-critique. One factor not discussed here is the measurement instrument... I did not investigate what tests are used to calibrate the grade level equivalent scale.

I do think that on the first order, we want the average student in 7th grade to do better than the average 5th grader on a 5th grade test. In theory, a 5th grader who progresses to the 7th grade should be accumulating knowledge, so the entire score distribution should be moving up (variability may change though as some individuals may progress faster). But as you pointed out, the system is complex, and we are comparing different cohorts, so the reality may not be as clearcut.

I am mostly talking about the "average" student which doesn't exist but is a good aggregate measure. This is distinct from variability, which you are very concerned about.

[D]

Minor nitpick: if you refer to "item 2" or "item 3" in a bulleted list, it is easier to see those items if the list is numbered rather than simply using dots...

PS: I hate to be "that guy".

[Kaiser]

D: I fixed it, just for you :)

[Dzhaughn]

I think drawing an indifference curve is a terrible suggestion, because commute length data is garbage. It is not a proxy for distance to New York; one can just as well put Topeka on that graph; it would be a green dot, indicating a short commute.

Better to bin the whole thing!

[Luke Smith]

Some self-promotion here, but I did take a look at the data and attempt to aggregate the data myself (as best I could): https://seasmith.github.io/blog/reimagining_a_data_viz_good_schools_affordable_homes_nyt/

Also, the data is hidden behind an XHR, in a file called districts2.json, if I recall correctly. I didn't bother to dig into the Stanford data.