Proxy unmasking

Consider this paragraph from a FiveThirtyEight article about the small-schools movement (my italics):

Hanushek calculated the economic value of good and bad teachers, combining the “quality” of a teacher — based on student achievement on tests — with the lifetime earnings of an average American entering the workforce. He found that a very high-performing teacher with a class of 20 students could raise her pupils’ average lifetime earnings by as much as $400,000 compared to an average teacher. A very low-performing teacher, by contrast, could have the opposite effect, reducing her students’ lifetime earnings by $400,000.

If I had told you that students who performed higher on achievement tests have higher average lifetime earnings by as much as $400,000 compared to students who performed average on achievement tests, you'd not be surprised--unless you are a skeptic of achievement tests. This is evidence that achievement test scores predict lifetime earnings.

Now, this is not what Hanushekthe journalist wants you to believe. HShe said that high-performing teachers "could raise" pupils' average lifetime earnings. Two logical jumps are made in one breath here: one is the use of student achievement scores as a proxy for "teacher quality"; the second is "causation creep," (i.e. allowing a causal interpretation to creep on correlational evidence), which is signaled by the use of weasel words like "could" and "may".

The use of proxy measures is the source for many "statistical lies". One tool I use is "proxy unmasking". Substitute the proxy metric with the actual metric. So in this example, when I see "high-performing teacher," I substitute "high-performing student" since the observed data measured students, not teachers. The sound you hear is air rushing out of the hyperventilated argument.

Comments

[conchis]

While 'proxy unmasking' is probably a useful tool in general, this isn't a great example to use: Hanushek isn't just using student scores as a proxy for teacher quality, he's actually referring to a whole literature that attempts to estimate teacher quality separately from student ability. (See his paper on 'The economic value of teacher quality', which a quick google will turn up.)

FWIW, I generally find that it's risky to base one's assessments of researchers' claims on journalists' descriptions or sound bites without looking at the original work. A lot tends to get distorted, either due to misunderstandings, or just the need to simplify for a lay audience.

[junkcharts]

conchis: this post is primarily about the reporter's description of the research, which I'll clarify in the original post. However, I do not take the position that every reader needs to consult the original research in order to form opinions of reported work.

I don't have time to read the whole paper at the moment, but just reading the introduction, I see Hanushek write these lines: "The analysis presented below is built on a simple premise: The key element defining a school’s impact on student achievement is teacher quality. In turn, the demand for teacher quality is derived from just this impact of teachers on student outcomes." This leads me to think that the reporter's version is not far from the truth.

I am aware that a lot of researchers use this premise. I have seen some of the value-added modeling work. I'm just not a fan of it.

[conchis]

Hey, appreciate the reply, but I'm not sure your edit really addresses the core misrepresentation that flows through from the 538 article to the OP: that 'teacher quality' is really just the same thing as 'student quality'. Whatever you think of the teacher value-added work (and I agree that there are issues with it), it's clear that it doesn't just naively equate 'teacher quality' with 'student test scores' in the way suggested by the 538 description and in your proposed 'unmasking'.

Re going to the source: I agree that in an ideal world it shouldn't be necessary to do this in order to evaluate a research claim, but I've been burned too many times to be willing to rely on press descriptions being detailed and/or accurate enough to form more than a suggestive basis for potential criticism. So while I think it's totally fine to say 'hey, it seems like it would be problematic if this guy was just using test scores as a proxy for teacher quality; I wonder if that's actually what he did?', and to them go and check that; I don't really think it's fair to say 'this claim is clearly rubbish because it's just using test scores as a proxy for teacher quality', without checking if that's actually the case. As it happens, that's not the case here, and so your criticism ends up just perpetuating the original misrepresentation.

NB: the line from Hanushek that you quote is not simply an assumption, but rather a summary of his interpretation of the empirical value-added research, and (at least as I read it) isn't saying 'teacher quality is the main driver of student achievement'; but rather that teacher quality is the main way schools can influence achievment (i.e. independently of student quality, and other non-school influences).

[junkcharts]

conchis: I don't think we are disagreeing here. My post is about the first step. Don't take what journalists (and in some cases, this includes scholars) say for granted. Be aware what part of the analysis is based on data and which part requires assumptions ("the simple premise"). After that, one can argue over whether the assumptions make sense.

In the case of Hanushek, if your interpretation is correct, then he should have rewritten that paragraph since I can't fault the journalist for her interpretation of it.