« Another example of misleading time-based correlation | Main | Figuring out what data supports the argument, and what is just window-dressing »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

conchis

While 'proxy unmasking' is probably a useful tool in general, this isn't a great example to use: Hanushek isn't just using student scores as a proxy for teacher quality, he's actually referring to a whole literature that attempts to estimate teacher quality separately from student ability. (See his paper on 'The economic value of teacher quality', which a quick google will turn up.)

FWIW, I generally find that it's risky to base one's assessments of researchers' claims on journalists' descriptions or sound bites without looking at the original work. A lot tends to get distorted, either due to misunderstandings, or just the need to simplify for a lay audience.

junkcharts

conchis: this post is primarily about the reporter's description of the research, which I'll clarify in the original post. However, I do not take the position that every reader needs to consult the original research in order to form opinions of reported work.

I don't have time to read the whole paper at the moment, but just reading the introduction, I see Hanushek write these lines: "The analysis presented below is built on a simple premise: The key element defining a school’s impact on student achievement is teacher quality. In turn, the demand for teacher quality is derived from just this impact of teachers on student outcomes." This leads me to think that the reporter's version is not far from the truth.

I am aware that a lot of researchers use this premise. I have seen some of the value-added modeling work. I'm just not a fan of it.

conchis

Hey, appreciate the reply, but I'm not sure your edit really addresses the core misrepresentation that flows through from the 538 article to the OP: that 'teacher quality' is really just the same thing as 'student quality'. Whatever you think of the teacher value-added work (and I agree that there are issues with it), it's clear that it doesn't just naively equate 'teacher quality' with 'student test scores' in the way suggested by the 538 description and in your proposed 'unmasking'.

Re going to the source: I agree that in an ideal world it shouldn't be necessary to do this in order to evaluate a research claim, but I've been burned too many times to be willing to rely on press descriptions being detailed and/or accurate enough to form more than a suggestive basis for potential criticism. So while I think it's totally fine to say 'hey, it seems like it would be problematic if this guy was just using test scores as a proxy for teacher quality; I wonder if that's actually what he did?', and to them go and check that; I don't really think it's fair to say 'this claim is clearly rubbish because it's just using test scores as a proxy for teacher quality', without checking if that's actually the case. As it happens, that's not the case here, and so your criticism ends up just perpetuating the original misrepresentation.

NB: the line from Hanushek that you quote is not simply an assumption, but rather a summary of his interpretation of the empirical value-added research, and (at least as I read it) isn't saying 'teacher quality is the main driver of student achievement'; but rather that teacher quality is the main way schools can influence achievment (i.e. independently of student quality, and other non-school influences).

junkcharts

conchis: I don't think we are disagreeing here. My post is about the first step. Don't take what journalists (and in some cases, this includes scholars) say for granted. Be aware what part of the analysis is based on data and which part requires assumptions ("the simple premise"). After that, one can argue over whether the assumptions make sense.

In the case of Hanushek, if your interpretation is correct, then he should have rewritten that paragraph since I can't fault the journalist for her interpretation of it.

The comments to this entry are closed.

NEW BOOTCAMP



Link to Principal Analytics Prep

See our curriculum, instructors. Apply.
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR.

See my Youtube and Flickr.

Next Events

Jun: 5 NYPL Public Lecture on Analytics Careers, New York, NY

Past Events

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Courses

Numbersense: Statistical Reasoning in Practice, Principal Analytics Prep

Applied Analytics Frameworks & Methods, Columbia

The Art of Data Visualization, NYU

Junk Charts Blog



Link to junkcharts

Graphics design by Amanda Lee

Search3

  • only in Big Data

Community