« A data communications starter kit for university re-opening | Main | Answers on excess deaths, test accuracy and quick testing »


Feed You can follow this conversation by subscribing to the comment feed for this post.

Roel Peters

Hi there, it's brave to be the devil's advocate in this post. But I'm not sure the attack on the model is misguided. I don't think many argue that the rate of A-levels should be around 25%. The problem are the parameters that go into the model. The final paragraphs of this post elaborate on this briefly: existing inequities are propagated through the model. The feeling of no/less impact on the predictions must be tough to cope with.


RP: Of course, in the U.S. also, any measure of school achievement is highly correlated with household income and housing prices. Asking a model to resolve this deep-seated problem is asking too much. If society is willing to agree on targets, such as downgrading "good schools" so their students get fewer As than before, then a model could be built to operationalize that. But now we will get cries of reverse discrimination. I'm thinking the biggest mistake in this A-Levels exercise is to have teachers grade on the same scale as the actual test. Will elaborate on this in a new post.

Thomas Dietterich

Isn't the main issue that the grade should depend only on the student and not on the other factors? When the student sits the exam, the score they earn is entirely determined by them. But this was replaced by a model of "guilt by association", or perhaps we should call it "upgrade/downgrade by association". Such a model has the causality wrong, and is therefore rightly viewed as unjust. While a statistical model with group predictors may give accurate predictions in aggregate, it is not appropriate for making individual decisions. The entire effort is ill-conceived.


TD: I hear that argument, and I'm grasping at a response to it because the entire industry of predictive modeling (or its other monikers, superviesd learning, AI) is guilt by association. The entire recommendation systems (collaborative filtering) industry is also guilt by association. Predictive policing is guilt by association. Clinical trials too. And I've also been preaching that a shortest-path solution for the average driver does not yield the shortest path for each driver. It's a deep question.

Ian Watkins

The predicted grades from teachers were not necessarily an example of 'grade inflation'.

If, in a class of 10 pupils, you know that 4 are capable of achieving an A/A* then you should predict their grade at that level. You also know that 1 of them is likely to mess up and not achieve their grade (this could be through misreading a question, or a change in home circumstances, etc) as historically you achieve 30% A/A* grades. Which of your A/A* pupils are you going to downgrade arbitrarily?

Another issue with the algorithm is that it assigned grades to pupils at schools from failure (ungraded) upwards, meaning that ungraded was filled first. If the algorithm decided that there was a non-zero chance of ungraded at a school then someone was given a U. This also leads to downgrading as the lower achievement grades are filled before the higher ones.

And the question that I don't think I've seen asked, why are so many pupils failing to achieve the grade they're capable of when faced with exams? Are exams the correct method for pupil assessment?
Why is it that grades are inflated this year, when you could say that grades are deflated in previous years because of exams.


IW: All discussion of grading always come back to whether pupils or people should be graded at all. What is the purpose of grading?

The comments to this entry are closed.

Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.
Numbers Rule Your World:
Amazon - Barnes&Noble

Amazon - Barnes&Noble


  • only in Big Data

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here


R Fundamentals, Principal Analytics Prep

Numbersense: Statistical Reasoning in Practice, Principal Analytics Prep

Applied Analytics Frameworks & Methods, Columbia

The Art of Data Visualization, NYU

Signed copies at McNally-Jackson, NYC

Excerpts: Numbersense Ch. 1, 7, 8. NRYW

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee