The research on SAT bias is great data science

I just put up a follow-up video on the SAT controversy, including a speculation about what might happen if the SAT were to be toppled from college admission. Link to video.

***

One of the key arguments against the SAT is its alleged bias against certain subgroups of test-takers. There has been a lot of research into ways to measure and combat such biases. Chapter 5 of Numbers Rule Your World (link) gets into the science behind this.

In the video, I discuss the following example featured also in the book

Is this test item biased against some subgroup? If so, which subgroup, by how much, and for what reason? It turns out that it is basically impossible to answer these questions purely by intuition. Test designers run experiments to measure the bias, and are often surprised by what they find.

This whole area of research is a great example of a practical application of data science. The issues range from how to collect the right data, and how to analyze such data to make causal conclusions. You can read about it in my book (link).

Or try the video first.

 

Comments

[Ken]

It surprises me that this is still a problem as my memory is that ETS has done a massive amount of work on differential item functioning to identify when this problem occurred. However, it would seem best to get rid of this type of question, which basically is just checking how big someones vocabulary is. One option would be to supply a page or two of text explaining something and then ask questions related to the text.

It is interesting to compare what happens in the US to Australia. In Australia at the end of high school students sit a set of exams for the different subjects. The individual results are scaled and combined to give an overall estimate of what percentile result of all 18 year olds including those that did not complete high school or didn't complete subjects suitable for uni admission. This is then used as an entrance score for university. There is a sight complication that the half a dozen states all have their own curriculum and exams. There is also the option to sit for an international qualification instead.

[Kaiser]

Ken: yes, the book chapter covers the work on DIF. And analogy questions have been deprecated in the test. The problem of test item bias though is not limited by question type.

In the U.S., there has always been opposition to the kind of nummeric exam-based scoring used in many other countries. Admissions officials favor some kind of "holistic" approach, which is undoubtedly more subjective. The recent unsuccessful lawsuit against Harvard reveals the existence of subjective scores of applicant's "personality". That's the inconsistency I was pointing out... the SAT is being pushed out not because of subjective bias, that's not the real reason.