The LA Times (link) made the following comment as it describes the shameful situation in which the Dean of Admissions at the prestigious Claremont McKenna College (#9 on US News ranking of Liberal Arts colleges) inflated the average SAT scores of incoming students in order to manipulate national college rankings:

The collective score averages often were hyped by about 10 to 20 points in sections of the SAT tests, Gann said. **That is not a large increase**, considering that the maximum score for each section is 800 points.

Not a large increase? Are they wilfully ignorant, or just ignorant? I hope it's not a quote from CMC President Pamela Gann but an embellishment by the reporter. When one interprets whether 10 to 20 points is a "large" increase or not, one must find the right reference distribution of scores.

The maximum score is 800 but that is for individual scores. The 10 to 20 points manipulation is of the average scores of the freshmen class (about 300 students). The distribution of individual scores is much, much more variable than the distribution of average scores. So while 10 or 20 points for an individual may not be material, shifting the average score by 10 or 20 points is fraud of a massive scale.

Let's take a rough guess. According to the College Board, the standard deviation of individual scores is about 110 points (See the footnote on "Recentering" on this page). This means the standard deviation of the average scores of samples of 300 is 6.4 (this is known as the standard error). A 10-point fraud is about 1.6 standard errors. A 20-point fraud is just over 3 standard errors.

It's easier to visualize the scale of this:

Imagine the college's true SAT score average to be at "Z Score" = 0. Think of that as the median value (50th percentile). A 10-point fraud moves the average to 1.6 on the Z-Score scale, and as the diagram shows, that is moving from 50th to 95th percentile! And according to LA Times, that is the lower bound of the alleged manipulation.

Another way to see the size of this manipulation is to look at the average SAT scores for the top colleges (I found some data here but it's from 2004.) For instance, there is only about a 10-point spread between Columbia, Penn, Duke and Rice. Even a few points will shift the SAT score rankings.

***

So, after failing ethics, maybe the College is failing statistics too.

PS. [2/1/2012] @rags and I have been discussing what value of standard deviation should be used in the standard error formula. The proper value should be the standard deviation of the SAT scores of typical freshmen at CMC (or similar schools). The number I found is the standard deviation for the entire SAT test taking population so it is an over-estimate. If you find a school-level standard deviation number, please let me know and I'll adjust the computation. I don't think the conclusion would change much though given what we see in the table of average scores by college shown above.

PS. [2/4/2012] If the standard error was over-estimated, then the distribution of average scores would be even tighter than stated. This would make a 10- or 20-point manipulation even more egregious.

I used IPEDS data from 2010 to show mean test scores (IPEDS only gives 25th and 75th percentiles...so I took the midpoint, which I know is not statistically perfect, but I thought it close enough for these purposes).

http://bit.ly/zdXNoq

Posted by: TenSecondCynic | 02/01/2012 at 08:58 AM