In a previous post, I reviewed an analysis by Columbia professor Michael Thaddeus of Columbia's data that underlie its fast rise up the US News undergraduate ranking table. Even seemingly simple metrics such as class size are subject to subjective manipulation.
Thaddeus discovered the existence of two datasets, one for Columbia College and Columbia Engineering, and one for the School of General Studies (SGS). It turns out that SGS houses a lot of transfer students. Overall, 30% of Columbia undergrads are transfers, the highest proportion among Tier 1 private universities; in SGS, >75% of the students have some transfer credit.
This little detail has huge implications on Columbia's ranking. The US News ranking assigns about 23% of the weight to "outcomes", one of which is the graduation rate within six years. Notably, the graduation rate formula excludes transfer students.
From data Columbia submitted to the government, we learn that transfer students at Columbia (mostly at SGS) have a graduation rate of 85% compared to 96% at Columbia College and Columbia Engineering. The blended graduation rate is thus 92%. That apparently little difference of 4% would have shifted Columbia's graduation rate ranking from 6th to 26th.
Reclassifying data to game statistics is a professional trick. I covered an analogous situation in Chapter 1 of Numbersense (link). Claremont McKenna College was caught inflating SAT scores when reporting data to US News. The President of the College said it was just small potatoes as inflating the average score by 10-20 points on a scale of 800 was nothing (see Figure 1-8 if you have the book).
The scale of 800 is meaningless when the average SAT scores of CMC applicants fluctuate within a tight range (dozens of points) from year to year. In statistical terminology, we should look at a standardized scale that takes into account variability.
Similarly, the graduation rates at top-tier universities tightly bunch up in a narrow range. That's why a decrease of 4% drops Columbia by 20 spots.
This type of gaming has real effects on students. Thaddeus explains:
The picture coming into focus is that of a two-tier university, which educates, side by side in the same classrooms, two large and quite distinct groups of undergraduates: non-transfer students and transfer students. The former students lead privileged lives: they are very selectively chosen, boast top-notch test scores, tend to hail from the wealthier ranks of society, receive ample financial aid, and turn out very successfully as measured by graduation rates. The latter students are significantly worse off: they are less selectively chosen, typically have lower test scores (one surmises, although acceptance rates and average test scores for the Combined Plan and General Studies are well-kept secrets), tend to come from less prosperous backgrounds (as their higher rate of Pell grants shows), receive much stingier financial aid, and have considerably more difficulty graduating.
No one would design a university this way, but it has been the status quo at Columbia for years. The situation is tolerated only because it is not widely understood.
***
There is a final wrinkle to this issue of transfer students. They are not created equal. What makes Columbia special is that the transfer students tend to have lower achievement and higher financial need. This contrasts with transfer students at the UC system, for example, where they have higher graduation rates than non-transfers.
Again, Thaddeus provides useful background:
Some leading public universities, including several in the University of California system, have a higher proportion of transfer students than Columbia. Their transfer students largely come from two-year community colleges in the same state, providing a route to a bachelor’s degree for students at these colleges.
I have met this type of transfer students, and they are good students who work very hard to compete for a second chance to enter a school of their choice. So I'm not surprised that their outcomes are comparable to, or even better than, non-transfer students.
From an analytical perspective, this exercise demonstrates that one sometimes has to dig several layers deep in the data. Columbia has more transfer students, unlike its peers; plus, Columbia's transfer students have worse outcomes than its non-transfer students, unlike other schools.
Comments