Graduation rates at 47 new small public high schools that have opened since 2002 are substantially higher than the citywide average, an indication that the Bloomberg administration’s decision to break up many large failing high schools has achieved some early success.
Most of the schools have made considerable advances over the low-performing large high schools they replaced. Eight schools out of the 47 small schools graduated more than 90 percent of their students.
This graphic included in the NYT article lent support to the "small schools movement". In particular, note the last sentence of the above quotation: it incorporates the oft-used device of subgroup support of a hypothesis, in this case, the subgroup of eight top-performing schools.
Such analysis is "dangerous", according to Howard Wainer, who discusses this and other examples of misapplication in a recent article in American Scientist, entitled "The Most Dangerous Equation". He alleged that billions have been wasted in the pursuit of small schools.
The issue concerns sample size. Dr. Wainer and associates analyzed math scores from Pennsylvania public schools. Average scores for smaller schools are based on smaller number of students, and therefore less stable (more variable). More variability means more extremes. Thus, by chance alone, we expect to find more smaller schools among the top performers. Similarly, by chance alone, we also expect to find more smaller schools among the worst performers.
The scatter plot lays out their argument. Focusing only on the top performers (blue dots), one might conclude that smaller schools do better. However, when the bottom performers (green) are also considered, the story no longer holds. Indeed, the regression line is essentially flat, indicating that scores are not correlated with school size.
File this as another comparability problem. Because estimates based on smaller samples are less reliable, one must take extra care when comparing small samples to large samples.
Dr. Wainer is publishing a new book next year, called "The Second Watch: navigating the uncertain world". I'm eagerly looking forward to it. His previous books, such as Graphic Discovery and Visual Revelations, both part of the Junk Charts collection.
Sources: "The Most Dangerous Equation", American Scientist, November 2007; "Small Schools Are Ahead in Graduation", New York Times, June 30 2007.
P.S. Referring back to the NYT chart above, one might wonder at the impossible feat of raising graduation rates across the board simply by breaking up large schools into smaller ones. This topic was taken up here, here and here. When evaluating the "small schools" policy, it is a mistake to discuss only the performance of small schools; any responsible analysis must look at improvement over all schools. Otherwise, it's a simple matter of letting small schools skim off the cream from larger schools.