Big Data, Plainly Spoken (aka Numbers Rule Your World)

How would we know if Google engineers really improved traffic flow at intersections

 

Mark P. and I have had discussions about evidence of efficacy related to Google Maps navigation: it is mighty hard to come up with data that prove that software like Google Maps or Waze actually delivers on the promise of reducing travel time. See this blog post for previous coverage.

These real-time optimization products often leave me with the feeling that they are executing "local" optimization, that is to say, they find the best solution within a highly constrained set of facts. A locally optimal solution is in general not optimal.

Since everyone has downloaded files from some web server, it's a good example to chew on. We have all seen the pop-up window that traces the estimated remaining time to finish downloading a large file. We also know that the prediction algorithm is so naive as to be laughable. It appears to perform a simple calculation: remaining file size divided by current download speed. This is the locally optimal answer. But internet download speeds are erratic, and so this local optimum is rarely accurate. We have all observed the estimated remaining time jump from 5 minutes to 5 hours, just because the internet speed has momentarily plunged.

The naive calculation is good if it is realistic to assume that the current download speed will hold steady till the time of completion. It is also good if it happens that the current download speed is close to the average speed of the entire download. (Neither of those are good assumptions.) The naive algo is reasonable if the set of facts available to the algorithm is constrained to (a) the current download speed and (b) the remaining file size. Imagine you are not allowed to use anything other than those two values, and you might decide to just divide one by the other.

Constraining these facts, though, is a choice. For example, the algo can be given the recent history of download speeds, rather than having just the current speed to work with.

***

Mark recently resurfaced this discussion because Google engineers have developed an optimization method for reducing the amount of congestion at road intersections (link)

Of course, the first thing I look for is proof. Here's what they say in the blog post:

Today, Green Light is live in over 70 intersections, helping to save fuel and lower emissions for up to 30 million car rides monthly. Early numbers indicate the potential to reduce stops by up to 30% and reduce emissions at intersections by up to 10%

There is no link to any papers, or data. So we just have to trust their word. The program manager elaborated:

We offer each city dedicated reports with tangible impact metrics, such as how many stops drivers saved at an intersection over time.

***

Let's think about how one might come up with evidence to prove the effectiveness of such optimization. The engineers say that the savings come from optimizing the timing of traffic lights to reduce stopping at intersections because the restart leads to pollution.

(Having driven around in Europe, I think the Europeans solve the problem completely by favoring roundabouts over traffic lights but that's a different thread.)

First, we need a baseline value for comparison. For any given intersection (one of those 70), the current traffic light timing is optimized, thus, we'd need to know the counterfactual, i.e. what would have been the amount of stoppage, had this intersection not implemented Google's optimization. To figure that out, we'd need to build a model that predicts the number of stops given a variety of factors, and we'd hope that this model has high accuracy.

Second, let's extend our focus from a single intersection to a stretch of highway involving a series of intersections. Is Google's algo implemented at every intersection, some selection of intersections, or just one intersection? For the sake of argument, let's say it's a single intersection, and we measure the change in stops at this intersection. It's highly likely that we may have pushed the congestion to another intersection, and what is gained locally is lost elsewhere.

***

Chapter 1 of Numbers Rule Your World (link) explores the complex world of traffic engineering through the tactic of ramp metering, i.e. controlling the flow of traffic onto highways. A key insight from that field is that congestion should be considered an unavoidable consequence when demand exceeds supply. Many tactics to reduce congestion merely re-distributes it.

The ramp-metering program is also notable for a valiant effort to evaluate real-life performance by running a principled experiment.

 

 

 

P esteem, bosom peril, flag to clamor, and anti-tech tech

What drug am I on this morning? Let me tell you, it's the anti-tech tech drug.

As you may have guessed, many new technologies are prone to abuse e.g. using generative AI to produce deep fakes, or what I learned earlier this week, QR code scams (link). The tech consensus is that we don't see any problems because one can use technology to combat such risks. I call this the anti-tech tech drug.

In the academic community, the digitization of knowledge has had profound impacts. One such impact is the creation of "metrics" - a whole field called bibliometrics - that purportedly measure the influence of specific researchers and research papers. What did I say about metrics? See this post from long time ago called "The inevitable perversion of metrics", which later got expanded into Chapters 1-2 of Numbersense (link).

Someone decided it would be a good idea to measure research contribution just like we measure the value of online advertising. Both count things like citations, clicks, downloads, views, etc. Citations are like hyperlinks. Google, whose initial algorithm was entirely based on counting hyperlinks, quickly discovered that hyperlinking is easily gamed. From very early on, humans invented multiple schemes to defeat it as a measure of importance, e.g. link exchanges, barters, buying links, selling links.

 

Not surprisingly, smart researchers have used myriad schemes to game bibliometrics. According to this recent article in Significance magazine, some less-than-honest researchers are flooding the zone with "fake" papers, written by bots. Just imagine having generative AI generate large numbers of "new" papers by feeding it already published articles.

Some anti-tech tech companies say don't worry, we have technology to combat such abuse. What is this tech? It's a plagiarism detection tool that can find copied contents.

What happens? The tricksters adapted.

***

Plagiarism detectors look for segments of text that have been seen before. What kind of text will evade detection? Unusual phrases, what we might call statistically unlikely phrases. Well, someone already wrote software that injects unusual phrases to replace common ones. The tricksters use such software to continue gaming the metrics.

Thus, these "rephrasing tools" have invented "P esteem" instead of "p value", "bosom peril" in lieu of "breast cancer", "flag to clamor" which means "signal to noise".

***

Even so, the authors of the Significance article were too polite to say out loud what's really broken about the peer review process, which has failed to reject many such papers. They reported that more than a thousand papers retrieved from PubMed contain the words "statically significant" instead of "statistically significant" in its title or abstract.

The authors respond to this enormous red flag by musing that "somehow, these goobledegook sentences also pass through peer review."

I can think of two likely scenarios:

1. some peer reviewers did not even open the papers they were supposed to opine upon
2. some editors colluded with authors in a publishing scam

 

Building a model of grade point averages

In the prior post, I featured the following chart:

This is a simulation of the distribution of GPAs of Yale graduates based on the just-published 2023 distribution of course grades.

What Yale released is a distribution of grades across all courses and departments during the last decade. The press picked up the 2023 grade distribution in particular, and I also used that as a basis for the simulation.

The published information is a distribution of grades, which is not the same as a distribution of GPAs. The GPA is an average over 36 courses, which is the requirement for graduation at Yale. In order to bridge the gap between course grades and student GPAs, we need to know how grades are distributed within students.

Since this information is missing, a statistician will leverage a "model" to fill in the blank.

The most naive model is that all students are alike, everyone is like the average student. In this case, everyone has the same distribution of grades as the average student. This means 22 As, 7 A-s, 4 B+s and 3 B or lower, for the 36 courses taken. This model replicates the average GPA of 3.70. In fact, every student has a GPA of 3.70.

An obvious improvement is to insert more variability so that different students have different GPAs while keeping the average GPA at 3.70.

***

The model that underlies the chart above is called "multinomial". It assumes all courses are like the average course. Any student enrolling in any course faces the following probabilities: 60% chance of getting an A, 20% chance of getting an A-, 10% B+, and 10% C+. (The last category C+ is a catch-all category: in real life, it stands for a grade B or below; based on the released information, the average of all grades in this category is 2.30, which equates to a C+.)

Each student takes 36 courses, and for each course, the student can obtain one of those four grades, with those stated probabilities. It turns out there are about 9,100 different grade combinations, starting with 36 As (perfect) and ending with 36 C+s. Each combination leads to a GPA value that ranges between 2.30 and 4.00.

Those 9,100 combinations reduce to under 200 distinct GPA values, if we round all GPAs to two decimal places. This is because there are many ways to obtain the same GPA. For example, anyone with 35 As and 1 A-s has a 3.99 GPA. But the one A- can come from the 1st, .... or 36th course taken.

For each value of GPA (e.g. 4.00, 3.99), I aggregate all the different grade combinations, and the probabilities of a student getting those combinations. Those probabilities are what you see in the chart above. The probabilities are cumulative meaning that they represent the probability of getting a GPA at or below some value.

For example, the probability of getting a GPA of 3.70 or below is 50%. This is often interpreted as half the students get 3.70 or below. The probability of getting a GPA at 3.87 or above (i.e. between 3.87 and 4.00) is 1 percent so the top 1 percent of graduates earn GPAs at 3.87 or above.

Notice how tightly the GPAs cluster. Half the student body is bunched between 3.64 and 3.75. Ninety-percent of the GPAs are between 3.48 and 3.87.

Since each GPA is an average of 36 grades, classic statistical theory predicts that the distribution of GPAs is substantially less variable than the distribution of grades. While 10 percent of grades are B or below, it is almost impossible to get a GPA of B+ or below.

***

The multinomial model itself uses many assumptions.

We assume every student faces the same odds. In fact, the grading distribution depends on which department, which professor, which course, etc. We also make many assumptions of independence. We assume that all students are independent of each other - this would be violated if some students collaborate on homework, or form study groups, for example. We assume that all courses are independent of each other. An obvious violation of this assumption is that some departments have stricter grading policies than others. As pointed out in my first post, it is also likely that a student who gets an A in one class tends to get an A in another class.

I also assume that the grade distribution at the course level is constant through time. There is disclosure in Yale's report that this assumption is not realistic - they talk about a Covid-19 effect that causes another inter-year jump in grades. Note that the first 9 courses taken by any graduating senior were from four years ago, and so I should have used the course grade distribution from that year, rather than that from 2023. This is an assumption of convenience, which is different from the structural assumptions described in the prior paragraph. I can easily fix this assumption by making the model more complex, and utilizing the additional data released by Yale. I just chose not to bother with the extra complexity for a blog post.

The independence assumptions, however, cannot be simply removed. I'd have to gather details like course departments and professors, and study groups, and so on. None of those data are simple to obtain.

On a high level, every assumption papers over some missing information. Many modern statisticians, especially those of the Bayesian variety, are highly comfortable making assumptions. Doing so is akin to clearing hurdles. Once past, the terrain is much smoother. However, it is very easy to forget about these assumptions, some of which may represent reality poorly.

After tasting the forbidden fruit, college admissions will...

The war against standards in college education continues to feature unscientific discourse. Consider this PBS piece, which starts with:

Standardized test scores in the U.S. used to all but make or break college applications for high school seniors.

Nope. U.S. colleges have not - for as long as I've lived - determined admission based solely on standardized test scores. In fact, the admission policy of the typical competitive American college differs from those in many other countries, primarily because they purport to evaluate applicants "holistically," incorporating subjective criteria such as high school GPAs, extracurricular activites, and athletics.

The PBS host then went on to describe a student who scored 18 out of 36 on the ACT, was told by Google that she should apply to community colleges instead of Emory, and complained that "[...] didn't consider my grades as an A student". This has multiple fallacies wrapped into an incoherent mess. Why should Google stop any student from applying to Emory? If she did apply, the school would have looked at her high school GPA because of the "holistic" admission policy. In fact, most Emory applicants are likely to have scored above 18 on the ACT, and those who scored above 18 are likely to also be A students at their high schools.

Later PBS informed us that this particular student chose to ignore Google, Emory did use a holistic application policy, and she did get admitted. Instead of writing a complimentary piece about the value of holistic policies in dealing with certain "edge cases" (what Gladwell would have called outliers), this journalist decided to use such edge cases to launch an attack on standardized testing.

***

Nevertheless, during Covid, many colleges made standardized test scores optional in the application process; and many colleges have decided since to remove standardized testing from their "holistic" admission policy. Universities like Berkeley and Chicago stopped requiring SAT scores prior to Covid; other schools like Columbia, and William & Mary have boarded the ship.

According to the PBS article, the reason is that these tests are "biased." This is one of my favorite topics, which I discussed extensively in Chapter 3 of Numbers Rule Your World (link). Bias is a word that is thrown around casually, with little appreciation of what it means. In the PBS article, they used a single example to define it:

White students were doing better on more heavily weighted easier items with common words like golf and canoe, but Black and Latinx students did better on items that were more academic and more difficult, although those were weighted less heavily, so, words like vehemence and sycophant, which I can barely say.

This is a very specific instance of a very convoluted type of bias, exactly the kind that is well-researched by ETS, the entity that runs the SAT. In the book, I cover the statistical discipline called "differential item functioning," which are fascinating experiments embedded into standardized tests that measure precisely this sort of biases of individual test items, a first step toward correcting for them.

The justification of bias elimination is typically the pursuit of "objectivity" or "fairness." What do these test critics propose as replacement for standardized testing? Teacher evaluations is one alternative mentioned by the PBS host.

What about teacher evaluation is "objective"? I have read thousands of applicant reference letters - the vast majority are routine letters that tell me nothing ("Student X is one of my best students who worked diligently and earned an A in my class."). The memorable ones are always personal - personal references are the opposite of "objective" or "fair".

I have read some extremely negative personal references as well. The writers clearly had taken on the mission to warn admission officers about the applicants. I had chosen to ignore them in the past. Why? Because such personal references are too subjective - I knew neither the applicant nor the reference writer, I had been served a sample size of one, for every negative letter there is likely to be another reference who wrote a positive one, so who was I to believe?

Flipped to the other side, why should I believe extremely positive personal references? The same set of circumstances applies.

Does "holistic" help here? Surprisingly no. Of course, I can believe the extremely positive personal references if the student also participated in the Math Olympiad or got perfect SAT scores. But then, the references do not add much new information. The reference letters provide additional information if and only if they are uncorrelated with the rest of the application, i.e. the straight-A student with high test scores in which a reference might disclose as a cheater, or the student with low grades in which a reference might disclose as a victim of bullying. If it's uncorrelated with the rest of the portfolio, then this is just a question of trust. It's far, far from the ideal of "objectivity" that these test critics are demanding.

If eliminating "bias" were their chief concern, then the movement to abolish standardized testing from college admissions will fail. It merely substitutes one form of bias with another.

[Don't get me started on the bias due to the differential writing abilities of the teachers writing these reference letters. Apparently, these test critics are not worried about it as I haven't encountered the movement to ban reference letters yet. Worse, has there been a proper scientific study of biases inherent in personal recommendation letters? Have there been any attempts to correct such biases?]

***

I have to credit the PBS host for hinting at one of the little-known drivers of the anti-standardized testing movement. Many colleges tasted the forbidden fruit during the Covid pandemic when 80 percent of colleges decided to make test scores optional.

Because it became easier to submit applications, many colleges including Emory experienced a surge in the number of applicants. According to PBS, applications to Emory jumped by 20 percent in a year.

What the host didn't mention is what the application surge did to Emory's "selectivity" rate, which is the number of admission offers divided by the number of applicants. Since class size is fixed in the near term, any increase in applicants just lowers the selectivity rate. Quickly looking around the web, I learned that Emory's selectivity rate was 15.6% for Fall 2020. So what if we added 20% more applicants and left the number of offers the same. The new selectivity rate would be 15.6%/1.2 = 13%. And what did Google say Emory's selectivity rate was in Fall 2021? You guessed it: 13%. The entire increase in applicants translated to an apparent rise in selectivity.

Did the quality of the average applicant pool improve? If the incremental applicants consist of those who do not want to take the ACT, or those who have taken the ACT but elected to not disclose the score, I'll let you take a guess.

I noticed and wrote about this more than 10 years ago - you can read all about this tactic and others used by colleges and grad schools in Numbersense (Chapter 1, link).

While the absolute selectivity rate of Emory has gone up, its relative selectivity rank against other competitive colleges might not have. That's because one of the enablers of the applicant surge is the Common App, a brilliant invention if there was ever.

This App, purportedly a time saver for college applicants, is a tide that raises all boats. The Common App increases the total number of applications filed across the entire pool of colleges that utilize it. It's the same concept as Amazon's one-click purchase. All these schools will experience a surge in applications and a corresponding improvement in selectivity, as Emory saw.

Of course, the Common App has been in play for a while. The new battleground is standardized test scores. Removing this requirement lowers the barrier to filing an application. Those schools that made it test optional are now riding the next tide. What will college admissions offices do after tasting the forbidden fruit?

***

One more thing. The PBS host was angry that Google had recommended that the student not apply to Emory due to an ACT score well below the average ACT score of prior admitted students. Ironically, one of the benefits of standardized test scores - and generally, of any statistical metrics - is its signaling value.

Using test scores, students and parents are able to make realistic assessments of their chances of admission at selective colleges, and opt out of applying, saving both time and money. Now, students can apply blissfully, and enter a lottery with very low odds of winning. The chance of admission for someone who scored 18 out of 36 on ACT to Emory is nearly zero. Without these scores, the schools pretend that everyone has a chance.

To learn a lot more tricks used by admission officers, see Chapter 1 of Numbersense (link).

There is no such thing as objective data

One of the persistent delusions about people running numbers is that they produce "objective" data. This delusion is frequently delivered in a self-righteous manner, directed at people who advocate adjusting the raw data. No no no, they say, you can't change the data because you are injecting subjectivity into the sacred raw data!

They claim that any adjustment is like injecting additives to pure, wholesome, raw foods.

In fact, all data are subjective. Even if the data quality is perfect, all data are still subjective because of measurement. This is one of the running themes in my second book  Numbersense (link). The first chapter addresses how school administrators can manipulate formulas to gain an edge in school rankings. The formulas are fixed but an endless number of tactics  delivers data that result in different values for the same underlying condition.

The latest demonstration of this malfeasance is trending news due to an investigation by a Columbia math professor (link). US News has now suspended Columbia from this year's rankings. What's shocking is their one bad apple framing. As documented in Chapter 1 of Numbersense (link), fraudulent data collection pervades all rankings at all levels of education: in any jurisdiction that matters enough to merit investigation, fraud has been revealed.

One of my favorite examples of fraudulent data collection from that chapter also happens to be an example of a "professional foul", i.e. a foul that is obvious to any outsider but deliberately oblivious to all insiders who partake and benefit from the foul.

This amusing example is the pushing of the "Common App", which lowers the barrier of application for any college applicant. Whereas in the past, each college has its own idiosyncratic application, post Common App, it takes about the same time to apply to one school as 15 schools. 

What the emergence of the Common App does is to "raise all boats". All selectivity rates (proportion of applicants admitted) will go up at all schools that participate in the Common App because the denominator (number of applicants) is jacked by the Common App. If schools are increasing their class sizes at all, the rate of increase in enrollment does not catch up to the rate of growth in applicants due to the Common App.

So - is selectivity rate of colleges an objective data point? In my view, it is a subjective metric, just like every other metric, easily manipulated.

***

This post is inspired by a statement I read about Wework - the controversial startup that rents office space without long-term contracts. (Disclosure: I'm a fan of Wework. I think it markets a product that delivers clear value to users without doing harm, unlike many startups that either delivers zero value or hides the harm done behind "convenience" and other positives. Wework's problem has always been figuring out a business model.)

The CEO claims that Wework's occupancy rate in this last quarter matched that of prepandemic levels, defined as late 2019.

Occupancy rate sounds like one of those pure, raw metrics that has an official definition, thus an analyst can report an objective value for it, beyond scrutiny.

Far from it. Let me point out a couple of ways to dress up this number. I'm not accusing anyone of doing anything. The following is merely a demonstration of possibility - as I disclosed above, I hope Wework finds a business model.

Like selectivity rate, occupancy rate is a fraction: the proportion of available office space that has a tenant. Like selectivity rate, let's focus on the denominator, the amount of available offices.

The occupancy rate may be 72% in both Q4 of 2019 and Q1 of 2022, while at the same time the number of tenants is lower in 2022 than 2019. It all depends on the number of offices available. Since the pandemic, Wework has closed locations - incidentally, any business should want to get rid of its lowest performers so this statement isn't about gaming numbers; I'm pointing out how the metric attaining the same value does not contain the same meaning.

Nevertheless, managers can choose to game the metric. If you get rid of the bottom 20% instead of the bottom 10%, you'd have pushed the occupancy rate even higher.

Now, let's take a tour of one of these buildings. You'll notice that some of the offices are being used for storage. There are chairs, desks, etc. stacked up in them. If I am the accountant, I'd remove these offices from "available" space.

There are also offices that have been converted to common areas for users who are just there for the day. An accountant has at least two options: a) remove these offices from available space, or b) count them as available space, and count them as "occupied". To legitimize option b), the accountant can also stipulate that such an office is counted as "occupied" so long as there is at least one user working in that space for x% of the days of the month.

***

If you came here because you're read Numbersense (link), those last two paragraphs might remind you of Chapter 6, in which I discuss the official unemployment rate. That chapter also portrays data's subjective nature. Unemployment levels depend on how the government defines employed and unemployed. In the book, you learn that someone may have only worked for one hour in a month, and the US government counts him/her as "employed". In addition, the government removes people from the employment count, classifying them as "not in labor force", just like the accountant who subtracts offices used as storage space as "not available".

tldr; If you don't know how something is measured, you don't know anything about the data.

 

 

 

 

Self-checkout and fast passes

In Chapter 1 of Numbers Rule Your World (link), I featured Disney's Fast Pass program, which is a virtual queueing setup that has delighted lots of amusement park goers.

The value of Fast Pass is commonly perceived to be reducing waiting time. It actually does nothing of this sort. What it does is to let people "multi-task" - do something else instead of standing in line. It's a masterpiece of managing perception.

This point is not obvious. Think about any optimization method applied to the queueing problem. A typical goal is stated as minimizing average waiting time. Waiting time is a loaded term - it is technically the difference between one's arrival time and service time (when one gets to the front of the line). If you take any person in the virtual queue, the "waiting time" is the same whether the person is physically present. (They did tell us that we keep our position in the line!) The average time between arrival and service has not changed, but guests are much happier. It's a win-win.

For a full analysis of how Fast Pass works, read my book (link),

***

A similar scenario is playing out in our supermarkets that have adopted self-checkouts.

Wait, am I saying self-checkouts do not actually reduce checkout times?

Let's compare a shopper doing your own groceries versus an experienced checkout clerk. Given the same basket of goods, who does it faster? Who knows where the buttons are? Who can look up codes on fruits and vegetables faster? Who knows how to apply coupons? Who has more practice? It's a slam dunk. 

Nevertheless, the lines for human clerks are frequently much longer. Are our eyes deceiving us?

At the Whole Foods near me, there are about 20 checkout counters with service clerks and maybe 10 self checkout stations. Typically when I'm there, fewer than 5 humans are working. Thus, if the experienced clerks work at twice my rate at bagging groceries, the line for human service is still going to be roughly the same length as the self-checkout line.

The value of self-checkout may not be coming from more efficiency; it comes from more capacity. The average self-service time may be longer than the average clerk service time, but the self-checkout user will still have waited less time because the supermarket management decides to devote more capacity to self-checkout.

As with the case of Disney, managing user perception matters.

***

One last thing.

Self-checkout also benefits from a self-selection effect. Shoppers who are buying just one item are much more likely to use self-checkout. In addition, those with baskets full of fruits and vegetables and don't want to deal with looking up codes are likely to choose to use human clerks. 

Thus, the mix of jobs experienced in the self-checkout line relative to the human service line is not the same, and the human clerks are likely to get the bulk of the jobs requiring longer service times.

***

If we set up a fair comparison, I'm betting that the store clerk finishes the job faster!

 

P.S. [7/27/2022] Minor edits to improve readability

Reclassifying data to manufacture statistics

In a previous post, I reviewed an analysis by Columbia professor Michael Thaddeus of Columbia's data that underlie its fast rise up the US News undergraduate ranking table. Even seemingly simple metrics such as class size are subject to subjective manipulation.

Thaddeus discovered the existence of two datasets, one for Columbia College and Columbia Engineering, and one for the School of General Studies (SGS). It turns out that SGS houses a lot of transfer students. Overall, 30% of Columbia undergrads are transfers, the highest proportion among Tier 1 private universities; in SGS, >75% of the students have some transfer credit.

This little detail has huge implications on Columbia's ranking. The US News ranking assigns about 23% of the weight to "outcomes", one of which is the graduation rate within six years. Notably, the graduation rate formula excludes transfer students.

From data Columbia submitted to the government, we learn that transfer students at Columbia (mostly at SGS) have a graduation rate of 85% compared to 96% at Columbia College and Columbia Engineering. The blended graduation rate is thus 92%. That apparently little difference of 4% would have shifted Columbia's graduation rate ranking from 6th to 26th.

Reclassifying data to game statistics is a professional trick. I covered an analogous situation in Chapter 1 of Numbersense (link). Claremont McKenna College was caught inflating SAT scores when reporting data to US News. The President of the College said it was just small potatoes as inflating the average score by 10-20 points on a scale of 800 was nothing (see Figure 1-8 if you have the book).

The scale of 800 is meaningless when the average SAT scores of CMC applicants fluctuate within a tight range (dozens of points) from year to year. In statistical terminology, we should look at a standardized scale that takes into account variability.

Similarly, the graduation rates at top-tier universities tightly bunch up in a narrow range. That's why a decrease of 4% drops Columbia by 20 spots.

This type of gaming has real effects on students. Thaddeus explains:

The picture coming into focus is that of a two-tier university, which educates, side by side in the same classrooms, two large and quite distinct groups of undergraduates: non-transfer students and transfer students. The former students lead privileged lives: they are very selectively chosen, boast top-notch test scores, tend to hail from the wealthier ranks of society, receive ample financial aid, and turn out very successfully as measured by graduation rates. The latter students are significantly worse off: they are less selectively chosen, typically have lower test scores (one surmises, although acceptance rates and average test scores for the Combined Plan and General Studies are well-kept secrets), tend to come from less prosperous backgrounds (as their higher rate of Pell grants shows), receive much stingier financial aid, and have considerably more difficulty graduating.

No one would design a university this way, but it has been the status quo at Columbia for years. The situation is tolerated only because it is not widely understood.

***

There is a final wrinkle to this issue of transfer students. They are not created equal. What makes Columbia special is that the transfer students tend to have lower achievement and higher financial need. This contrasts with transfer students at the UC system, for example, where they have higher graduation rates than non-transfers.

Again, Thaddeus provides useful background:

Some leading public universities, including several in the University of California system, have a higher proportion of transfer students than Columbia. Their transfer students largely come from two-year community colleges in the same state, providing a route to a bachelor’s degree for students at these colleges.

I have met this type of transfer students, and they are good students who work very hard to compete for a second chance to enter a school of their choice. So I'm not surprised that their outcomes are comparable to, or even better than, non-transfer students.

From an analytical perspective, this exercise demonstrates that one sometimes has to dig several layers deep in the data. Columbia has more transfer students, unlike its peers; plus, Columbia's transfer students have worse outcomes than its non-transfer students, unlike other schools.

 

 

 

Manufacturing statistics, US News style

US News school rankings are rigged. That much you should know after reading Chapter 1 of Numbersense (link).

Is it the school administrators' fault that they one-up each other gaming the ranking to the nth degree? Or is it the people's fault for selecting one school over another because it's ranked higher on US News? Should US News have fact-checked submitted data? Can we blame US News for providing a product that apparently is deemed highly valuable?

This investigative report by Columbia professor Michael Thaddeus (tip from Andrew Gelman's blog) serves two purposes:

a) it exposes the tricks of the trade that have pushed Columbia up the ranking table from #10-ish to #2 over the last two decades

b) it confirms that prior expose of bad behavior have not deterred gamers (and Columbia is certainly not the only school doing this)

***

Prof. Thaddeus started out by repeating US News's selling point that 80% of the ranking is "based entirely on numerical data collected by the institution" - data which are presumed to be "objective." (N.B. self-reported, unaudited)

He proceeded to follow the numbers trail, and in each case, discovered that such supposedly "objective" "numerical data" are not objective at all - which is a key lesson of Chapter 1 of Numbersense (link). In the professor's words, "several of the key figures supporting Columbia’s high ranking are inaccurate, dubious, or highly misleading." Most data are subjective, period.

Here as in Chapter 1, it is reasonable for a casual consumer of data insights to wonder how data such as "undergraduate class size", "percent of faculty who are full-time", and "student-faculty ratio" can be subjectively interpreted.

Aren't there clear, simple formulas for each item?

Undergraduate class size is the total number of class enrollments by all undergraduates divided by the total number of classes.

Percent of faculty who are full-time is the number of full-time faculty divided by the total number of faculty.

Student-faculty ratio is the number of enrolled students divided by the total number of faculty.

Is there any room for interpretation for such simple statistics?

***

Let Prof. Thaddeus tell you. It's more than your eyes can see!

According to US News, Columbia said 83% of its undergraduate classes have <20 students while 9% have over 50 students, so about 8% have between 20-50 students.

Prof. Thaddeus pointed out that the 83% number is well above most colleges in the U.S. The median value for all National Universities was 48%. The second and third highest numbers behind Columbia in the top 100 were both at 78%.

This statistic seems to contradict hearsay as I have never heard someone describe Columbia as a place with small class sizes, in fact, the opposite (admittedly, most of my sources are in graduate schools).

Prof. Thaddeus learned that US News applies the same set of criteria to determine class size as used in the "Common Data Set", which publishes standardized data on U.S. colleges. (Columbia is not part of this group, and is one of the eight schools in the Top 100 that do not belong.)

This is where you learn that "undergraduate class size" is not what you think it is. It's not what the simple formula above computes.

First, an undergraduate class is "a class that enrolled at least one undergraduate". This means that many graduate school classes are counted as undergraduate just because one out of 20 students is an undergraduate. Further, if a student is doing a joint bachelor's/master's program, then that student by him/herself will cause all his/her graduate-level classes to be classified as undergraduate!

Second, a long list of class types is excluded no matter how many undergraduates enroll in them. An internship class that grants course credit, for example, is not considered an undergraduate class.

Back to Columbia. Thaddeus now made a major discovery - that Columbia prepares two separate datasets (non public): one for "Columbia College and Columbia Engineering" and one for "School of General Studies". You may be reminded of 'In labor force' and "Not in Labor Force" when governments compute unemployment rates (Chapter 6 of Numbersense (link)). As I read this part, I'm anticipating that as this story unfolds, the students in the SGS will be forgetten and abandoned.

***

Leaving Columbia aside for a second. The class size metric is misleading even if schools provide honest data. The problem is the units. When US News say 83% of Columbia's classes have fewer than 20 students, the base unit is a class. At most schools, class size has a skewed distribution: a small number of courses (such as ECON 101) have enormous enrollments while a long tail of courses have relatively few students.

Let's say a school with 100 students offers 20 undergraduate classes. Each student takes 4 classes, resulting in 400 total enrollments. Two of these classes are compulsory so the enrollment is 100 each. The other 200 enrollments are split between 18 classes, so each of the smaller classes enrolls 11 students. Each student therefore has the same schedule, 2 large classes (n=100) and 2 small classes (n=11). Therefore, the average class size experienced for each student is (100+11)/2 = 56.

What if we apply the industry formula for computing class size? It's based on classes not enrollments. So 18 of the classes have size 11 while 2 of them have size 100, thus the average class size is 20.

Ninety percent of the classes have enrollment <= 20 students and yet each student experienced an average class size of 56. Ouch!

[P.S. If enrollments rather than classes were counted, then 50% of enrollments are in classes with size 100 while 50% were in classes below 15. This presents a more accurate description of the reality.]

***

Undeterred by the lack of disclosure, Prof. Thaddeus does the prototypical data science thing - scrape the web, in particular, Columbia's Directory of Classes, for information on every class and its maximum enrollment size.

While analyzing scraped data provides a good approximation, this method (like most web scraping exercises) cannot claim to be accurate. One problem is that capacity is not the same as actual enrollment. In fact, some classes may be offered in the catalog and later cancelled. But this information is sufficient to establish an upper bound for the class size, as well as a reasonable guess at the true average. Using this, the professor concluded roughly 63 to 67% of classes are <20 students, "nowhere near the figure of 82.5% claimed by Columbia."

For the other metrics, I'll refer you to Prof. Thaddeus report.

 

 

Numbers indeed rule our world, 10 years out

Statistics is the subject of generalizing data. In statistics, we study how to use imperfect data to draw general conclusions. We often have to wait for the future to reveal the evidence of success.

Ten years ago, McGraw-Hill published my first book, a best-seller, Numbers Rule Your World (link). I’ve hoped to explain five of the most profound concepts from my field in a way that anyone can understand and apply to their daily lives. I attempted the balancing act of featuring specific stories while bringing out the general principles. In the last couple of months, the Covid-19 pandemic has generated immense attention to data and statistics, and provided me confirmation, ten years on, that the contents of Numbers Rule Your World (link) have general applicability. It affirms that statistics is a great subject!

If you’re approaching my book for the first time, or re-reading it, here’s how you can work your way through it, making connections at various points to the Covid-19 crisis.

***

Chapter 2: Bad Spinach / Bad Scores

This is your perfect starting point. Half of the chapter details a disease outbreak investigation. You learn about the information networks the CDC puts in place to monitor unusual patterns of disease around the country. You discover data analysts whose job is to set off the alarm. How do they decide if the first handful of cases foretell an epidemic?

You encounter “shoe leather,” the painstaking collection of data to pinpoint the origin of an outbreak. It’s much harder than you might think. The U.S. politicians are making much noise about the early days of Wuhan. An epidemiologist may learn that 10 out of 15 first patients all visited the Wuhan seafood market before they got sick. But the Wuhan market might be a hugely popular destination, like Central Park in New York City. Finding 10 out of 15 people went to Central Park over a weekend could be more common than you think. If all 10 touched a particular rare species of wildlife, that's another story.

Another controversy of the moment is the huge cost of shutting down the economy, against the benefit of saving lives. I invite readers to think about this cost-benefit tradeoff. Because the disease outbreak was traced to a spinach processing plant in California, the FDA ordered a recall of all spinach, which depressed sales of salad greens for six months or so. The cost was much lower than for this pandemic, but the bacteria is well-known and thus predictable, and the lives saved were most likely zero, and even harder to pin down than in the current crisis.

People are already complaining that social distancing and other measures have not improved our numbers, by saying deaths were “unexpectedly low”. You discover how hard it is to prove the effectiveness of a policy decision because you need the counterfactual – what would have happened if the FDA did not recall the spinach.

Chapter 4: Timid Testers / Magic Lassoes

Everything you should know about the statistics of diagnostic testing and predictive models in one place. The motivating examples are (a) the fight against performance enhancing drugs in sports via anti-doping tests and (b) the false promise of terrorism prediction algorithms.

I wrote several posts (1, 2, 3) about the flawed Stanford antibody study that reported 50 times more infections in Santa Clara county in California than reported cases. Despite this headline, the raw data is also consistent with a scenario of zero proportion having antibodies. The statistical reason is the low underlying prevalence of antibodies, which caps the number of true positives in any sample. The number of false positives will be large relative to true positives, because the false positive rate applies to those without antibodies, which is the bulk of the population.

An extreme case of this phenomenon is terrorist prediction using a dragnet surveillance dataset. The set of terrorists is a drop in the ocean of people under mass surveillance. For a predictive model to capture most of these terrorists (low false negative rate), it must flag a lot of people – even a tiny false positive rate when applied to a huge population results in a lot of false accusations.

You see the distinction between a “chemical” false negative (positive) and a “practical” false negative (positive). To evade steroids testing, dopers use all kinds of tactics, such as handing in someone else’s urine. The diagnostic test finds a true negative but in practical terms, it’s a false negative. For swab testing, one may test negative if the swab doesn’t reach a virus-rich area, or if the patient is tested at the “wrong” time.

All tests have errors. What are the consequences of errors? Also, how likely will the error be discovered? For anti-doping, do dopers inform the test lab they made a mistake?

Chapter 1: Fast Passes / Slow Merges

The first chapter gives important insights on managing capacity, which is a pivotal issue for hospitals during the coronavirus crisis. I cover two case studies: (a) Disney’s management of theme-park capacity, and (b) state’s management of highway congestion.

The strategy of building more capacity is unavailable during a crisis, and wasteful in general. In this extraordinary time, some localities were able to bring up temporary hospitals. Transportation experts discovered that policies of regulating congestion are more effective. Highway ramps slow down the influx of new vehicles onto the highway, and are turned on before congestion arrives. The key is to delay as much as possible the onset of congestion. Sounds familiar? It’s the analogue to “flattening the curve”. Social distancing is supposed to reduce contacts, slowing down community spread. This tactic delays the peak demand for hospital resources. The chapter explains the scientific support for why such tactics work.

Chapter 5: Jet Crashes / Jackpots

Statistical significance is the topic of this chapter, one of the most common and also most commonly mis-used concepts in statistics.

In the Stanford antibody study referenced above, statisticians argued that the observed level of positive results is consistent with zero prevalence. This is a way of saying the result is not statistically significant. What we mean is that the study’s result (50 positives out of over 3,000 tests) would be considered normal even if nobody in the population actually had antibodies. This conclusion does not assert that the true prevalence is zero. The topic of statistical significance is made inaccessible to non-mathematicians by textbook authors piling on equations, but it need not be so. I even made a video on explaining "not statisitical significant" (link).

Before you apply a test of statistical significance, you must make sure you’re comparing apples to apples. In the context of air safety, you shouldn’t compare U.S. and foreign airlines on all routes; you should only compare them on routes they share, in which case, you’ll find no statistical difference. One of the bad stats that is circulating during this pandemic is the purported harm of ventilators. This is usually stated as the vast majority of patients put on ventilators eventually die (presumably, relative to those patients not put on ventilators). But only the most serious patients are placed on ventilators. The proper analysis should focus on patients who qualify for ventilators, and compare those who were put on ventilators to those who got other treatments.

We have already seen a host of invalid comparisons. Comparing complete data from years ago with incomplete data this year. Comparing regions that are in different phases of the epidemiological timeline. Comparing countries that have taken different approaches to managing the crisis.

Chapter 3: Item Bank / Risk Pool

This chapter addresses a deeper issue in using statistical averages. An illustrative example is educational testing like the SAT. Critics pointed to the gap in the average scores across racial groups as evidence of bias. But the racial grouping masks a third factor, which is differential ability between students. Due to various socio-economic factors, white students have higher average ability than black students. So the real question is: Among those students with comparable abilities, does the test still show bias toward one racial group? (The answer is not straightforward.)

During the pandemic, the “denier” faction has argued that the mortality rate of Covid-19 is “on a par with” that of seasonal influenza. Are those two rates comparable? The simple answer is no. Comparing mortality rates masks a third factor, which is the level of immunity in the population. Even at the most optimistic, the current level of immunity to the novel coronavirus is less than 20 percent (This statement generously presumes that (a) having antibodies confer immunity without question and (b) the New York City test result is not a scam, and the test is 100% accurate). By contrast, because of vaccination and recurrence, a good chunk of people have immunity to influenza. The mortality rate of Covid-19 should be compared to that of influenza among a population that has no immunity to either, or a population with similar levels of immunity to each.

The other example in this chapter explains how the insurance industry is an outgrowth of the law of large numbers. In any given year, health or auto insurers cannot predict with high accuracy which covered individuals will file claims, but the magic of statistics allows them to predict with great accuracy the aggregate claims. Disaster insurers run into trouble when rare events happen, such as when Class 5 hurricanes hit Florida. Suddenly, potentially all covered individuals tap into the insurance pool all at once. Insurers try to pool together risks of different types and different areas. The global pandemic is a huge threat to the insurance industry. Perhaps there are lawyers reading this, and they might tell me there are natural disaster exclusions. If those don’t apply, we will see a gigantic number of covered entities filing claims all at once.

Florida’s hurricane insurance market was dysfunctional also because the risk is predictable along some parameters. Those living inland realized they were subsidizing the coastal dwellers and wanted out. The coronavirus also creates different risk subgroups. The younger generations have much lower risks than the elderly. When social-distancing and other practices are applied to the whole community, the subgroup with lower risk peels away. A key difference is community transmission – the risk of hurricane damages does not spread by contact.

***

The above covers less than half of the book. Almost everything in those pages has direct relevance to the Covid-19 world. It’s quite heartening to find out ten years later numbers indeed rule our world.

Here’s a link to get your copy of Numbers Rule Your World. There are also translated copies in Chinese (simplified and traditional), Japanese, Korean, and Portuguese, as far as I know.

It's not us; it's the weather

If you are a frequent flier, you already know the gist of this nice article by the BBC: that airlines are allowed to sandbag the flight durations. A flight that takes 60 minutes will be portrayed to fliers as taking twice as long, if not longer.

The airlines are even allowed to lie about this practice. When your flight is delayed taking off, the captain claims that s/he will “make up for the delay,” as if the plane could be driven faster on command. (Were they deliberately going slower before?) The truth is that the schedule is padded, so that it can absorb a limited amount of delay.

This quote sums the situation up: “By padding, airlines are gaming the system to fool you.”

At the very bottom of the article, you’d find the potential motivation – to avoid compensating travelers for long delays, as required by law in some countries.

***

The situation here is similar to the road congestion problem discussed in Chapter 1 of Numbers Rule Your World (link). Managing perceived time is as important as managing actual time experienced by the traveler. Of course, reducing actual wasted time is preferred, especially to scientists working on the problem, but when the road/sky capacity is fixed and over-subscribed, it’s almost impossible to attain. The second half of the article addresses “why don’t the airlines work on efficiency instead of lengthening flight times?”

***

Another quote reveals: “Over 30% of all flights arrive more than 15 minutes late every day despite padding.”

The “on-time arrival rate” blessed by the Department of Transportation (DoT) is not what you think it is.

Let’s take a random flight that takes 60 minutes. This flight schedule might be padded and advertised as departing at 2 pm and arriving at 4 pm.

If the flight departs at 2 pm and takes 60 minutes, then you’d think on-time arrival is defined as arriving at 3 pm. You might agree to allow for some slack, say, 15 minutes. In this case, on-time arrival is arriving before 3:15 pm.

Given the discussion, you now know that on-time arrival is actually arriving before 4 pm since the schedule is padded not by 15 minutes but by 60 minutes.

And then you’d be wrong! Because there is padding upon padding. Airlines are allowed to claim “on-time arrival” if the flights arrive within 15 minutes of the scheduled arrival time, which in our example, has been padded by 60 minutes. So any flight arriving before 4:15 pm is counted as “on-time.”

***

Padding is not purely a bad thing. A certain amount of padding is necessary because lots of flights are vying for a limited amount of airport and air space. A padded schedule is a more accurate schedule. It acknowledges other factors that cause delays in arrival.

The gaming of the padded metric is what works people up. Gaming is possible because padding inserts subjectivity into the measurement. So long as subjectivity cannot be avoided, gaming is here to stay.

***

The reporter said airlines have spent billions on technologies to improve efficiency, i.e., managing actual experienced time. But, “this has not moved the needle on delays, which are stubbornly stuck at 30%.”

Now square that statement with this one: “Billions of dollars in investment [in modernising air traffic control] have in fact halved air traffic control-caused delays since 2007 while airline-caused delays have soared.”

Does this sound like the new technologies have successfully reclassified delays from air traffic control caused to airline caused? Passing the buck?

The next time your flight is delayed, the airlines will likely tell you, “it’s not us, it’s the weather.”