Big Data, Plainly Spoken (aka Numbers Rule Your World)

Know your data 46: using our data to set pricing

Google's former CEO Eric Schmidt infamously said something along the lines of "If you don't want others to know, you shouldn't be doing it in the first place". It showed the hubris of Silicon Valley at the time, and a certain deceitfulness. Because the truth is if they have your data, they can use the data to harm you, even if you haven't done anything wrong!

Finally, we have some evidence of what's been going on behind closed doors for a long time. They use our data to price discriminate. Same product, different prices, based on analyzing our data.

This practice, known as "surveillance pricing", caught notice because an industry body is suing New York State about a new law that requires companies to disclose that they are using algorithms (and data) to set different prices for different people (link). Look, the state is not banning surveillance pricing; they are requiring notification.

The industry doesn't want us to know.

The pushback from industry follows the usual script:

They bring up alternative scenarios of potential benefit to dismiss scenarios of harm. The state alleges that prices are raised on those who can afford them. In this case, they claim that the same algorithms are used to lower prices by offering discounts to selected customers.

I don't doubt that algos target special deals at specific customers. But I fail to understand why customers who love receiving coupons would object to the required disclosure of surveillance pricing - in fact, it would be great marketing to inform these customers that algos found them good deals; they might learn to like algorithms!

Surely, no consumers should object to such disclosure.

The industry apparently wants us to believe that the primary objective of surveillance pricing is to deliver discounts to customers. Anyone who read Chapters 3 to 5 (marketing data) of Numbersense (link) should recognize it as hogwash. Discounts result in lower revenues, unless the business can somehow prevent existing customers from using them. Whatever "leakage" happens, the business has to make up for these "lost" revenues. Thus, the same algos are likely to raise prices on other customers. With the amount of data at their disposal, it's not hard to figure out which customers are less price-sensitive, or have higher "willingness to pay".

I have worked on such algos. What kind of reception do you think data scientists would get from the business teams if we present to them an algorithm that delivers discounts to selected customers, leading to the predicted outcome of lower total expected revenues (and lower profitability)?

***

The usual script from the industry bodies also includes the false claim that telling the truth is "misleading". This is the same script used to oppose non-GMO labels. In this instance, they assert that customers will interpret the mandated disclosure as evidence of "price gouging".

These industry honchos aren't alarmed when consumers falsely believe that prices are fixed for everyone - when such disclosure isn't required!

I must digress to complain about another industry practice that is gaining popularity by the day, at least in the U.S. Many stores don't even bother putting up price tags. Some restaurants and coffee shops put up menus without prices. I just walked into a Vietnamese diner this afternoon, hoping to get an iced coffee to combat the oppressive heat in New York - well, the menu of bahn mi and side dishes is reasonably priced but the beverage and dessert sections have no prices! I walked out, disgusted. The iced coffee, based on redacted prices, is probably outrageously expensive, either compared to their past pricing, or compared to their peers. I'm guessing at least $6 (plus tax and the delightful... tip screen).

In their war against disclosures, some businesses won't even put up their prices.

And yet, we should believe that they won't use our data to maximize their profits.

 

 

Aging well, book-wise

2024 is a year that has brought me quite a bit of satisfaction. More precisely, world events have elevated the relevance of my two books, which is energizing. When I developed the themes of these books, I consciously sought topics that have enduring value.

The second chapter of Numbers Rule Your World (link) describes one of the grand successes of modern statistical science: the investigation of foodborne disease outbreaks. Toward the end of 2024, the U.S. has seen a spate of food recalls due to outbreaks of E.coli, salmonella, listeria, etc. (cucumbers, frozen waffles, frozen chicken, frozen cookie dough, carrots, deli meat, ...), the most famous of which was the McDonald’s E.coli outbreak of November 2024. Investigators believed that the outbreak was caused by contaminated onions served at McDonald’s stores (link). How epidemiologists make such determination is the subject of (half of) Chapter 2. The spinach outbreak featured in the chapter represents a best-case scenario, in which the investigators were able to confirm their cause-effect hypothesis by finding contaminated spinach at its source that tested positive for the exact strain of E.coli that caused the outbreak. As I wrote on this post (link), the McDonald’s investigation was not as lucky. The cost-benefit of food recalls deserves more scrutiny.

During the hurricane season, I revisited a key question of Chapter 3 of Numbers Rule Your World (link): is hurricane risk insurable? The insurance market is a game of musical chairs, in which players shift the risk around, and once in a while, the music stops, and a few get scorched. Many large insurers have already exited the scene, while small startups keep popping up, take incentives from the government, get run into the ground, and then bailed out by the government. See this blog post for more.

Numbers Rule Your World (link)’s Chapter 4, half of which concerns steroids testing in professional sports, became relevant again this year, as the top tennis player of either gender has become embroiled in controversies due to positive tests (link; link). At the time of writing, my take on steroids testing, that false negative rather than false positive is the Achilles heel of the testing programs, was controversial; later, Lance Armstrong got snitched and subsequently confessed, which laid waste to hundreds of “negative” tests over his long career (hello Holger Rune). Both tennis stars got a slap on the wrist because the governing bodies accepted their alternative explanation of contamination. Accidental contamination, and a myriad of other excuses, have always followed positive tests; it’s one of those applications that reveal the limit of statistical evidence: we can prove only the likelihood of foul play; to really nail a perpetrator, we need a confession, or physical evidence (e.g. caught with blood bags). My post about the tennis controversy is here.

***

Inflation continues to run unimpeded in New York City, and probably also where you live. The place where I used to get croissants (cornetti, to be geographically correct) now charges $7; it was $6 a couple of months ago, $5 a few years ago, and $4 when they opened about 10 years ago. (Those prices don’t include tax and tip, which adds at least 25%.) If you do the math, an increase from $4 to $7 in 10 years implies an average growth rate of 6% per year; and if you look up official statistics (link), the CPI index moved from 237 in Nov 2014 to 316 in Nov 2024, suggesting a growth rate of 3% per year. Chapter 7 of Numbersense (link) explains why the official statistic deviates so dramatically from lived experience. It demonstrates a very important key to success in data analytics: the analyst must know inside-out how numbers have come into being.

Chapter 6 of Numbersense (link) addresses the other big economic statistic, unemployment, in which, again, the official statistic seems to misrepresent reality.

Obesity drugs are all the rage, being pitched as the ultimate solution to the obesity crisis in the U.S. Chapter 2 of Numbersense (link) looked at the measurement of obesity, and treatments ranging from diets to surgery. CDC recently announced that the obesity rate of Americans has apparently plateaued, with the latest number showing a (statistically insignificant) slight decline (link). People were quick to attribute this to the rise of obesity drugs; thus, they are making the case that obesity drugs are causing a fall in average BMI, which is the standard metric used to measure obesity. I wonder how many of these people complained about using BMI to measure obesity during an earlier time when most treatments appear to be ineffective. The jury is still out on whether these drugs lead to long-term weight loss, which is one of the toughest tests of obesity treatments.

***

If you're looking for a gift this holiday season, get Numbers Rule Your World (link), get Numbersense (link), or get both!

P esteem, bosom peril, flag to clamor, and anti-tech tech

What drug am I on this morning? Let me tell you, it's the anti-tech tech drug.

As you may have guessed, many new technologies are prone to abuse e.g. using generative AI to produce deep fakes, or what I learned earlier this week, QR code scams (link). The tech consensus is that we don't see any problems because one can use technology to combat such risks. I call this the anti-tech tech drug.

In the academic community, the digitization of knowledge has had profound impacts. One such impact is the creation of "metrics" - a whole field called bibliometrics - that purportedly measure the influence of specific researchers and research papers. What did I say about metrics? See this post from long time ago called "The inevitable perversion of metrics", which later got expanded into Chapters 1-2 of Numbersense (link).

Someone decided it would be a good idea to measure research contribution just like we measure the value of online advertising. Both count things like citations, clicks, downloads, views, etc. Citations are like hyperlinks. Google, whose initial algorithm was entirely based on counting hyperlinks, quickly discovered that hyperlinking is easily gamed. From very early on, humans invented multiple schemes to defeat it as a measure of importance, e.g. link exchanges, barters, buying links, selling links.

 

Not surprisingly, smart researchers have used myriad schemes to game bibliometrics. According to this recent article in Significance magazine, some less-than-honest researchers are flooding the zone with "fake" papers, written by bots. Just imagine having generative AI generate large numbers of "new" papers by feeding it already published articles.

Some anti-tech tech companies say don't worry, we have technology to combat such abuse. What is this tech? It's a plagiarism detection tool that can find copied contents.

What happens? The tricksters adapted.

***

Plagiarism detectors look for segments of text that have been seen before. What kind of text will evade detection? Unusual phrases, what we might call statistically unlikely phrases. Well, someone already wrote software that injects unusual phrases to replace common ones. The tricksters use such software to continue gaming the metrics.

Thus, these "rephrasing tools" have invented "P esteem" instead of "p value", "bosom peril" in lieu of "breast cancer", "flag to clamor" which means "signal to noise".

***

Even so, the authors of the Significance article were too polite to say out loud what's really broken about the peer review process, which has failed to reject many such papers. They reported that more than a thousand papers retrieved from PubMed contain the words "statically significant" instead of "statistically significant" in its title or abstract.

The authors respond to this enormous red flag by musing that "somehow, these goobledegook sentences also pass through peer review."

I can think of two likely scenarios:

1. some peer reviewers did not even open the papers they were supposed to opine upon
2. some editors colluded with authors in a publishing scam

 

On the evidence raised by SAT distractors

After reading my prior post, long-time reader Mark (link) alerted me to a series of posts he wrote 10 years ago on the anti-SAT movement. This is a topic that bubbles up every few years, and it appears that the current generation of college administrators may go where their predecessors didn't. Some colleges have started to drop the SAT requirement for admissions.

Post 1, Post 2, Post 3, Post 4, Post 5

Let me summarize Mark's various posts and add some color commentary.

***

In his first post (link), Mark explores the relationship between SAT scores and college GPAs. This exercise in analyzing correlation is meaningful to both sides of the debate. The distractors detractors argue that "there already is something that accurately mirrors the grades a student gets in school. Namely: the grades a student gets in school" while proponents say SAT scores are the best predictors of college success.

For example, when MIT reinstated the SAT requirement after the Covid-19 pandemic (link), the dean of admissions stated:

Our research has shown that, in most cases, we cannot reliably predict students will do well at MIT unless we consider standardized test results [especially the math component] alongside grades, coursework, and other factors.

Mark points out that the right measure of any predictor is incremental insight – what additional nuance it adds to the collection of predictors. This explains why the SAT looks more like a general intelligence test than a high-school exam. Otherwise, the SAT score would be an unnecessary duplicate of high-school GPA. (Why the previous sentence is stated forwards but not backwards is another sign of weak analytics in this space.)

Mark's point is self-evident: assume that high-school GPAs predict college performance just as well as the SAT, as the distractors detractors suggest, then SAT scores should mirror high-school GPAs, and if they did, the two metrics would result in identical rankings of the students; if so, what's the cause for complaint?

***

The second post (link) examines various reasons SAT critics have given for dumping the test. Mark is unimpressed by the quality of the evidence, and in many cases, its absence.

The weirdest argument is that according to Google's description of its own hiring practices, Google does not consider standardized test scores or college ranking. Mark remarked that "Google doesn't talk much about SATs at least in part because they've largely maxed out the metric."

Let's say we tally up the background data of Google hires; we are likely to find a good number of Ivy League graduates with high SAT scores. Can Google legitimately claim their hiring does not consider test scores or college ranking?

Sure, someone is going to say "Those people weren't hired because they graduated from an Ivy League school, or that they had high SAT scores. They just happen to have those characteristics. Correlation is not causation!"

Well, admitted students to top colleges not only have high SAT scores, but they also have high GPAs, and other great things on their applications so did they get accepted because of the SAT scores, or the high GPAs?

***

The third post presents a series of math questions from actual SAT exams because Mark realized that many critics of the SAT attacked an imaginary scarecrow version of the SAT test, rather than what the test really is. Seeing these real questions helps ground the debate.

He also touched on one virtue of the SAT test, which is its transparency. Past exams with answers are readily available. By contrast, transparency is definitely missing in high-school GPAs. Nevertheless, in real life, transparency frequently backfires. Where are these SAT critics when it comes to high-school GPAs? Let's compile GPAs and split them by race - I'd be surprised if they didn't find a race gap. The reason that the race gap in GPAs isn't in the news is that schools don't release such data.

***

The fourth post is a must-read. Here is Andrew's recap of what Mark wrote.

Mark trashed those critics who grumbled about the point deduction for wrong answers in the SAT. Mark explained why the point deduction scheme in fact simplifies life for test-takers, because the penalty makes it counterproductive to do "random guessing" (what Mark describes as "guessing without first reading the question"). Tests that don't deduct points for wrong answers are more annoying because smart test-takers should guess on certain questions based on their level of confidence in knowing the answer.

The unspoken point behind Mark's post is that the SAT is rigorously designed, and continuously improved, by lots of statisticians, and therefore it is the best instrument we have. In Numbers Rule Your World (link), half of Chapter 3 is devoted to the statistical work behind a niche issue in SAT tests, work that reduces specific biases due to race and other characteristics.

The biggest irony in the anti-SAT movement is that on the major complaints about the SAT, alternative instruments such as high school GPAs and reference letters are far worse.

***

In the last post of this series (link), Mark turned our intuition upside down. He argued that the students most likely to benefit from the SAT admission requirement are those good students who go to public schools – or if we extend this line of reasoning, good students from underprivileged backgrounds. That is to say, the SAT requirement benefits precisely those people whom its distractors detractors say they care the most about, with the caveat that the benefit accrues to the cream of that crop.

Ten years later, the MIT dean of admissions echoed Mark's point when he said that MIT uses the SAT score to identify disadvantaged students who are most likely to succeed at MIT.

unlike some other inequalities — like access to fancy internships or expensive extracurriculars — our empirical research shows the SAT/ACT actually do help us figure out if someone will do well at MIT.

***

This last discussion leads us to some general topics in the area of statistical measures. No metric is perfect, most metrics are correlated with each other, all metrics promote perverse behavior, alternatives often appear better only because of lack of data or analyses. See Chapter 2 of Numbersense (link) for more on measurement, where I dig into the anti-BMI movement. BMI is the widely accepted metric for assessing obesity; like SAT scores, the metric has attracted a small loud group of distractors detractors.

 

There is no such thing as objective data

One of the persistent delusions about people running numbers is that they produce "objective" data. This delusion is frequently delivered in a self-righteous manner, directed at people who advocate adjusting the raw data. No no no, they say, you can't change the data because you are injecting subjectivity into the sacred raw data!

They claim that any adjustment is like injecting additives to pure, wholesome, raw foods.

In fact, all data are subjective. Even if the data quality is perfect, all data are still subjective because of measurement. This is one of the running themes in my second book  Numbersense (link). The first chapter addresses how school administrators can manipulate formulas to gain an edge in school rankings. The formulas are fixed but an endless number of tactics  delivers data that result in different values for the same underlying condition.

The latest demonstration of this malfeasance is trending news due to an investigation by a Columbia math professor (link). US News has now suspended Columbia from this year's rankings. What's shocking is their one bad apple framing. As documented in Chapter 1 of Numbersense (link), fraudulent data collection pervades all rankings at all levels of education: in any jurisdiction that matters enough to merit investigation, fraud has been revealed.

One of my favorite examples of fraudulent data collection from that chapter also happens to be an example of a "professional foul", i.e. a foul that is obvious to any outsider but deliberately oblivious to all insiders who partake and benefit from the foul.

This amusing example is the pushing of the "Common App", which lowers the barrier of application for any college applicant. Whereas in the past, each college has its own idiosyncratic application, post Common App, it takes about the same time to apply to one school as 15 schools. 

What the emergence of the Common App does is to "raise all boats". All selectivity rates (proportion of applicants admitted) will go up at all schools that participate in the Common App because the denominator (number of applicants) is jacked by the Common App. If schools are increasing their class sizes at all, the rate of increase in enrollment does not catch up to the rate of growth in applicants due to the Common App.

So - is selectivity rate of colleges an objective data point? In my view, it is a subjective metric, just like every other metric, easily manipulated.

***

This post is inspired by a statement I read about Wework - the controversial startup that rents office space without long-term contracts. (Disclosure: I'm a fan of Wework. I think it markets a product that delivers clear value to users without doing harm, unlike many startups that either delivers zero value or hides the harm done behind "convenience" and other positives. Wework's problem has always been figuring out a business model.)

The CEO claims that Wework's occupancy rate in this last quarter matched that of prepandemic levels, defined as late 2019.

Occupancy rate sounds like one of those pure, raw metrics that has an official definition, thus an analyst can report an objective value for it, beyond scrutiny.

Far from it. Let me point out a couple of ways to dress up this number. I'm not accusing anyone of doing anything. The following is merely a demonstration of possibility - as I disclosed above, I hope Wework finds a business model.

Like selectivity rate, occupancy rate is a fraction: the proportion of available office space that has a tenant. Like selectivity rate, let's focus on the denominator, the amount of available offices.

The occupancy rate may be 72% in both Q4 of 2019 and Q1 of 2022, while at the same time the number of tenants is lower in 2022 than 2019. It all depends on the number of offices available. Since the pandemic, Wework has closed locations - incidentally, any business should want to get rid of its lowest performers so this statement isn't about gaming numbers; I'm pointing out how the metric attaining the same value does not contain the same meaning.

Nevertheless, managers can choose to game the metric. If you get rid of the bottom 20% instead of the bottom 10%, you'd have pushed the occupancy rate even higher.

Now, let's take a tour of one of these buildings. You'll notice that some of the offices are being used for storage. There are chairs, desks, etc. stacked up in them. If I am the accountant, I'd remove these offices from "available" space.

There are also offices that have been converted to common areas for users who are just there for the day. An accountant has at least two options: a) remove these offices from available space, or b) count them as available space, and count them as "occupied". To legitimize option b), the accountant can also stipulate that such an office is counted as "occupied" so long as there is at least one user working in that space for x% of the days of the month.

***

If you came here because you're read Numbersense (link), those last two paragraphs might remind you of Chapter 6, in which I discuss the official unemployment rate. That chapter also portrays data's subjective nature. Unemployment levels depend on how the government defines employed and unemployed. In the book, you learn that someone may have only worked for one hour in a month, and the US government counts him/her as "employed". In addition, the government removes people from the employment count, classifying them as "not in labor force", just like the accountant who subtracts offices used as storage space as "not available".

tldr; If you don't know how something is measured, you don't know anything about the data.

 

 

 

 

Reclassifying data to manufacture statistics

In a previous post, I reviewed an analysis by Columbia professor Michael Thaddeus of Columbia's data that underlie its fast rise up the US News undergraduate ranking table. Even seemingly simple metrics such as class size are subject to subjective manipulation.

Thaddeus discovered the existence of two datasets, one for Columbia College and Columbia Engineering, and one for the School of General Studies (SGS). It turns out that SGS houses a lot of transfer students. Overall, 30% of Columbia undergrads are transfers, the highest proportion among Tier 1 private universities; in SGS, >75% of the students have some transfer credit.

This little detail has huge implications on Columbia's ranking. The US News ranking assigns about 23% of the weight to "outcomes", one of which is the graduation rate within six years. Notably, the graduation rate formula excludes transfer students.

From data Columbia submitted to the government, we learn that transfer students at Columbia (mostly at SGS) have a graduation rate of 85% compared to 96% at Columbia College and Columbia Engineering. The blended graduation rate is thus 92%. That apparently little difference of 4% would have shifted Columbia's graduation rate ranking from 6th to 26th.

Reclassifying data to game statistics is a professional trick. I covered an analogous situation in Chapter 1 of Numbersense (link). Claremont McKenna College was caught inflating SAT scores when reporting data to US News. The President of the College said it was just small potatoes as inflating the average score by 10-20 points on a scale of 800 was nothing (see Figure 1-8 if you have the book).

The scale of 800 is meaningless when the average SAT scores of CMC applicants fluctuate within a tight range (dozens of points) from year to year. In statistical terminology, we should look at a standardized scale that takes into account variability.

Similarly, the graduation rates at top-tier universities tightly bunch up in a narrow range. That's why a decrease of 4% drops Columbia by 20 spots.

This type of gaming has real effects on students. Thaddeus explains:

The picture coming into focus is that of a two-tier university, which educates, side by side in the same classrooms, two large and quite distinct groups of undergraduates: non-transfer students and transfer students. The former students lead privileged lives: they are very selectively chosen, boast top-notch test scores, tend to hail from the wealthier ranks of society, receive ample financial aid, and turn out very successfully as measured by graduation rates. The latter students are significantly worse off: they are less selectively chosen, typically have lower test scores (one surmises, although acceptance rates and average test scores for the Combined Plan and General Studies are well-kept secrets), tend to come from less prosperous backgrounds (as their higher rate of Pell grants shows), receive much stingier financial aid, and have considerably more difficulty graduating.

No one would design a university this way, but it has been the status quo at Columbia for years. The situation is tolerated only because it is not widely understood.

***

There is a final wrinkle to this issue of transfer students. They are not created equal. What makes Columbia special is that the transfer students tend to have lower achievement and higher financial need. This contrasts with transfer students at the UC system, for example, where they have higher graduation rates than non-transfers.

Again, Thaddeus provides useful background:

Some leading public universities, including several in the University of California system, have a higher proportion of transfer students than Columbia. Their transfer students largely come from two-year community colleges in the same state, providing a route to a bachelor’s degree for students at these colleges.

I have met this type of transfer students, and they are good students who work very hard to compete for a second chance to enter a school of their choice. So I'm not surprised that their outcomes are comparable to, or even better than, non-transfer students.

From an analytical perspective, this exercise demonstrates that one sometimes has to dig several layers deep in the data. Columbia has more transfer students, unlike its peers; plus, Columbia's transfer students have worse outcomes than its non-transfer students, unlike other schools.

 

 

 

Manufacturing statistics, US News style

US News school rankings are rigged. That much you should know after reading Chapter 1 of Numbersense (link).

Is it the school administrators' fault that they one-up each other gaming the ranking to the nth degree? Or is it the people's fault for selecting one school over another because it's ranked higher on US News? Should US News have fact-checked submitted data? Can we blame US News for providing a product that apparently is deemed highly valuable?

This investigative report by Columbia professor Michael Thaddeus (tip from Andrew Gelman's blog) serves two purposes:

a) it exposes the tricks of the trade that have pushed Columbia up the ranking table from #10-ish to #2 over the last two decades

b) it confirms that prior expose of bad behavior have not deterred gamers (and Columbia is certainly not the only school doing this)

***

Prof. Thaddeus started out by repeating US News's selling point that 80% of the ranking is "based entirely on numerical data collected by the institution" - data which are presumed to be "objective." (N.B. self-reported, unaudited)

He proceeded to follow the numbers trail, and in each case, discovered that such supposedly "objective" "numerical data" are not objective at all - which is a key lesson of Chapter 1 of Numbersense (link). In the professor's words, "several of the key figures supporting Columbia’s high ranking are inaccurate, dubious, or highly misleading." Most data are subjective, period.

Here as in Chapter 1, it is reasonable for a casual consumer of data insights to wonder how data such as "undergraduate class size", "percent of faculty who are full-time", and "student-faculty ratio" can be subjectively interpreted.

Aren't there clear, simple formulas for each item?

Undergraduate class size is the total number of class enrollments by all undergraduates divided by the total number of classes.

Percent of faculty who are full-time is the number of full-time faculty divided by the total number of faculty.

Student-faculty ratio is the number of enrolled students divided by the total number of faculty.

Is there any room for interpretation for such simple statistics?

***

Let Prof. Thaddeus tell you. It's more than your eyes can see!

According to US News, Columbia said 83% of its undergraduate classes have <20 students while 9% have over 50 students, so about 8% have between 20-50 students.

Prof. Thaddeus pointed out that the 83% number is well above most colleges in the U.S. The median value for all National Universities was 48%. The second and third highest numbers behind Columbia in the top 100 were both at 78%.

This statistic seems to contradict hearsay as I have never heard someone describe Columbia as a place with small class sizes, in fact, the opposite (admittedly, most of my sources are in graduate schools).

Prof. Thaddeus learned that US News applies the same set of criteria to determine class size as used in the "Common Data Set", which publishes standardized data on U.S. colleges. (Columbia is not part of this group, and is one of the eight schools in the Top 100 that do not belong.)

This is where you learn that "undergraduate class size" is not what you think it is. It's not what the simple formula above computes.

First, an undergraduate class is "a class that enrolled at least one undergraduate". This means that many graduate school classes are counted as undergraduate just because one out of 20 students is an undergraduate. Further, if a student is doing a joint bachelor's/master's program, then that student by him/herself will cause all his/her graduate-level classes to be classified as undergraduate!

Second, a long list of class types is excluded no matter how many undergraduates enroll in them. An internship class that grants course credit, for example, is not considered an undergraduate class.

Back to Columbia. Thaddeus now made a major discovery - that Columbia prepares two separate datasets (non public): one for "Columbia College and Columbia Engineering" and one for "School of General Studies". You may be reminded of 'In labor force' and "Not in Labor Force" when governments compute unemployment rates (Chapter 6 of Numbersense (link)). As I read this part, I'm anticipating that as this story unfolds, the students in the SGS will be forgetten and abandoned.

***

Leaving Columbia aside for a second. The class size metric is misleading even if schools provide honest data. The problem is the units. When US News say 83% of Columbia's classes have fewer than 20 students, the base unit is a class. At most schools, class size has a skewed distribution: a small number of courses (such as ECON 101) have enormous enrollments while a long tail of courses have relatively few students.

Let's say a school with 100 students offers 20 undergraduate classes. Each student takes 4 classes, resulting in 400 total enrollments. Two of these classes are compulsory so the enrollment is 100 each. The other 200 enrollments are split between 18 classes, so each of the smaller classes enrolls 11 students. Each student therefore has the same schedule, 2 large classes (n=100) and 2 small classes (n=11). Therefore, the average class size experienced for each student is (100+11)/2 = 56.

What if we apply the industry formula for computing class size? It's based on classes not enrollments. So 18 of the classes have size 11 while 2 of them have size 100, thus the average class size is 20.

Ninety percent of the classes have enrollment <= 20 students and yet each student experienced an average class size of 56. Ouch!

[P.S. If enrollments rather than classes were counted, then 50% of enrollments are in classes with size 100 while 50% were in classes below 15. This presents a more accurate description of the reality.]

***

Undeterred by the lack of disclosure, Prof. Thaddeus does the prototypical data science thing - scrape the web, in particular, Columbia's Directory of Classes, for information on every class and its maximum enrollment size.

While analyzing scraped data provides a good approximation, this method (like most web scraping exercises) cannot claim to be accurate. One problem is that capacity is not the same as actual enrollment. In fact, some classes may be offered in the catalog and later cancelled. But this information is sufficient to establish an upper bound for the class size, as well as a reasonable guess at the true average. Using this, the professor concluded roughly 63 to 67% of classes are <20 students, "nowhere near the figure of 82.5% claimed by Columbia."

For the other metrics, I'll refer you to Prof. Thaddeus report.

 

 

Changing metrics when you dislike the results

(Source: xkcd)

 

Chapter 2 of Numbersense (link) is titled "Can a new statistic make us less fat?" In the context of recent remarks by the acting New York State Governor, perhaps I should have called it "Can a new statistic save some people from dying?"

Trick question:

An executive tracks performance using a key metric displayed on a dashboard. The executive requests a review of the definition of the metric, then decides the definition must be revised, and after the revision, the new metric looks better than before. Which of the following is true?

a) Using the prior definition, the key metric was outperforming expectation

b) Using the prior definition, the key metric was underperforming expectation

***

The New York TImes reported that "Ms. Hochul said that some hospital executives have told her between 20 and 50 percent of their Covid patients are not suffering from severe symptoms, but that they are testing positive in the hospital incidentally, after being admitted for other reasons such as car accidents...As a result, beginning Tuesday, the state will begin to ask hospitals to break down how many patients are being admitted for acute Covid-19 symptoms, in an effort to further decipher this wave’s severity."

This is statistical alchemy that is frequently requested by people who don't like what the numbers are showing. It is a trick because they did not commit to restating the entire data series back to the start of data collection. This aforementioned issue is a "problem" only if they had recently changed the counting rules to include admissions for non Covid-19 reasons, therefore causing recent data to be inflated relative to older data, but that's not what they're saying.

Further, notice that they are commingling two separate re-definitions: one is Covid-19 vs not Covid-19; the other is mild vs "acute" Covid-19. I have never heard that "cases" only refer to "acute" Covid-19 cases, until now. The new definition relies on a subjective definition of "acute" and so they can report whatever number they want going forward.

To have any credibility, they need to (a) publish all the definitions that have been deployed in the last 2+ years, including the latest one, and (b) apply the latest definition to restate all historical data. Otherwise, no one can tell whether the trend is up or down since we are not comparing apples to apples!

To have maximum credibility, they should do (c) a comprehensive review of the definition, and correct for problems that undercount as well as overcount. For example, are they missing lots of cases found by people doing at-home testing? How are they dealing with people with multiple test results? (This is no small matter: if an infected person will take further tests till s/he tests negative, then each positive test is likely to be balanced by one or more negative tests in the near future. This dynamic affects how we can interpret the time series of positive tests divided by number of tests.)

***

In Numbersense (link), I trace how every few years, someone writes an article saying we need a new definition of obesity. The typical reason is that BMI classifies fit athletes as obese. I have never seen someone provide an analysis of how BMI misclassifies the average person. It's always the outliers. Notice in the above quote, the Governor did not give an aggregate proportion of what she considers misclassified. She said "some" hospitals gave her a concerning statistic.

As a data analyst, you have to think about the likelihood that the sample of hospitals she heard from is representative of all hospitals. Imagine you're working at a hospital for which most of the patients testing positive for Covid-19 are being treated for Covid-19. How likely are you to pick up the phone and call the Governor to tell her that you do not have a problem?

Thus, the sample she is working with is biased towards those hospitals experiencing this "problem". The real proportion of cases being "misclassified" is surely lower than what she announced.

Sharper understanding of vaccine effectiveness

Recently, some researchers changed the way they describe vaccine effectiveness. It used to be the Pfizer vaccine is 90% effective at preventing (first it was cases, then it was severe cases, finally it was) deaths. Now, they are saying the vaccine cuts (cases, severe cases, deaths) by 11 times. (See, for example, this recent report on the booster shots in Israel.)

Cutting some bad metric down by 11 times sounds extremely impressive. Is 11 times better than 90%?

The answer: they are saying exactly the same thing. If the rate for unvaccinated is 11x, and that for vaccinated is x, then the vaccine effectiveness is 1 - (x/11x) = 91%.

Cutting a number by 11 times is great but hard to achieve. Now, if we have a vaccine that cuts infection rate in half, that is still quite good, right?

That vaccine would have a VE of 50%. CDC doesn't even want to talk about such a vaccine.

***

I will say it again. A 90% vaccine effectiveness does not mean that 9 out of 10 people who got vaccinated are protected. As explained here, VE is a ratio of two infection rates. Both of these infection rates are very small, typically below 10%, and sometimes below 1%, depending on which clinical trial the data came from. In other words, over 90% of the people in the placebo arm did not get sick during the study period. It's logically impossible for 90 out of 100 people to be protected when fewer than 10 unvaccinated people would have gotten sick.

Statements such as "99% of vaccinated people are not hospitalized with Covid-19" are not very informative, unless we are also told what proportion of unvaccinated people are not hospitalized. If we have both, and flip the focus from not hospitalized to hospitalized, then we are looking at a ratio of rates, which is the vaccine effectiveness measure.

Those alternative descriptions are distracting and harmful when misinterpreted.

***

One last thing. A 90% effectiveness against cases and a 90% effectiveness against deaths are not at all similar things.

That's because the rate of death is a fraction of the rate of cases. When talking about cases, a 90% VE may mean pushing case rates from 10% to 1%. When talking about deaths, a 90% VE may mean pushing death rates from 0.5% to 0.05%. It is ludicrous to say that 99.95% of vaccinated people are protected from deaths when 99.5% of unvaccinated people would not have died.

A big drop in death rates is impressive on its own. Less effective is not the same as not effective. I don't like these alternative descriptions of effectiveness because the math is incorrect.

If you're not counted, you don't count

If you're not counted, you don't count.

The first law of counting is to count everything, and I do mean everything - not just the things you think you care about.

***

Imagine - as I did in Chapter 1 of Numbersense (link) - that a college advertises that 95% of its graduates found jobs within 6 months of graduation. It turns out the school collects data from a post-graduation survey - it's not disclosed anywhere in the marketing materials that it's 95% of survey respondents, not 95% of graduates. What's also not reported is the survey's response rate, which was 20%.

Nevertheless, the college administrators stand behind their statistics. They say the respondents and non-respondents did not show statistically significant differences in their gender distribution, average age, and type of degree; therefore, they assume that non-respondents also have a 95% employment rate within 6 months.

Do we think graduates who failed to land jobs have the same chance of filling out an employment survey as those who are gainfully employed after graduation? You can draw your own conclusion.

If you only have biased data on 20% of the graduates of this school, do you think you have a full understanding of the employment prospects of its graduates?

***

At the start of the Covid-19 pandemic - think March or April 2020, the U.S. suffered from a shortage of diagnostic tests. The official policy at the time was to test only people with severe symptoms.

We knew next to nothing about the novel coronavirus at the time. The data on cases became the basis of our knowledge - not only of the severity of the pandemic but also its trends, its demographics, its symptoms and injuries, and so on.

But triage testing leads to biased data. When the pandemic also grows quietly through asymptomatic transmission, our data led to a partially blind, misleading assessment of the situation. I warned about this in a Wired column - subsequently, colleges have shown that comprehensive testing is crucial to controlling the spread of Covid-19. Random testing of small samples of the population is a viable alternative we can still implement today to monitor new variants.

***

A curious thing happened during the pandemic to our unemployment statistics. Eight million Americans not only lost their jobs overnight but suddenly lost all interest in working. That's according to official statistics, which showed that the number of people in the "civilian labor force" dropped by 8 million.

People "not in the labor force" have always been a strange species. The government claims that these people either do not want work at all, or have not been looking for a job for an extended period of time. Their unemployed status is ignored in the unemployment rate. Oddly, as more people drop out of the labor force, the unemployment rate improves, all else being equal. This group is simply not counted; they are neither employed nor unemployed.

If one has the responsibility of reducing the unemployment rate, one will focus on unemployed people who are counted in the data, not the ones who are not counted. When you are not counted, you don't count.

In Chapter 6 of Numbersense (link), I explain how the government computes the unemployment rate in detail. Not counting a group of unemployed people hides some of the bad news. As I pointed out in this previous blog post, the sharp exodus out of the labor force at the start of the pandemic proves the serious shortcoming of the concept. None of the people who just lost their jobs due to Covid-19 are people who do not want work at all; their recently-ended employment proves otherwise. There also was not enough time that passed to qualify them as having given up on a job search.

***

The Bureau of Labor Statistics (BLS) does count everyone, including data on how many people have been removed from the labor force tally. These data just show up in an appendix, with much less attention and care.

We can't say as much about how the CDC counts Covid-19 cases in 2021. As this New York Times story disclosed, the CDC "stopped investigating breakthrough infections among fully vaccinated people unless they become so sick that they are hospitalized or die."

Just as BLS defines unemployed people as not really unemployed if they "don't want to work", CDC defines infected people as not really infected if they are fully vaccinated and are not hospitalized for severe Covid-19 symptoms. In this way, our data have come full circle, back to the same sad state that it was in March or April 2020. It tells us only about severe cases. To the extent that new variants spread through asymptomatic infections, we are as blind to that possibility as we were a year ago.

Furthermore, the change in counting rules upsets any trend analysis. Consider a real-world study of vaccine effectiveness. If the researchers report that the case rate of fully vaccinated people is much lower than the case rate of unvaccinated people, is it because of the vaccine or because we count mild or asymptomatic cases for unvaccinated people but not for vaccinated people?

It's not just fun and games with counting. Don't forget: infected, hospitalized, or dead people who got sick before 14 days are also treated as not really infected, hospitalized or dead. When you're not counted, you don't count. The CDC turns a blind eye to them because they happen to catch the disease prior to reaching so-called full vaccination status. Why do their lives not matter? That's something I have been wondering about throughout the pandemic, and I still don't have a good answer.