Big Data, Plainly Spoken (aka Numbers Rule Your World)

On the evidence raised by SAT distractors

After reading my prior post, long-time reader Mark (link) alerted me to a series of posts he wrote 10 years ago on the anti-SAT movement. This is a topic that bubbles up every few years, and it appears that the current generation of college administrators may go where their predecessors didn't. Some colleges have started to drop the SAT requirement for admissions.

Post 1, Post 2, Post 3, Post 4, Post 5

Let me summarize Mark's various posts and add some color commentary.

***

In his first post (link), Mark explores the relationship between SAT scores and college GPAs. This exercise in analyzing correlation is meaningful to both sides of the debate. The distractors detractors argue that "there already is something that accurately mirrors the grades a student gets in school. Namely: the grades a student gets in school" while proponents say SAT scores are the best predictors of college success.

For example, when MIT reinstated the SAT requirement after the Covid-19 pandemic (link), the dean of admissions stated:

Our research has shown that, in most cases, we cannot reliably predict students will do well at MIT unless we consider standardized test results [especially the math component] alongside grades, coursework, and other factors.

Mark points out that the right measure of any predictor is incremental insight – what additional nuance it adds to the collection of predictors. This explains why the SAT looks more like a general intelligence test than a high-school exam. Otherwise, the SAT score would be an unnecessary duplicate of high-school GPA. (Why the previous sentence is stated forwards but not backwards is another sign of weak analytics in this space.)

Mark's point is self-evident: assume that high-school GPAs predict college performance just as well as the SAT, as the distractors detractors suggest, then SAT scores should mirror high-school GPAs, and if they did, the two metrics would result in identical rankings of the students; if so, what's the cause for complaint?

***

The second post (link) examines various reasons SAT critics have given for dumping the test. Mark is unimpressed by the quality of the evidence, and in many cases, its absence.

The weirdest argument is that according to Google's description of its own hiring practices, Google does not consider standardized test scores or college ranking. Mark remarked that "Google doesn't talk much about SATs at least in part because they've largely maxed out the metric."

Let's say we tally up the background data of Google hires; we are likely to find a good number of Ivy League graduates with high SAT scores. Can Google legitimately claim their hiring does not consider test scores or college ranking?

Sure, someone is going to say "Those people weren't hired because they graduated from an Ivy League school, or that they had high SAT scores. They just happen to have those characteristics. Correlation is not causation!"

Well, admitted students to top colleges not only have high SAT scores, but they also have high GPAs, and other great things on their applications so did they get accepted because of the SAT scores, or the high GPAs?

***

The third post presents a series of math questions from actual SAT exams because Mark realized that many critics of the SAT attacked an imaginary scarecrow version of the SAT test, rather than what the test really is. Seeing these real questions helps ground the debate.

He also touched on one virtue of the SAT test, which is its transparency. Past exams with answers are readily available. By contrast, transparency is definitely missing in high-school GPAs. Nevertheless, in real life, transparency frequently backfires. Where are these SAT critics when it comes to high-school GPAs? Let's compile GPAs and split them by race - I'd be surprised if they didn't find a race gap. The reason that the race gap in GPAs isn't in the news is that schools don't release such data.

***

The fourth post is a must-read. Here is Andrew's recap of what Mark wrote.

Mark trashed those critics who grumbled about the point deduction for wrong answers in the SAT. Mark explained why the point deduction scheme in fact simplifies life for test-takers, because the penalty makes it counterproductive to do "random guessing" (what Mark describes as "guessing without first reading the question"). Tests that don't deduct points for wrong answers are more annoying because smart test-takers should guess on certain questions based on their level of confidence in knowing the answer.

The unspoken point behind Mark's post is that the SAT is rigorously designed, and continuously improved, by lots of statisticians, and therefore it is the best instrument we have. In Numbers Rule Your World (link), half of Chapter 3 is devoted to the statistical work behind a niche issue in SAT tests, work that reduces specific biases due to race and other characteristics.

The biggest irony in the anti-SAT movement is that on the major complaints about the SAT, alternative instruments such as high school GPAs and reference letters are far worse.

***

In the last post of this series (link), Mark turned our intuition upside down. He argued that the students most likely to benefit from the SAT admission requirement are those good students who go to public schools – or if we extend this line of reasoning, good students from underprivileged backgrounds. That is to say, the SAT requirement benefits precisely those people whom its distractors detractors say they care the most about, with the caveat that the benefit accrues to the cream of that crop.

Ten years later, the MIT dean of admissions echoed Mark's point when he said that MIT uses the SAT score to identify disadvantaged students who are most likely to succeed at MIT.

unlike some other inequalities — like access to fancy internships or expensive extracurriculars — our empirical research shows the SAT/ACT actually do help us figure out if someone will do well at MIT.

***

This last discussion leads us to some general topics in the area of statistical measures. No metric is perfect, most metrics are correlated with each other, all metrics promote perverse behavior, alternatives often appear better only because of lack of data or analyses. See Chapter 2 of Numbersense (link) for more on measurement, where I dig into the anti-BMI movement. BMI is the widely accepted metric for assessing obesity; like SAT scores, the metric has attracted a small loud group of distractors detractors.

 

After tasting the forbidden fruit, college admissions will...

The war against standards in college education continues to feature unscientific discourse. Consider this PBS piece, which starts with:

Standardized test scores in the U.S. used to all but make or break college applications for high school seniors.

Nope. U.S. colleges have not - for as long as I've lived - determined admission based solely on standardized test scores. In fact, the admission policy of the typical competitive American college differs from those in many other countries, primarily because they purport to evaluate applicants "holistically," incorporating subjective criteria such as high school GPAs, extracurricular activites, and athletics.

The PBS host then went on to describe a student who scored 18 out of 36 on the ACT, was told by Google that she should apply to community colleges instead of Emory, and complained that "[...] didn't consider my grades as an A student". This has multiple fallacies wrapped into an incoherent mess. Why should Google stop any student from applying to Emory? If she did apply, the school would have looked at her high school GPA because of the "holistic" admission policy. In fact, most Emory applicants are likely to have scored above 18 on the ACT, and those who scored above 18 are likely to also be A students at their high schools.

Later PBS informed us that this particular student chose to ignore Google, Emory did use a holistic application policy, and she did get admitted. Instead of writing a complimentary piece about the value of holistic policies in dealing with certain "edge cases" (what Gladwell would have called outliers), this journalist decided to use such edge cases to launch an attack on standardized testing.

***

Nevertheless, during Covid, many colleges made standardized test scores optional in the application process; and many colleges have decided since to remove standardized testing from their "holistic" admission policy. Universities like Berkeley and Chicago stopped requiring SAT scores prior to Covid; other schools like Columbia, and William & Mary have boarded the ship.

According to the PBS article, the reason is that these tests are "biased." This is one of my favorite topics, which I discussed extensively in Chapter 3 of Numbers Rule Your World (link). Bias is a word that is thrown around casually, with little appreciation of what it means. In the PBS article, they used a single example to define it:

White students were doing better on more heavily weighted easier items with common words like golf and canoe, but Black and Latinx students did better on items that were more academic and more difficult, although those were weighted less heavily, so, words like vehemence and sycophant, which I can barely say.

This is a very specific instance of a very convoluted type of bias, exactly the kind that is well-researched by ETS, the entity that runs the SAT. In the book, I cover the statistical discipline called "differential item functioning," which are fascinating experiments embedded into standardized tests that measure precisely this sort of biases of individual test items, a first step toward correcting for them.

The justification of bias elimination is typically the pursuit of "objectivity" or "fairness." What do these test critics propose as replacement for standardized testing? Teacher evaluations is one alternative mentioned by the PBS host.

What about teacher evaluation is "objective"? I have read thousands of applicant reference letters - the vast majority are routine letters that tell me nothing ("Student X is one of my best students who worked diligently and earned an A in my class."). The memorable ones are always personal - personal references are the opposite of "objective" or "fair".

I have read some extremely negative personal references as well. The writers clearly had taken on the mission to warn admission officers about the applicants. I had chosen to ignore them in the past. Why? Because such personal references are too subjective - I knew neither the applicant nor the reference writer, I had been served a sample size of one, for every negative letter there is likely to be another reference who wrote a positive one, so who was I to believe?

Flipped to the other side, why should I believe extremely positive personal references? The same set of circumstances applies.

Does "holistic" help here? Surprisingly no. Of course, I can believe the extremely positive personal references if the student also participated in the Math Olympiad or got perfect SAT scores. But then, the references do not add much new information. The reference letters provide additional information if and only if they are uncorrelated with the rest of the application, i.e. the straight-A student with high test scores in which a reference might disclose as a cheater, or the student with low grades in which a reference might disclose as a victim of bullying. If it's uncorrelated with the rest of the portfolio, then this is just a question of trust. It's far, far from the ideal of "objectivity" that these test critics are demanding.

If eliminating "bias" were their chief concern, then the movement to abolish standardized testing from college admissions will fail. It merely substitutes one form of bias with another.

[Don't get me started on the bias due to the differential writing abilities of the teachers writing these reference letters. Apparently, these test critics are not worried about it as I haven't encountered the movement to ban reference letters yet. Worse, has there been a proper scientific study of biases inherent in personal recommendation letters? Have there been any attempts to correct such biases?]

***

I have to credit the PBS host for hinting at one of the little-known drivers of the anti-standardized testing movement. Many colleges tasted the forbidden fruit during the Covid pandemic when 80 percent of colleges decided to make test scores optional.

Because it became easier to submit applications, many colleges including Emory experienced a surge in the number of applicants. According to PBS, applications to Emory jumped by 20 percent in a year.

What the host didn't mention is what the application surge did to Emory's "selectivity" rate, which is the number of admission offers divided by the number of applicants. Since class size is fixed in the near term, any increase in applicants just lowers the selectivity rate. Quickly looking around the web, I learned that Emory's selectivity rate was 15.6% for Fall 2020. So what if we added 20% more applicants and left the number of offers the same. The new selectivity rate would be 15.6%/1.2 = 13%. And what did Google say Emory's selectivity rate was in Fall 2021? You guessed it: 13%. The entire increase in applicants translated to an apparent rise in selectivity.

Did the quality of the average applicant pool improve? If the incremental applicants consist of those who do not want to take the ACT, or those who have taken the ACT but elected to not disclose the score, I'll let you take a guess.

I noticed and wrote about this more than 10 years ago - you can read all about this tactic and others used by colleges and grad schools in Numbersense (Chapter 1, link).

While the absolute selectivity rate of Emory has gone up, its relative selectivity rank against other competitive colleges might not have. That's because one of the enablers of the applicant surge is the Common App, a brilliant invention if there was ever.

This App, purportedly a time saver for college applicants, is a tide that raises all boats. The Common App increases the total number of applications filed across the entire pool of colleges that utilize it. It's the same concept as Amazon's one-click purchase. All these schools will experience a surge in applications and a corresponding improvement in selectivity, as Emory saw.

Of course, the Common App has been in play for a while. The new battleground is standardized test scores. Removing this requirement lowers the barrier to filing an application. Those schools that made it test optional are now riding the next tide. What will college admissions offices do after tasting the forbidden fruit?

***

One more thing. The PBS host was angry that Google had recommended that the student not apply to Emory due to an ACT score well below the average ACT score of prior admitted students. Ironically, one of the benefits of standardized test scores - and generally, of any statistical metrics - is its signaling value.

Using test scores, students and parents are able to make realistic assessments of their chances of admission at selective colleges, and opt out of applying, saving both time and money. Now, students can apply blissfully, and enter a lottery with very low odds of winning. The chance of admission for someone who scored 18 out of 36 on ACT to Emory is nearly zero. Without these scores, the schools pretend that everyone has a chance.

To learn a lot more tricks used by admission officers, see Chapter 1 of Numbersense (link).

NYT in story-first mode

NYT published a brain-dead article filled with statistical fallacies (link).

Since the Times has people on staff who know statistics, the real explanation for such an article is "story-first thinking". Data mined to support a story line.

***

The headline of the article provokes its least convincing statistic - that only seven black students were offered spots at Stuyvesant, one of the nation's top public high schools famous for STEM, out of 762 offers.

The position of the author is that the above statistic reveals systematic "racial and ethnic disparities". In statistics of education, "disparity" is a descriptive statistic describing a gap, which is different from "discrimination," which suggests foul play. The former is correlation while the latter is causation. This author mischievously used the term disparity while avoiding discrimination, even though the entire article presumes the disparity is unfair.

***

In fact, Stuyvesant (plus seven other similar schools in New York) only considers one factor in admissions, the student's score in the Specialized High Schools Admissions Test (SHSAT), and so the admission decision is not tainted by any judgmental criteria. All test-takers are ranked in order of test score, and then, each student is allocated a spot in their most preferred school - if all slots in the school have been filled, then the next preferred school is considered, and so on.

This allocation method guarantees "fairness" in the sense that if a student isn't allocated to a school higher on his/her preference list, it's because every offer for that school went to students who have scored higher on the SHSAT.

***

If we are interested in the question of whether the allocation of offers is fair across ethnic groups, the right question is: among students of similar ability (for example, score bands), do black or Latino students have a lower offer rate than whites and Asians?

This question of fairness pops in everywhere in education. In Chapter 3 of Numbers Rule Your World (link), I examined it in the context of designing fair SAT questions.

Surprise, surprise. Such data have not been released. Since there are no data, I'll explain conceptually how the numbers work.

***

Let's assume 30% of test-takers are Asians, and 20% each are blacks, latinos, and whites, and 10% others. Assume 50% of offers are given to Asians, 30% to whites, 5% each to blacks and latinos, and 10% to others. Overall, 15% of the test-takers receive an offer.

Given those assumptions, the offer rate for Asians is 25%, for whites it's 23%, for blacks and Latinos, it's 4%. Asians and whites are 1.5-1.7x more likely than the average to get an offer, while the offer rates of blacks and Latinos are a quarter of the average.

The NYT claims this proves that the process unfairly favors Asian and white students at the detriment of blacks and Latinos.

Not so fast. Remember all offers are completely determined by test scores, without any judgmental factors. So the proportion of students who receive offers is the same as the proportion who score above the cutoff score. Think of the cutoff score as the score of the last student taken up by a school. This cutoff score is an outcome of the offer process, not an input.

***

We can now split each subgroup into those who score above the cutoff and those below. Among Asians, the split is 25%-75%; whites, 23%-77%; blacks, 4%-96%; Latinos, 4%-96%; others, 15%-85%.

The reason why Asians and whites have more offers is because they have higher SHSAT scores.

Let's now insist that the offer rate for every ethnic group must equal the overall rate of 15%. For Asians and whites with a higher score distribution, this necessarily means that we cannot accept everyone above the cutoff score. It turns out that we can only accept about 60-67% of those who score above the cutoff score, and 0% of those who score below, in order to get to a weighted average of 15%.

For blacks and Latinos, we'd take 100% of those who score above the cutoff, but that isn't enough to get to 15%. We also must take 12% of those who score below the cutoff.

This is the unspoken math behind the NYT article. They are proposing: rejecting 40% of high-scoring Asians and whites while accepting 100% of high-scoring blacks and Latinos; and accepting 12% of low-scoring blacks and Latinos while rejecting 100% of low-scoring Asians and whites.

Shame.

 

 

Primer on regression adjustments 3

This is the third post in a series on using regression to correct for biases in observational data.

Let's review what we've learned so far. In the first post, I gave an example that shows regression will correct bias in a biased dataset. We looked at a sample of height data, in which women are over-represented relative to the underlying population. Without adjustment, the sample average height under-estimates the population average since women on average are shorter than men. After the regression adjustment, the gender-adjusted sample average height is almost identical to the population average.

In the second post, I expose a hidden assumption in the so-called regression-adjusted estimates, that everything is evenly distributed. In a gender adjustment (with only males and females in the population), the assumption is 50% males, 50% females. In an age-group adjustment (with 5 levels), the assumption is 20% in each level. If age is entered as a continuous variable, the age-adjusted estimate concerns persons of average age in the sample.

Needless to say, most real-world populations are not evenly balanced, nor do they contain only average persons. Therefore, if the goal is to estimate the population value, the regression adjustment may only correct the bias partially, and may even backfire, moving the sample average further from the population average. In other words, don't assume that dumping data into a regression software is a magic wand that cures all biases. And don't trust "experts" who wave this magic wand.

***

Chapter 3 of Numbers Rule Your World (link) is titled "the dilemma of being together." The chapter explores a fundamental problem in statistical analysis: the tension between capturing complex relationships in the data and finding generalizable features. The analyst resolves this question by deciding how much to aggregate the data. In regression models, this relates to how many averages do we include in the equation.

In the most naive model, the analyst summarizes the entire sample in one number, the sample average, and asserts that it is the best estimate of the population average. Thus:

[Naive] Population Average Height = Sample Average Height

Recall the statistics of our sample B of 900 people:

The naive model simply states:

[Naive] Population Average Height = 66.1

The average height of Population B is 67.1 inches, and the error of 1 inch is massive, equal to the width of a 95% confidence interval, as explained in post #1. This model makes systematic mistakes: the errors for males tend to be negative while the errors for females tend to be positive. A good model is one in which errors are as if random.

The underlying problem is that this naive model is too simple. On average, males are taller than females. So we err by summarizing everyone to one average.

To deal with this issue, we add complexity to the regression model. The so-called gender adjustment adds a gender term:

[Gender-stratified] Population Average Height = 66.6 + 2.6 if Male - 2.6 if Female

This is the regression output if I throw the data into software. It's the same equation I showed in post #2. People then take the constant term (66.6) and call that the "gender-adjusted" sample average height.

Note that the above equation is equvalent to saying:

[Gender-stratified 2] Population Average Height for Males = 69.2 while Population Average Height for Females = 64.0.

The reason for calling this "gender-stratified" should now be clear. In fact, this is a better representation of the model because the value of the constant term is not unique.

A mathematically equivalent formulation of the above model is:

[Gender-stratified 2b] Population Average Height = 69.2 - 5.2 if Female

This little wrinkle catches many software users by surprise. Depending on how the software developer codes the data, the regression outputs are different, even though they represent the same model. If you don't believe it, check that the new formula is also identical to:

[Gender-stratified 2] Population Average Height for Males = 69.2 while Population Average Height for Females = 64.0.

***

The standard way of thinking about gender adjustment is that it makes gender irrelevant. A gender-adjusted estimate removes the effect of gender, allowing us to focus on the outcome of interest.

A different - in my view, better - way of thinking about gender adjustment is that it acknowledges gender's effect on our outcome of interest. The regression adjustment involves building gender into the structure of the model.

The gender-stratified model finds that males and females have different heights: for males, we estimate, an average of 69.2 inches while for females, we estimate an average of 64.0 inches. By contrast, the naive model treats everyone as average, and we estimate the average height to be 66.1.

Gender adjustment does not prove that gender is unimportant. It's the opposite: building gender into the model means gender cannot be ignored. If gender does not matter, then we expect the gender-stratified model to reduce to the naive model. In other words, the coefficient of the gender term is zero (statistically speaking).

[Gender-stratified 3: Gender not important] Population Average Height for Males = 66.1 while Population Average Height for Females = 66.1.

And, formulated with a gender term:

[Gender-stratified 3: Gender not important] Population Average Height = 66.1 + 0 if Male - 0 if Female

So the average women in this special place is as tall as the average men.

What about the gender-adjusted sample average height of 66.6? This is the number that researchers in certain fields would have extracted from the stratified models -- recently, they have featured prominently in studies of vaccine effectiveness, including those by CDC scientists.

Mechanistically, these researchers have built gender-stratified regression models, then they ignore the gender coefficient, and report the constant term as gender-adjusted. As explained in post #2, this procedure is equivalent to averaging the gender-level height estimates, placing equal weights on male and female.

In statistics textbooks, we routinely warn readers that the coefficients of a regression model ought to be interpreted holistically. Should this rule apply to vaccine studies? If not, why not?

I'll pick this question up in the next part of this series.

***

If you missed the previous posts, you can still catch up on post #1 and post #2. 

Watching statistical gravity in action: elections and vaccines

A quick post today to let you know what's on my mind this week - I'm still chewing on these ideas, so if you have opinions, feel free to comment below. Will follow up on some of these in future posts.

Four Quick Takes on the Election Aftermath

1. In 2016, we mocked most election forecasters who said Hillary Clinton would win with 99% probability while agreeing that Nate Silver won the least bad award. Fast forward to 2020. Nate gave Biden a 89% chance of winning while the Economist said 96%. Who's got bragging rights here?
   
   
2. A former colleague pointed out to me that people mistakenly confuse a high chance of winning with a wide margin of victory. In a simulation model, a 96 percent chance of winning just means that Biden was predicted to win the most electoral votes in 96 percent of the simulated scenarios -- the 96 percent number does not contain information about the margin of victory in any scenario.
   
   
3. What Trump voters witnessed in Wisconsin, Michigan, Georgia and Pennsylvania was "statistical gravity."
   
   
4. In a Bayesian forecasting model, we get a posterior probability distribution as the output. This is typically explained as the proportion of scenarios in a simulation that end up with a given electoral-vote share. Tail events are rare and capture unusual scenarios - e.g. James Comey in the last days of 2016. Within this framework, how does one express the view that in the forthcoming election, one expects to see a tail event?

 

Four Quick Takes on the Coronavirus Vaccine

We finally got some news yesterday via Pfizer*, which sent out a press release claiming that its vaccine candidate may have "over 90 percent efficacy". They have not published a formal report, so there isn't much we can go on right now. Given the urgent need for a viable vaccine, this is one step in the right direction.

1. The claim is based on what happened seven days after the second dose so what the statement meant was that people who got the vaccine was 10 times less likely to be counted as a case within seven days of the second dose than those who got the placebo. This is an impressive early finding. Will "statistical gravity" kick in later?
   
   
2. People keep misinterpreting vaccine efficacy. The leader of the current White House coronavirus task force tweeted that the Pfizer vaccine "prevented infection in 90% of its volunteers". As I said before, this is not what it means. The case rate for vaccinated volunteers was about 0.1 percent while that for unvaccinated volunteers was 1 percent (99 percent of placebo takers did not get infected in the first 7 days). The former is 90% smaller than than latter.
   
   
3. I was scrambling to read the Pfizer protocol. I had assumed theirs and Moderna's would be similar since their vaccine candidates belong to the same family. I was surprised to learn that Pfizer was going to measure efficacy based on seven days as I already knew that Moderna doesn't count cases until 14 days after the second dose. So if we had run these two clinical trials side by side, Pfizer's analysis would have been finished before Moderna even started theirs. When both trials report results, we will have a much richer understanding of this type of vaccines.
   
   
4. People are calling for subgroup analysis. That will take time. This interim analysis is based on a total of about 90 infections. If you break that number down to subgroup level, there's not enough data yet to provide a useful anaswer. Be patient. Allow statistical gravity to work.

* Should give credit to BioNTech which is the company in Germany that develops this vaccine. Pfizer is running the vaccine trial.

If people would buy the real thing, there would be no counterfeits

In Chapter 3 of Numbersense (link), I told the story of the software industry complaining that pirates stole billions worth of software each year. This estimate of the value of software embeds the fantasy that people who use pirated software would have purchased legal software at list price should piracy be eradicated. It's a failure to think counterfactually. Many people who use pirated software cannot afford, or do not want, to pay for the legal version.

The tech industry invokes this logical flaw frequently. Earlier this week [a month ago by the time this post is published], news broke that the U.S. Customs and Borders Protection (CBP) intercepted "counterfeit" Apple Airpods but quickly, tech-savvy observers pointed out that those earbuds are made by OnePlus for Android phones, an Apple competitor. The photos included in the CBP press release even showed the seized products in OnePlus packaging.

CBP estimated that the seized products had a value of $389,000, by multiplying the number of products with the list price of Apple Airpods. Such an estimate assumes that if OnePlus earbuds were banned, consumers would have all purchased the much more expensive Apple product. The list price for OnePlus earbuds is less than half that of Airpods.

In this ArsTechnica article, the CBP claims $1.5 billion worth of counterfeit products are seized in a year. This number appears to suffer from the same counterfactual fallacy. What would consumers do if OnePlus earbuds were banned? A proportion of them will go back to wired headphones. Another group will migrate to other earbuds that are in a similar price range as OnePlus. These are not merely speculation - we do know that these customers elected to buy OnePlus instead of paying double for Apple.

If everyone who purchased the counterfeits would buy the real thing at list price, why didn't they buy the real thing in the first place?

 

Numbers indeed rule our world, 10 years out

Statistics is the subject of generalizing data. In statistics, we study how to use imperfect data to draw general conclusions. We often have to wait for the future to reveal the evidence of success.

Ten years ago, McGraw-Hill published my first book, a best-seller, Numbers Rule Your World (link). I’ve hoped to explain five of the most profound concepts from my field in a way that anyone can understand and apply to their daily lives. I attempted the balancing act of featuring specific stories while bringing out the general principles. In the last couple of months, the Covid-19 pandemic has generated immense attention to data and statistics, and provided me confirmation, ten years on, that the contents of Numbers Rule Your World (link) have general applicability. It affirms that statistics is a great subject!

If you’re approaching my book for the first time, or re-reading it, here’s how you can work your way through it, making connections at various points to the Covid-19 crisis.

***

Chapter 2: Bad Spinach / Bad Scores

This is your perfect starting point. Half of the chapter details a disease outbreak investigation. You learn about the information networks the CDC puts in place to monitor unusual patterns of disease around the country. You discover data analysts whose job is to set off the alarm. How do they decide if the first handful of cases foretell an epidemic?

You encounter “shoe leather,” the painstaking collection of data to pinpoint the origin of an outbreak. It’s much harder than you might think. The U.S. politicians are making much noise about the early days of Wuhan. An epidemiologist may learn that 10 out of 15 first patients all visited the Wuhan seafood market before they got sick. But the Wuhan market might be a hugely popular destination, like Central Park in New York City. Finding 10 out of 15 people went to Central Park over a weekend could be more common than you think. If all 10 touched a particular rare species of wildlife, that's another story.

Another controversy of the moment is the huge cost of shutting down the economy, against the benefit of saving lives. I invite readers to think about this cost-benefit tradeoff. Because the disease outbreak was traced to a spinach processing plant in California, the FDA ordered a recall of all spinach, which depressed sales of salad greens for six months or so. The cost was much lower than for this pandemic, but the bacteria is well-known and thus predictable, and the lives saved were most likely zero, and even harder to pin down than in the current crisis.

People are already complaining that social distancing and other measures have not improved our numbers, by saying deaths were “unexpectedly low”. You discover how hard it is to prove the effectiveness of a policy decision because you need the counterfactual – what would have happened if the FDA did not recall the spinach.

Chapter 4: Timid Testers / Magic Lassoes

Everything you should know about the statistics of diagnostic testing and predictive models in one place. The motivating examples are (a) the fight against performance enhancing drugs in sports via anti-doping tests and (b) the false promise of terrorism prediction algorithms.

I wrote several posts (1, 2, 3) about the flawed Stanford antibody study that reported 50 times more infections in Santa Clara county in California than reported cases. Despite this headline, the raw data is also consistent with a scenario of zero proportion having antibodies. The statistical reason is the low underlying prevalence of antibodies, which caps the number of true positives in any sample. The number of false positives will be large relative to true positives, because the false positive rate applies to those without antibodies, which is the bulk of the population.

An extreme case of this phenomenon is terrorist prediction using a dragnet surveillance dataset. The set of terrorists is a drop in the ocean of people under mass surveillance. For a predictive model to capture most of these terrorists (low false negative rate), it must flag a lot of people – even a tiny false positive rate when applied to a huge population results in a lot of false accusations.

You see the distinction between a “chemical” false negative (positive) and a “practical” false negative (positive). To evade steroids testing, dopers use all kinds of tactics, such as handing in someone else’s urine. The diagnostic test finds a true negative but in practical terms, it’s a false negative. For swab testing, one may test negative if the swab doesn’t reach a virus-rich area, or if the patient is tested at the “wrong” time.

All tests have errors. What are the consequences of errors? Also, how likely will the error be discovered? For anti-doping, do dopers inform the test lab they made a mistake?

Chapter 1: Fast Passes / Slow Merges

The first chapter gives important insights on managing capacity, which is a pivotal issue for hospitals during the coronavirus crisis. I cover two case studies: (a) Disney’s management of theme-park capacity, and (b) state’s management of highway congestion.

The strategy of building more capacity is unavailable during a crisis, and wasteful in general. In this extraordinary time, some localities were able to bring up temporary hospitals. Transportation experts discovered that policies of regulating congestion are more effective. Highway ramps slow down the influx of new vehicles onto the highway, and are turned on before congestion arrives. The key is to delay as much as possible the onset of congestion. Sounds familiar? It’s the analogue to “flattening the curve”. Social distancing is supposed to reduce contacts, slowing down community spread. This tactic delays the peak demand for hospital resources. The chapter explains the scientific support for why such tactics work.

Chapter 5: Jet Crashes / Jackpots

Statistical significance is the topic of this chapter, one of the most common and also most commonly mis-used concepts in statistics.

In the Stanford antibody study referenced above, statisticians argued that the observed level of positive results is consistent with zero prevalence. This is a way of saying the result is not statistically significant. What we mean is that the study’s result (50 positives out of over 3,000 tests) would be considered normal even if nobody in the population actually had antibodies. This conclusion does not assert that the true prevalence is zero. The topic of statistical significance is made inaccessible to non-mathematicians by textbook authors piling on equations, but it need not be so. I even made a video on explaining "not statisitical significant" (link).

Before you apply a test of statistical significance, you must make sure you’re comparing apples to apples. In the context of air safety, you shouldn’t compare U.S. and foreign airlines on all routes; you should only compare them on routes they share, in which case, you’ll find no statistical difference. One of the bad stats that is circulating during this pandemic is the purported harm of ventilators. This is usually stated as the vast majority of patients put on ventilators eventually die (presumably, relative to those patients not put on ventilators). But only the most serious patients are placed on ventilators. The proper analysis should focus on patients who qualify for ventilators, and compare those who were put on ventilators to those who got other treatments.

We have already seen a host of invalid comparisons. Comparing complete data from years ago with incomplete data this year. Comparing regions that are in different phases of the epidemiological timeline. Comparing countries that have taken different approaches to managing the crisis.

Chapter 3: Item Bank / Risk Pool

This chapter addresses a deeper issue in using statistical averages. An illustrative example is educational testing like the SAT. Critics pointed to the gap in the average scores across racial groups as evidence of bias. But the racial grouping masks a third factor, which is differential ability between students. Due to various socio-economic factors, white students have higher average ability than black students. So the real question is: Among those students with comparable abilities, does the test still show bias toward one racial group? (The answer is not straightforward.)

During the pandemic, the “denier” faction has argued that the mortality rate of Covid-19 is “on a par with” that of seasonal influenza. Are those two rates comparable? The simple answer is no. Comparing mortality rates masks a third factor, which is the level of immunity in the population. Even at the most optimistic, the current level of immunity to the novel coronavirus is less than 20 percent (This statement generously presumes that (a) having antibodies confer immunity without question and (b) the New York City test result is not a scam, and the test is 100% accurate). By contrast, because of vaccination and recurrence, a good chunk of people have immunity to influenza. The mortality rate of Covid-19 should be compared to that of influenza among a population that has no immunity to either, or a population with similar levels of immunity to each.

The other example in this chapter explains how the insurance industry is an outgrowth of the law of large numbers. In any given year, health or auto insurers cannot predict with high accuracy which covered individuals will file claims, but the magic of statistics allows them to predict with great accuracy the aggregate claims. Disaster insurers run into trouble when rare events happen, such as when Class 5 hurricanes hit Florida. Suddenly, potentially all covered individuals tap into the insurance pool all at once. Insurers try to pool together risks of different types and different areas. The global pandemic is a huge threat to the insurance industry. Perhaps there are lawyers reading this, and they might tell me there are natural disaster exclusions. If those don’t apply, we will see a gigantic number of covered entities filing claims all at once.

Florida’s hurricane insurance market was dysfunctional also because the risk is predictable along some parameters. Those living inland realized they were subsidizing the coastal dwellers and wanted out. The coronavirus also creates different risk subgroups. The younger generations have much lower risks than the elderly. When social-distancing and other practices are applied to the whole community, the subgroup with lower risk peels away. A key difference is community transmission – the risk of hurricane damages does not spread by contact.

***

The above covers less than half of the book. Almost everything in those pages has direct relevance to the Covid-19 world. It’s quite heartening to find out ten years later numbers indeed rule our world.

Here’s a link to get your copy of Numbers Rule Your World. There are also translated copies in Chinese (simplified and traditional), Japanese, Korean, and Portuguese, as far as I know.

Deep-dive analysis goes off track: do women earn lower test scores because it's too cold?

A couple of researchers recently claimed to have found evidence of “a battle of the thermostat”: specifically, they argued that women perform significantly worse in a math test when it is administered at lower room temperatures while men’s performance does not decline. They did not find a similar gender gap in a word test.

The analysis strategy is similar to one that is employed by business analysts so readers here might be interested in how it went off track.

First, the researchers looked at the effect of temperature on test scores in aggregate. They found none. See the chart below, which is a replicate of similar charts in the research paper.

Note that in my chart, the scores have been standardized (to be explained below). For both math and word scores, the difference in scores between the extreme temperatures of 15 to 32 celsius (60 to 90 Fahrenheit) is smaller than 0.25 standard units, clearly not statistically significant by any measure.

The next step in the analysis strategy is to rotate through other factors that might influence test scores, such as gender, language spoken, and college major. In the business world, this is often called a deep-dive analysis. The question being asked is whether the effect of temperature on test scores differed by [factor X]. The researchers struck gold when splitting the data by gender. Here are the relevant charts:

In the top chart displaying math scores, it appears that the trend line for men is almost flat while women’s scores progressively increase as temperature increases. According to the usual convention of statistical significance, the effect is strong enough to be published. It is even possible to explain this observation using "battle of the thermostat".

***

Many other researchers have expressed skepticism that the reported effect can be replicated in other settings. See, for example, here.

I explored the data a bit more. Here is the raw data shown in side-by-side boxplots. I have arranged the sessions by the average temperatures, ordered from lowest to highest.

 

The following features caught my attention:

• The math scores of women (top chart, red boxplots) show unusually low variability. The dispersion of these scores is much below that of men’s math scores. The dispersion of these women’s math scores is also much below that of women’s word scores (bottom chart, red boxplots).
• Focusing only on the math scores (top chart), and inspecting the data session by session, from the right to the left, I found that the gender difference is hard to see (often obscured by the high variability of the male scores), except in a few sessions with the lowest temperatures (boxed area).

In terms of math scores, the shape of the distribution for men is clearly different from that for women. The scores for men were generally higher, and contain more extreme values on the high side. If a male and a female student score the same in absolute terms, the scores do not mean the same thing in relative terms – each relative to other students of the same gender.

***

One way to deal with this difference in variability is to standardize the scores by gender. This leads to the following chart, in which the scores are expressed relative to each gender’s distribution.

Notice how the math and word charts look almost identical. Also compare to the chart above using the raw data. This shows the fragility of statistical significance: the top chart shows marginally significant while the bottom chart shows not significant.

***

[Technical note: Here is a scatter plot of the standardized math scores against the raw scores, grouped by gender. You can see that the raw women's scores are in a narrower range, and that men's scores are more disperse and contain some high extremes.]

***

Deep-dive analysis sometimes lead to unexpected discoveries. Aggregate data sometimes mask subgroup differences. But at other times, the subgroup difference is a mirage. At the disaggregated level, the sample size is smaller, and there may also be a difference in variability.

For more discussion of subgroup analysis, see Chapter 3 of Numbers Rule Your World (link), in which I discuss how the insurance industry and the educational testing community tackle this problem.

 

 

 

 

Reading some wild traffic statistics from New Zealand

Only 6% of crashes in New Zealand involve foreign drivers, according to the latest figures provided by the Ministry of Transport.

But in some remote regions of the South Island particularly popular with tourists for their scenery... foreign drivers are involved in about a quarter of all crashes.

These sentences come from a CNN article about a vigilante movement in those regions popular with tourists. The vigilantes snatch car keys from tourists who annoy them by holding up traffic along the scenic routes. 

My friend Tonny saw this article and thought about Numbers Rule Your World. I love to hear stories about how you're able to relate the stories in my book to other real-world situations.

***

The 6% aggregate number hides the effect of tourists on the rate of accidents. The effect of tourists is different depending on which region one is looking at. For this particular article, only the 25% number is relevant--and even this point is not clearcut. I'd like to know whether the vigilante incidents are exclusively occurring in those "remote regions of the South Island", as implied.

The 25% figure does not address the more important question of whether locals or tourists are more likely to get into accidents in those regions. While the tourists accounted for one out of four accidents, they also comprise a large proportion of the traffic. If, say, 50% of the cars on those roads are driven by tourists, then they are disproportionately less likely to be involved in accidents. The base rate of tourist traffic is the missing data.

Further, a margin of error is useful, especially if few accidents occur in those remote areas.

Chapter 1 of Numbers Rule Your World deals with the notion of the statistical average and Chapter 3 investigates when it is appropriate to aggregate data and when it's not. Learn more about the book here. 

 

I don't like 401(k) either

Felix Salmon hates the 401(k), and he explains his reasoning here. His strongest argument is the data, which shows that the first generation of retirees who grew up with these individual retirement savings accounts find themselves with meager retirement savings (average: $120,000, excluding those with zero).

I have always disliked 401(k), and here are some reasons:

I hate the myth of individual control. These accounts (just like health savings accounts and other similar things) are sold to us as an expression of individual freedom and choice. But it is a mirage.

First, it is never a good idea to chop up money into little pockets. The same banks who every day praise the benefit of diversifying our portfolios also lobby the government to establish these individual accounts. If you earn $60,000 a year, and get paid bi-weekly, then your paycheck is $2,500 less tax withholding. If you put 5% of your earnings into the 401(k), then every two weeks, you put away an extra $125. Imagine you then split the $125 into five or ten investments. It just doesn't make much sense.

And this doesn't even consider the per-trade fees that Charles Schwab, or E*Trade, or any of those firms charge us, as a percentage of the expected gains from your trade. Transaction costs are where individuals will always lose to funds.

Second, how many of us can truly beat investment professionals at the investment game? Most of us don't have the time or inclination to be monitoring our portfolio 24/7. The professionals have access to better information. They are able to react faster when something unexpected happens. In fact, our too-big-to-fail banks routinely have zero days of trading losses in a given quarter when they trade on their own accounts. (This does not mean these banks will perform the same when it comes to our money.) I'm aware that most professional fund managers are shown to do no better than some simulated simple-minded strategy. Is there a study that takes the investment performance of the average Joe and compare them to the average fund manager?

Third, moving from a defined benefit plan to a 401(k) is to shift risk from institutions to individuals. When we were sold the myth of individual control, we were also gifted the Trojan Horse of financial risk. Now, we are responsible for our own individual bad decisions. This is an important point. More people in the pool lowers the average risk.

Fourth, the 401(k) is regressive. The lower your salary, the smaller your biweekly contribution. The risk borne by the individual is higher when the principal is smaller.

If those are not enough, individual control is simply a lie, and becoming more so each year. Most of us find that the banks that run the 401(k) only allow us to put our money in one of the funds they administer. They charge a fee to put the money elsewhere. In one case, I was told if I really want to invest in funds not in the approved list, I must first buy one of the designated funds, and then transfer the money to the fund I really want. Needless to say, the in-and-out is not free. There is another company who matches my contribution but in company stock, which is the opposite of diversification; and if I want to transfer the money to a different asset, I have to pay fees.

It's a steep price to pay to enjoy that "individual control."