Big Data, Plainly Spoken (aka Numbers Rule Your World)

NYT in story-first mode

NYT published a brain-dead article filled with statistical fallacies (link).

Since the Times has people on staff who know statistics, the real explanation for such an article is "story-first thinking". Data mined to support a story line.

***

The headline of the article provokes its least convincing statistic - that only seven black students were offered spots at Stuyvesant, one of the nation's top public high schools famous for STEM, out of 762 offers.

The position of the author is that the above statistic reveals systematic "racial and ethnic disparities". In statistics of education, "disparity" is a descriptive statistic describing a gap, which is different from "discrimination," which suggests foul play. The former is correlation while the latter is causation. This author mischievously used the term disparity while avoiding discrimination, even though the entire article presumes the disparity is unfair.

***

In fact, Stuyvesant (plus seven other similar schools in New York) only considers one factor in admissions, the student's score in the Specialized High Schools Admissions Test (SHSAT), and so the admission decision is not tainted by any judgmental criteria. All test-takers are ranked in order of test score, and then, each student is allocated a spot in their most preferred school - if all slots in the school have been filled, then the next preferred school is considered, and so on.

This allocation method guarantees "fairness" in the sense that if a student isn't allocated to a school higher on his/her preference list, it's because every offer for that school went to students who have scored higher on the SHSAT.

***

If we are interested in the question of whether the allocation of offers is fair across ethnic groups, the right question is: among students of similar ability (for example, score bands), do black or Latino students have a lower offer rate than whites and Asians?

This question of fairness pops in everywhere in education. In Chapter 3 of Numbers Rule Your World (link), I examined it in the context of designing fair SAT questions.

Surprise, surprise. Such data have not been released. Since there are no data, I'll explain conceptually how the numbers work.

***

Let's assume 30% of test-takers are Asians, and 20% each are blacks, latinos, and whites, and 10% others. Assume 50% of offers are given to Asians, 30% to whites, 5% each to blacks and latinos, and 10% to others. Overall, 15% of the test-takers receive an offer.

Given those assumptions, the offer rate for Asians is 25%, for whites it's 23%, for blacks and Latinos, it's 4%. Asians and whites are 1.5-1.7x more likely than the average to get an offer, while the offer rates of blacks and Latinos are a quarter of the average.

The NYT claims this proves that the process unfairly favors Asian and white students at the detriment of blacks and Latinos.

Not so fast. Remember all offers are completely determined by test scores, without any judgmental factors. So the proportion of students who receive offers is the same as the proportion who score above the cutoff score. Think of the cutoff score as the score of the last student taken up by a school. This cutoff score is an outcome of the offer process, not an input.

***

We can now split each subgroup into those who score above the cutoff and those below. Among Asians, the split is 25%-75%; whites, 23%-77%; blacks, 4%-96%; Latinos, 4%-96%; others, 15%-85%.

The reason why Asians and whites have more offers is because they have higher SHSAT scores.

Let's now insist that the offer rate for every ethnic group must equal the overall rate of 15%. For Asians and whites with a higher score distribution, this necessarily means that we cannot accept everyone above the cutoff score. It turns out that we can only accept about 60-67% of those who score above the cutoff score, and 0% of those who score below, in order to get to a weighted average of 15%.

For blacks and Latinos, we'd take 100% of those who score above the cutoff, but that isn't enough to get to 15%. We also must take 12% of those who score below the cutoff.

This is the unspoken math behind the NYT article. They are proposing: rejecting 40% of high-scoring Asians and whites while accepting 100% of high-scoring blacks and Latinos; and accepting 12% of low-scoring blacks and Latinos while rejecting 100% of low-scoring Asians and whites.

Shame.

 

 

It's random because we say so

Some of you are wondering why I haven't written about Covid-19 studies recently. Lamentably, the quality of such studies has not improved. It may have deteriorated even more. I wrestle with whether it's a waste of time to read them. The problems with many studies are apparent from the first page: how do they pass the peer review process?

This post concerns a half page of a recent jourmal publication called "Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data" (link). It's only half of one page because there are already enough problems with what I read that I don't see the point of continuing.

A friend sent me this paper because the authors share my concern about significant misrepresentation of vaccine effectiveness in observational studies of Covid-19 vaccines. (Here is a post about my paper published last month with two co-authors that highlighted three common biases that are not properly handled in studies.)

My comments are limited to the section with the header "Sequential Target Trial Emulation" on page 2 of the paper. In this section, the authors describe the first of two proposed analytical devices for dealing with biases in observational data.

For any observational study, there must be a well-defined starting and ending date of study enrollment. For each day in this enrollment window (and for each region), they pretend that a new target trial is being launched. This is how they put it:

on each day and in each region, eligible persons who have not yet been vaccinated are randomly assigned to receive immediate vaccination or to remain unvaccinated throughout follow-up and are then followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first

They then suggest that each daily trial can be analyzed on its own, and then the results averaged to obtain an overall effectiveness. They claim this outcome is representative of the "intention to treat" effect in a hypothetical randomized clinical trial. That's a bold claim! (Not even Pfizer, Moderna, etc. claimed to have produced ITT vaccine efficacies even in their RCTs.)

I'm not making this up. They literally claimed that "The trial-specific estimates of the effect of vaccination from each sequential trial can then be combined to estimate an observational analogue of the intention-to-treat effect of assignment to the intervention."

***

Why am I shaking my head violently?

Their statement that people are "randomly assigned to receive immediate vaccination or to remain unvaccinated" is unserious.

We are talking about observational data - data analysts poking around in a database after the fact. It doesn't matter what analytical technique is used - it remains the case that the vaccinated people self-selected the treatment. They were not randomly assigned to receive immediate vaccination.

In fact, the authors contradicted themselves in the very next sentence when they explained (my bolding):

Each calendar day is considered as time zero for a new emulated trial, with persons who are determined to be eligible assigned to the vaccination group if they were vaccinated on that day or to the no-vaccination group if they were not vaccinated,

For the vaccinated, there is no "assignment", let alone random assignment. Whoever decided to get vaccinated lands passively in the treatment group for that day. Because the vaccinated are self-selected, the unvaccinated are self-selected as well. Whatever selection biases exist in the raw data persists in this ex-post experimental setup. This step absolutely does not support presumption of random treatment assignment.

If this procedure resulted in random treatment assignment, then the original dataset contains no biases, which annuls the central premise of the entire paper.

***

Of course, they aren't taking every unvaccinated person. So the data analyst actively selects who comprise the unvaccinated group. The authors give two mechanisms for selecting unvaccinated.

The first is random selection. I had to take a deep breath before reading on. I doubled back to see if I read this correctly. "One or more persons can be selected at random from all eligible persons who were unvaccinated on that day and in that region, with exclusion of unvaccinated persons selected for a comparison made on a previous day."

If they randomly selected unvaccinated, then the random sample of unvaccinated would have the same statistical composition as the population of unvaccinated. This ensures no correction for any kind of biases. And yet at the end of this process, they claim to have found the ITT effect as if a randomized clinical trial were run!

The second mechanism for selecting unvaccinated is by matching on key "characteristics that reflect Covid-19 risk". This method is definitely more acceptable as it at least attempts to make the two treatment groups more comparable.

In practice, the matching method faces several big challenges (many of which I have discussed before on this blog):

• the matchable population is a small proportion of the total population, e.g. most studies using matching heavily undersamples older people and oversamples younger people. The method is guaranteed to undersample demographic segments in which vaccination rate is high, and oversample demographic segments in which vaccination rate is low
  
  
• VE in the matched population may be different from VE in the general population. The estimated VE in the matched population is surely not the ITT for the entire population
  
  
• most important confounders are unmeasured (e.g. taking other mitigation measures, vaccine hesitancy, exposure to virus). Matching does not infer balance on non-matching variables, a key reason why ex-post matching on observational data cannot replace randomized treatment assignment
  
  
• most countries enforce highly targeted vaccination programs which generate irreducible biases, e.g. if older people are encouraged to get vaccinated first, and have done so in large numbers (say 80%), what's the chance that the 20% unvaccinated older people are comparable to the other 80% vaccinated older people?

***

There is another aspect of the observational studies that are buried away from the summary of the methodology. In the sentences cited above, they said that the study subjects were "followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first." That surely cannot be true!

Another event that happens is vaccination. For people in the unvaccinated group, by definition, they have not been vaccinated up to the day of enrollment; however, many or most of them would get the vaccine during the follow-up period. How did the data analyst deal with this event? We are not given a hint in this methodology section.

Upon vaccination, clearly the follow-up period for the unvaccinated person must end. We simply don't know the counterfactual: what would have happened if the person did not get vaccinated? If we follow the time series of vaccine efficacy from day of vaccination forward, more and more people in the unvaccinated group are dropped from follow-up because they got themselves vaccinated. The people who stay in the unvaccinated group for a long time are the vaccine holdouts, who are clearly different from the general population, e.g. they are much less likely to take other precautions such as wearing masks, distancing, avoiding crowds, etc. How well does matching on demographics and basic health data control for these differences?

If those differences are not adequately accounted for, then the right side of the VE curve cannot be trusted.

***

Also, note that variability is artificially suppressed in this type of study. Anyone who got vaccinated during the study period appear once in the vaccinated group on the day of vaccination, but also may appear in the unvaccinated group on any day prior to vaccination. Unlike a typical RCT in which there are no overlaps in the group membership, in this type of observational studies, the majority of the people show up in both groups. So, if we subsequently run regression analysis comparing the two groups, adjusting for demographics and other factors, the variance between groups is markedly under-stated!

 

 

Residual problems of intelligence

Modern data science/machine learning/AI has achieved much through efficiently performing certain tasks to an acceptable level of quality. It remains to be seen whether there are means to tackle the residual problems, or perhaps, whether there is the will to do so.

I ran into a couple of problems while using Yelp.

First, I sorted the restaurant reviews by date, and the first two reviews that popped up were these:

These two reviews are almost identical. If we take them at face value, these are two people, one from Midtown West and the other from Staten Island, who both walked in for lunch two days apart (assuming they wrote the review on the same day of their respective meals), and managed to grab one of the last tables. They both ordered steak frites and loved it, especially the fries. And, they both decided to come back for breakfast in the future. They also learned English composition from the same teacher. What a coincidence!

Yelp has long had implemented AI that supposedly detects fake reviews but apparently something that is so obvious is missed by the algorithm.

There's a lot to unpack here but one key immutable fact is that all review sites attract fake reviews because good reviews are great marketing for businesses. Review sites that stringently authenticate reviewers will have much lower volumes of reviews, which creates a negative network effect. Authentication does not solve the root cause anyway, since businesses hire real people to write fake reviews.

***

The next situation I ran into isn't really about Yelp but about Google Maps. Within Yelp, I clicked on the address to see the location of a restaurant. I read in one of the reviews that this restaurant is inside a hotel. In order to confirm that, I entered the hotel name and asked for directions inside Google Maps.

Here is the recommended route:

It literally says walk out of the building (the hotel), make a circle around the block, then re-enter the building (the restaurant). It doesn't realize that the beginning and ending addresses are identical.

***

It's surprising that these issues persist so many years after these services were launched. Make no mistake - most users have most of their problems solved using Yelp and Google Maps. What's the incentive to solve the residual problems? That's what's on my mind today.

Game of telephone for data messaging

We all know about the game of telephone. If one person says something to another, and the second person to a third, and so on, the message received at the end of the chain will be substantively different. What gets me thinking about this today is an article by Gelman and friends, which essentially describes the game of telephone as it applies to data messaging: "Through endless repetition, numbers of dubious origin take on the veneer of scientific fact, in many cases in the context of vital public-policy debates."

I love their leading example. Apparently, (at least one) university in the U.S. circulated a public-service notice telling people to wear a hat on cold days because "most body heat is lost through the top of the head." The authors attempted to trace the origin of this message, and discovered that the advice purportedly came from a statistic, which claims that 50 or even 80 percent of body heat is lost through the (apparently cap-less) head.

[Most of the article is about tracing statistics on "corruption", and about the concern that politicians mislead the public using junk statistics.]

***

This little scenario reveals a number of things about our relationship with data.

First, repeating data seems to follow what Kahneman calls "fast thinking", not "slow thinking". People tend not to slow down, and apply logic, and ask: could we really be losing 80 percent of body heat through our head? Could a human being survive after losing 80 percent of its body heat? (Apparently, the more correct answer is 10 percent.)

Second, because of the fad built upon "unconventional wisdom," spread by the likes of Malcolm Gladwell and Steven Levitt, people may believe illogical data -- the more bizarre, the better. Shock value is the currency of social media, so this fad lives.

Third, I'm interested not just in the likelihood that the message has changed through repetition, but in what direction the message shifts. My working assumption is that the default setting of most people is story-first (as opposed to data-first). The way data are consumed is typically to support some hypothesis, i.e. preconceived notion. In the game of telephone, the message is continuously shaped by the story-first mentality of the messenger.

This idea is an expanded version of what the authors called "decorative statistics", i.e. making numbers "sound big and impressive".

Each person receives a message, but only partially due to memory failure. Each person has its own belief, such as about the effect of not wearing hats on cold days. The outgoing message will tend to be pulled toward those prior beliefs.

When I find some time, I should run a simulation using Bayes' Rule to propagate messages at each stage and explore the above idea more rigorously.

***

Another factor enables this game of telephone in data stories. Every statistic ever reported has nuances - it is only true for specific populations, after certain exclusions, accounting for particular biases, etc. However, in messaging data, we don't have the luxury of attaching two paragraphs of explanations for each number. Therefore, the nuances are lost.

The orphaned data are out in the open. This invites anyone to adopt these statistics and shape them to their own liking. Fill in the blanks. If one is completely rigorous, one plugs the gaps by researching the origins of the data, but who has the time, right? So, instead, one just adds whatever one thinks "makes sense"... this is where we ends up in a story-first mode, because making sense is in the context of that preconceived story. If we want people to wear hats, we like the story that hats keep heat contained in the body, and if we can't quite remember the statistic, we are susceptible to guesstimating a number that supports our story.