Big Data, Plainly Spoken (aka Numbers Rule Your World)

Is it worth it to flaunt parking rules?

Significance magazine (link) has a fun article about parking in front of fire hydrants in New York City. It's great data journalism.

The researchers conducted a "field experiment" in which they observed "six routes through Manhattan streets" on July 11, 2022 from 1-5 pm, during which they observed 138 violations. They then merged the hand-collected data with NYC's open data recording all tickets, matching license-plate numbers to learn that only 7 of those drivers have been served tickets.

Based on that, they deduced that most drivers who park in front of hydrants get away with the violation. They computed a detection rate of 5% (= 7/138). During that period of time, the traffic cops issued 423 tickets, and therefore, they deduced that the total violations to be 423/5% = 8,460. This number includes both the 423 tickets and a further 8,037 instances for which no tickets were issued.

They now model the cat-and-mouse game between the rule-bending drivers and the cops as an "exponential game". A timer is started each time some driver parks at a hydrant. The game is to see whether the cop shows up to flag the violation first, or the driver returns to pick up the car first. The model assumes that the time until the next check is exponentially distributed. An exponential distribution is where most values are small, but a few values can be very large. Such a distribution is fully described by its average value, which is estimated to be 6 hours. On average, a cop checks a given hydrant once every 6 hours. (That's the average interval, but each interval has a different value.)

They apply the same exponential model structure for the rule-bending drivers. In this case, the time until the driver returns to a given hydrant - i.e. the duration of violation -  is exponentially distributed, with a mean of 18 minutes (0.3 hours). Comparing the two exponential distributions, we can already see that drivers who leave their cars for a short period of time (say under 60 minutes) have a low chance of getting ticketed but if they park their cars illegally for hours at a time, their risk of detection is much higher. The authors did these calculations: the probability of detection is 16% for one hour, and increases to 99% for 24 hours.

Given the above model - and we assume also that the cops and the drivers act independently, the probability that the cop arrives before the driver returns is the key statistic we care about. That's the chance of the driver receiving a ticket, which as we know from above, is 5% based on the observed data. A consequence of the exponential game model is that this number is the average time until a cop arrives divided by the sum of the average times of a cop arriving and of the driver returning. Thus, 0.05 = 0.3/(0.3+6). [This verifies that the above calculation is correct.]

***

This type of model has been deployed to analyze gambling. If one buys lottery tickets every day, what is the chance that one hits the jackpot before one goes broke?

***

What's left unexplained is how they got the 6-hour estimate for the time until detection. This is a clever bit of work. Remember the researchers matched a subset of the observed violations to actual tickets issued. For these violations, the researchers determined the duration between observing the violation and issuing the ticket to be 6 hours on average.

But... that duration is counted from the moment the researchers found the illegally-parked car, not from the moment the driver parked illegally. What's unknown is the passage of time before the researchers documented the violation.

Conveniently, that unknown duration can be ignored. This convenience comes with using the exponential model. In the exponential world, the future does not depend on the past. Let's say the car has been illegally parked for 10 minutes already. How much time will elapse before a cop shows up? The answer is the same as before: it's exponentially distributed with average of 6 hours.

***

Based on the math above, the researchers found that illegally parking makes sense. The penalty for getting caught is $115. Returning within 5 minutes limits the expected penalty to under $2, which is similar to the cost of feeding a parking meter.

NYT in story-first mode

NYT published a brain-dead article filled with statistical fallacies (link).

Since the Times has people on staff who know statistics, the real explanation for such an article is "story-first thinking". Data mined to support a story line.

***

The headline of the article provokes its least convincing statistic - that only seven black students were offered spots at Stuyvesant, one of the nation's top public high schools famous for STEM, out of 762 offers.

The position of the author is that the above statistic reveals systematic "racial and ethnic disparities". In statistics of education, "disparity" is a descriptive statistic describing a gap, which is different from "discrimination," which suggests foul play. The former is correlation while the latter is causation. This author mischievously used the term disparity while avoiding discrimination, even though the entire article presumes the disparity is unfair.

***

In fact, Stuyvesant (plus seven other similar schools in New York) only considers one factor in admissions, the student's score in the Specialized High Schools Admissions Test (SHSAT), and so the admission decision is not tainted by any judgmental criteria. All test-takers are ranked in order of test score, and then, each student is allocated a spot in their most preferred school - if all slots in the school have been filled, then the next preferred school is considered, and so on.

This allocation method guarantees "fairness" in the sense that if a student isn't allocated to a school higher on his/her preference list, it's because every offer for that school went to students who have scored higher on the SHSAT.

***

If we are interested in the question of whether the allocation of offers is fair across ethnic groups, the right question is: among students of similar ability (for example, score bands), do black or Latino students have a lower offer rate than whites and Asians?

This question of fairness pops in everywhere in education. In Chapter 3 of Numbers Rule Your World (link), I examined it in the context of designing fair SAT questions.

Surprise, surprise. Such data have not been released. Since there are no data, I'll explain conceptually how the numbers work.

***

Let's assume 30% of test-takers are Asians, and 20% each are blacks, latinos, and whites, and 10% others. Assume 50% of offers are given to Asians, 30% to whites, 5% each to blacks and latinos, and 10% to others. Overall, 15% of the test-takers receive an offer.

Given those assumptions, the offer rate for Asians is 25%, for whites it's 23%, for blacks and Latinos, it's 4%. Asians and whites are 1.5-1.7x more likely than the average to get an offer, while the offer rates of blacks and Latinos are a quarter of the average.

The NYT claims this proves that the process unfairly favors Asian and white students at the detriment of blacks and Latinos.

Not so fast. Remember all offers are completely determined by test scores, without any judgmental factors. So the proportion of students who receive offers is the same as the proportion who score above the cutoff score. Think of the cutoff score as the score of the last student taken up by a school. This cutoff score is an outcome of the offer process, not an input.

***

We can now split each subgroup into those who score above the cutoff and those below. Among Asians, the split is 25%-75%; whites, 23%-77%; blacks, 4%-96%; Latinos, 4%-96%; others, 15%-85%.

The reason why Asians and whites have more offers is because they have higher SHSAT scores.

Let's now insist that the offer rate for every ethnic group must equal the overall rate of 15%. For Asians and whites with a higher score distribution, this necessarily means that we cannot accept everyone above the cutoff score. It turns out that we can only accept about 60-67% of those who score above the cutoff score, and 0% of those who score below, in order to get to a weighted average of 15%.

For blacks and Latinos, we'd take 100% of those who score above the cutoff, but that isn't enough to get to 15%. We also must take 12% of those who score below the cutoff.

This is the unspoken math behind the NYT article. They are proposing: rejecting 40% of high-scoring Asians and whites while accepting 100% of high-scoring blacks and Latinos; and accepting 12% of low-scoring blacks and Latinos while rejecting 100% of low-scoring Asians and whites.

Shame.

 

 

It's random because we say so

Some of you are wondering why I haven't written about Covid-19 studies recently. Lamentably, the quality of such studies has not improved. It may have deteriorated even more. I wrestle with whether it's a waste of time to read them. The problems with many studies are apparent from the first page: how do they pass the peer review process?

This post concerns a half page of a recent jourmal publication called "Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data" (link). It's only half of one page because there are already enough problems with what I read that I don't see the point of continuing.

A friend sent me this paper because the authors share my concern about significant misrepresentation of vaccine effectiveness in observational studies of Covid-19 vaccines. (Here is a post about my paper published last month with two co-authors that highlighted three common biases that are not properly handled in studies.)

My comments are limited to the section with the header "Sequential Target Trial Emulation" on page 2 of the paper. In this section, the authors describe the first of two proposed analytical devices for dealing with biases in observational data.

For any observational study, there must be a well-defined starting and ending date of study enrollment. For each day in this enrollment window (and for each region), they pretend that a new target trial is being launched. This is how they put it:

on each day and in each region, eligible persons who have not yet been vaccinated are randomly assigned to receive immediate vaccination or to remain unvaccinated throughout follow-up and are then followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first

They then suggest that each daily trial can be analyzed on its own, and then the results averaged to obtain an overall effectiveness. They claim this outcome is representative of the "intention to treat" effect in a hypothetical randomized clinical trial. That's a bold claim! (Not even Pfizer, Moderna, etc. claimed to have produced ITT vaccine efficacies even in their RCTs.)

I'm not making this up. They literally claimed that "The trial-specific estimates of the effect of vaccination from each sequential trial can then be combined to estimate an observational analogue of the intention-to-treat effect of assignment to the intervention."

***

Why am I shaking my head violently?

Their statement that people are "randomly assigned to receive immediate vaccination or to remain unvaccinated" is unserious.

We are talking about observational data - data analysts poking around in a database after the fact. It doesn't matter what analytical technique is used - it remains the case that the vaccinated people self-selected the treatment. They were not randomly assigned to receive immediate vaccination.

In fact, the authors contradicted themselves in the very next sentence when they explained (my bolding):

Each calendar day is considered as time zero for a new emulated trial, with persons who are determined to be eligible assigned to the vaccination group if they were vaccinated on that day or to the no-vaccination group if they were not vaccinated,

For the vaccinated, there is no "assignment", let alone random assignment. Whoever decided to get vaccinated lands passively in the treatment group for that day. Because the vaccinated are self-selected, the unvaccinated are self-selected as well. Whatever selection biases exist in the raw data persists in this ex-post experimental setup. This step absolutely does not support presumption of random treatment assignment.

If this procedure resulted in random treatment assignment, then the original dataset contains no biases, which annuls the central premise of the entire paper.

***

Of course, they aren't taking every unvaccinated person. So the data analyst actively selects who comprise the unvaccinated group. The authors give two mechanisms for selecting unvaccinated.

The first is random selection. I had to take a deep breath before reading on. I doubled back to see if I read this correctly. "One or more persons can be selected at random from all eligible persons who were unvaccinated on that day and in that region, with exclusion of unvaccinated persons selected for a comparison made on a previous day."

If they randomly selected unvaccinated, then the random sample of unvaccinated would have the same statistical composition as the population of unvaccinated. This ensures no correction for any kind of biases. And yet at the end of this process, they claim to have found the ITT effect as if a randomized clinical trial were run!

The second mechanism for selecting unvaccinated is by matching on key "characteristics that reflect Covid-19 risk". This method is definitely more acceptable as it at least attempts to make the two treatment groups more comparable.

In practice, the matching method faces several big challenges (many of which I have discussed before on this blog):

• the matchable population is a small proportion of the total population, e.g. most studies using matching heavily undersamples older people and oversamples younger people. The method is guaranteed to undersample demographic segments in which vaccination rate is high, and oversample demographic segments in which vaccination rate is low
  
  
• VE in the matched population may be different from VE in the general population. The estimated VE in the matched population is surely not the ITT for the entire population
  
  
• most important confounders are unmeasured (e.g. taking other mitigation measures, vaccine hesitancy, exposure to virus). Matching does not infer balance on non-matching variables, a key reason why ex-post matching on observational data cannot replace randomized treatment assignment
  
  
• most countries enforce highly targeted vaccination programs which generate irreducible biases, e.g. if older people are encouraged to get vaccinated first, and have done so in large numbers (say 80%), what's the chance that the 20% unvaccinated older people are comparable to the other 80% vaccinated older people?

***

There is another aspect of the observational studies that are buried away from the summary of the methodology. In the sentences cited above, they said that the study subjects were "followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first." That surely cannot be true!

Another event that happens is vaccination. For people in the unvaccinated group, by definition, they have not been vaccinated up to the day of enrollment; however, many or most of them would get the vaccine during the follow-up period. How did the data analyst deal with this event? We are not given a hint in this methodology section.

Upon vaccination, clearly the follow-up period for the unvaccinated person must end. We simply don't know the counterfactual: what would have happened if the person did not get vaccinated? If we follow the time series of vaccine efficacy from day of vaccination forward, more and more people in the unvaccinated group are dropped from follow-up because they got themselves vaccinated. The people who stay in the unvaccinated group for a long time are the vaccine holdouts, who are clearly different from the general population, e.g. they are much less likely to take other precautions such as wearing masks, distancing, avoiding crowds, etc. How well does matching on demographics and basic health data control for these differences?

If those differences are not adequately accounted for, then the right side of the VE curve cannot be trusted.

***

Also, note that variability is artificially suppressed in this type of study. Anyone who got vaccinated during the study period appear once in the vaccinated group on the day of vaccination, but also may appear in the unvaccinated group on any day prior to vaccination. Unlike a typical RCT in which there are no overlaps in the group membership, in this type of observational studies, the majority of the people show up in both groups. So, if we subsequently run regression analysis comparing the two groups, adjusting for demographics and other factors, the variance between groups is markedly under-stated!

 

 

Residual problems of intelligence

Modern data science/machine learning/AI has achieved much through efficiently performing certain tasks to an acceptable level of quality. It remains to be seen whether there are means to tackle the residual problems, or perhaps, whether there is the will to do so.

I ran into a couple of problems while using Yelp.

First, I sorted the restaurant reviews by date, and the first two reviews that popped up were these:

These two reviews are almost identical. If we take them at face value, these are two people, one from Midtown West and the other from Staten Island, who both walked in for lunch two days apart (assuming they wrote the review on the same day of their respective meals), and managed to grab one of the last tables. They both ordered steak frites and loved it, especially the fries. And, they both decided to come back for breakfast in the future. They also learned English composition from the same teacher. What a coincidence!

Yelp has long had implemented AI that supposedly detects fake reviews but apparently something that is so obvious is missed by the algorithm.

There's a lot to unpack here but one key immutable fact is that all review sites attract fake reviews because good reviews are great marketing for businesses. Review sites that stringently authenticate reviewers will have much lower volumes of reviews, which creates a negative network effect. Authentication does not solve the root cause anyway, since businesses hire real people to write fake reviews.

***

The next situation I ran into isn't really about Yelp but about Google Maps. Within Yelp, I clicked on the address to see the location of a restaurant. I read in one of the reviews that this restaurant is inside a hotel. In order to confirm that, I entered the hotel name and asked for directions inside Google Maps.

Here is the recommended route:

It literally says walk out of the building (the hotel), make a circle around the block, then re-enter the building (the restaurant). It doesn't realize that the beginning and ending addresses are identical.

***

It's surprising that these issues persist so many years after these services were launched. Make no mistake - most users have most of their problems solved using Yelp and Google Maps. What's the incentive to solve the residual problems? That's what's on my mind today.