Big Data, Plainly Spoken (aka Numbers Rule Your World)

With adjustments comes responsibility

Yesterday, I laid out the framework for today's post, in which I compare intent-to-treat (ITT) analysis and per-protocol (PP) analysis of the colonoscopy trial. All the data I cite below are pulled from this paper and its supplement (link).

***

The outcome metric of the clinical trial is the risk of colon cancer (diagnosis) over a 10-year follow-up window (a final analysis will be performed after 15 years). Risk is based on the relative ratio of diagnosis rates between the treatment and usual-care (control) group. In the ITT analysis, the treatment group consists of everyone who's been randomized to receive invitations to colonoscopy while in the PP analysis, the treatment group consists of the "compliers" only, those who responded to the invitation and underwent colonoscopy. Because compliers self-select to accept screening, they cannot be treated as a random subset of the invited group, and thus, the researchers spun up a regression model used to "remove biases".

In the chart below, I present the results of the ITT analysis from the paper, which is pre-specified for this trial. The trial was run in three countries but Poland (64%) and Norway (31%) contributed 95% of the participants.

 

Immediately, two features of this data come to life. Firstly, the case rate in Poland (0.88%) is much lower than that in Norway (1.35%). The relative risk ratio of 35% is higher than the relative ratio between the treatment and control groups (18%). (In other words, variability between countries is larger than variability between treatment and control.) Secondly, the observed reduction in case rate among Norwegian participants was 24% while that in Poland was 16%. Since Poland comprises almost 61% of the overall population, the reported aggregate effectiveness is just north of Poland's value.

The next chart shows what happens when the researchers switched to an "adjusted per-protocol" analysis. They broke up the invited group into compliers and non-compliers. The paper reported diagnosis metrics for Poland, Norway and the aggregate. (They did not disclose mortality metrics, only reporting the relative ratio, and thus no one can validate that part of the analysis. Nor did they break out Sweden's data.)

In Norway, the paper printed the compliers' case rate to be 0.86%, which is 30% below the average of the invited group. (It is not clear from the paper how much of this shift is due to adjustment and how much is due to filtering out noncompliers. It's not even clear whether this rate is computed from data or estimated from a model.) In any case, the treatment group's case rate was marked down from 1.2% to 0.86%, and correspondingly, the effectiveness metric in Norway leaped from 24% to 45%!

In Poland, a completely different picture emerges. The compliers are said to have a case rate of 0.85%, almost the same as the 0.83% aggregate rate of the invited group. This means that the effectiveness metric in Poland has barely shifted compared to the ITT analysis. It's still about 15%.

From this, we learn that the change in analysis strategy embeds the following key assumption:

(A) Compliers are very different from noncompliers in Norway but not in Poland

Such an assumption might be justifiable; I am in no position to judge. Hold on tight, as there are more assumptions to be unearthed.

Not explicitly stated in the paper is the fact that the case rate for non-compliers must have been altered in the other direction: since in Norway, compliers form 40% of the invited group, and noncompliers 60%, we can back out the case rate of the non-compliers - implied by the adjusted per-protocol analysis - which is 1.77%. This observation reveals two other embedded assumptions:

(B) The diagnosis rate of colon cancer for Norwegians who refused to be screened when invited  (1.77%) is higher than that for people not invited to screening and also not screened (1.57%). This requires explanation since people were randomly selected to receive invitiations

(C) The implied gap between the case rates of the compliers and noncompliers in Norway is a whopping ~1%, which is very wide

 

In Poland, two-thirds of those invited did not comply. Using this information, we can back out the case rate of the noncompliers, and it's 0.82%, actually slightly below the aggregate value of 0.83% for the entire invited group. This leads to further observations:

(D) There is almost no difference in diagnosis rates in Poland between those who complied and those who didn't comply with colonoscopy screening. This is particularly jarring considering (C) above.

(E) Colonoscopy is 3 times more effective in Norway than in Poland. In Norway, it brought the case rate down from 1.57% (usual care) to 0.86% (compliers) which is about the same level as compliers in Poland - despite the control group in Poland coming in much lower at 0.99% than the control in Norway.

***

The researchers decided to do an "adjusted per-protocol analysis" which means they did two things: they filtered noncompliers from the treatment group, and they made adjustments using a regression model that includes variables like country of origin, age, etc.

Since the paper does not disclose the model details, we have to look at the model outputs to infer what happened. What happened is listed above as (A)-(E). These are all changes to the data that ultimately caused the estimated effectiveness to shift from 18% to 31%.

***

What I want to show is that current reporting practices for observational studies in medical journals is wholly inadequate. It should be much easier for readers to comprehend what adjustment models are doing. Much more space should be devoted to justifying the effects of adjustments on the conclusions. Any of the above assumptions can make sense, but there is nothing in the paper to back them up.

The models used to adjust data should be disclosed in full. Tables showing covariate balance between groups should be standard, for every pair of subgroups that feature in any of the metrics included in the publication. For example, age is clearly an important confounder in this study, since colon cancer tends to appear in older people but there is nothing in the paper that shows age distribution between the compliers subset and the usual-care group.

Why no serious researchers conduct "per protocol" analyses

All observational data must be adjusted in order to correct biases. But our science teems with irresponsible statistical adjustments - throwing a bunch of variables into a regression model, and declaring that all known biases have been satisfactorily resolved is all too common.

In the medical literature, publishing one's adjustment model is rarely done. I don't trust any adjusted models unless the researcher discloses the equations, the coefficients and goodness of fit statistics. Even better, the researcher should compare the adjusted to unadjusted results, note how the adjustments affected the outcome, and explain why these adjustments make sense.

Handwaving about the structure of the adjustment model, referencing other papers that supposedly use similar methods, etc. suggest that the researchers lack confidence in their methods.

In this blog post, I demonstrate how to evaluate statistical adjustments, using the recently released, highly controversial results from the colonoscopy clinical trial. (See my related writeup here.) It took a lot of work to put this together, because the disclosure of details in the paper is scarce (this is the norm rather than the exception, unfortunately.)

***

The colonoscopy trial was the first randomized clinical trial (RCT) conducted to measure the effect of colonoscopy screening on risks of diagnoses or deaths from colon cancer. The trial enrolled two groups of participants: those invited to screening, and the "usual care" group which means no screening (as is the practice in Poland, Norway and Sweden, where the trial took place).

For this post, I focus only on risk of colon cancer diagnoses (i.e. cases). That's because the disclosure on mortality outcomes is even worse, making it impossible to judge the adjustment procedure. The trial specifies an intent-to-screen analysis (more generally known as intent-to-treat, ITT), The ITT analysis shows that 0.93% of the invited group were diagnosed during the 10-year follow-up window, compared to 1.13% of the usual-care group. This result is described as a 18% reduction in risk of colon cancer, even though the difference in diagnosis rate is only 0.2%, that is to say, out of 1,000 people invited to colonoscopy, only two fewer cases will be detected relative to usual care.

As discussed in the previous blog, the medical profession was none too happy with this outcome because doctors, esp. American ones, have convinced themselves that colonoscopy could reduce both cases and deaths by 70% or more. Neither did the authors of the paper disclosing the trial results - in the section of the paper in which they discussed the findings, they did not mention the primary endpoint, but only talked about an "adjusted per-protocol" analysis, which moved the reported result from 18% reduction in risk to 31%. Still far from the 70% previously believed, but deemed more palatable.

The media then mis-reported this analysis as "per-protocol" analysis, dropping the all-important word "adjusted". By the end of this post, you'll understand why I say this.

***

Intent-to-treat vs per-protocol is a long-running debate but any serious researcher goes with the former, which is exactly what the designers of the colonoscopy trial did. Let's look at this debate using a marketing example.

Let's say Target is running a Black Friday promotion in which online customers get an extra 30% discount if they type the code SAVE30 when checking out. In order to measure the effectiveness of this promotion, Target data scientists design an A/B test in which shoppers are randomly divided into the exposed group and the non-exposed group upon visiting Target's website. They intend to analyze the results of this experiment by looking at metrics such as proportion of visitors who made a purchase, and average purchase amount, comparing the two groups. This analysis is known as an intent-to-treat (ITT) analysis.

Ouch! The ITT analysis showed that the additional sales generated by exposure to the SAVE30 promotion failed to cover the cost of giving out these extra discounts. Inevitably, there will be at least one business manager who complains that the wrong analysis was done. They argue that we should not include people who did not notice the promotion. Being exposed to the promotion just means Target shows notifications while the customers browse around but some fraction of customers did not pay attention. So, what do they think is the right analysis? We should only analyze those people for whom we know for sure they noticed the promotion - in other words, we only analyze those people who entered the promotion code on checkout (or other variants, such as only those who clicked on a promotion banner). This style of analysis is "per protocol".

Why would all serious researchers shy away from per-protocol analyses? In this example, everyone in the exposed group who subsequently entered the promotion code on checkout is someone who is surely, or almost surely, making a purchase. In the non-exposed group, there is no such thing as entering the promotion code, thus we effectively narrowed the exposed group to almost sure shoppers against the unexposed group that contains the average online shopper. Which group will show better metrics? Will the better results on the exposed group be due to the promotion or inherent differences in the group composition?

***

Back to the colonoscopy trial. In the per-protocol analysis, the data analyst threw out roughly half of the invited-to-screen group because they did not "comply". The reasoning is that they could not have benefited from the screening (read: SAVE30) and thus, they drag down the efficacy. But the "compliers" typically have different characteristics than the "non-compliers", and so per-protocol destroys the randomization aspect of the "randomized" controlled trial. In other words, the complier subset is different from the usual-care group not just in being screened but in unknowable dimensions. We've turned the RCT into an observational study.

People who advocate PP analysis say not to worry - they will throw a bunch of demographic variables into a regression model, and all biases will disappear. I find this quite fascinating. If such adjustment strategies indeed can cure all biases, then why would the community embrace RCTs as the gold standard? It would appear that observational data blended in a mixer known as regression modeling is equally as good as a randomized experiment!

In fact, the colonoscopy controversy clearly shows the opposite. Prior observational studies have consistently shown extremely positive results of 70% or so, and the first time a research team conducts an RCT, the outcome is below 20%. If anything, this proves that carefully analyzed observational studies cannot replace the gold-standard RCT.

***

Even if all biases are cured, and we are able to believe the PP analysis, the real-world results are guaranteed to disappoint.

Let's go back to the Target example. Let's say the PP analysis showed that customers who used the SAVE30 promotion were 80% more likely to complete the transaction than the customers who were not shown the promotion. So now, the head of marketing tells everyone if all shoppers are shown the promotion, then the experiment proves that they should see an 80% jump in number of transactions.

Of course, reality bites. The problem is that the 80% incremental sales apply only to the subset who used the promotion code. If only 10% of those exposed used the code during the A/B test, then only about 10% would contribute an 80% jump in sales. The other 90% are exposed but would not be using the code.

If half the trial population elect not to take the colonoscopy screening during the trial, it is likely the case that half the population would make this decision if colonoscopy was rolled out to all. The 30% reduction in colon cancer risk would still only apply to the subset choosing to be screened, and would not make a dent on those who chose to forego it.

***

It looks like I have to defer showing you my analysis till the next post. The plan is to compare the ITT analysis and the "adjusted PP analysis", pointing out how the adjustments affected the outcomes. Come back tomorrow.

(By the way, all Covid-19 vaccine trials were per-protocol analyses as well, even though such analyses are problematic.)

 

 

 

Gelman's Museum of Scholarly Misconduct

Today's short post is a tour of the imaginary Museum of Scholarly Misconduct that Andrew Gelman proposed here. If you've been reading his blog in the last few years, he's been dogged in his campaign to expose research fraud. Sadly, there is a lot more research fraud than we think. Academics are capable of doing the most questionable things. It's not just the junior professors trying to make a name; the senior, most decorated stars of the field have been tarred. (I don't have room in this post to discuss the other side of the issue, which is the closing of ranks when such frauds are exposed, and the complete lack of accountability. Also, how "peer review" fails repeatedly.)

Along the way, I'll tell you how I got hoodwinked too by a study that has now attracted infamy.

***

Our first objet d'art is "Shredder with the missing teeth" by Dan Ariely, Duke professor (formerly MIT). Readers of this blog probably know Prof. Ariely through his series of best-sellers, starting with Predictably Irrational and the Honest Truth about Dishonesty.

This shredder featured in one of his key studies on human dishonesty. He described the experiment in this interview (my bolding):

At the beginning of 2002, we started a series of studies called The Matrix Experiments. We gave people 20 simple math problems: each one was a matrix of numbers where people had to find two numbers that added up to ten. It was a simple enough exercise that anyone could do, but we didn’t give them enough time. At the end of five minutes, people had to put their pencils down and write on another piece of paper how many they solved correctly. They then put the original test paper in the shredder, so nobody would know the true number they had solved. They received $1 for each problem they claimed to have solved correctly.

What they didn't know is that we modified the shredder! We only shredded the sides of the page, whereas the body of the page remained intact. Over 40,000 people, from all walks of life, participated in The Matrix Experiments.

I decided to quote the entire two paragraphs and give him all the space to explain the experiment. It sounds like an ingenious experiment. If I had heard it for the first time, I'd have thought what a smart design.

Some years later, another researcher attempted to replicate the experiment. He and his graduate students simply couldn't find a way to "modify the shredder". They even enlisted a mechanical engineer and still they failed. They asked Ariely how he did it. Ariely left them a voice mail (which you can hear yourself here), in which he said get a regular shredder from Home Depot, and take out the teeth in the middle using a screwdriver. Needless to say, this instruction didn't pan out.

I own a "Home Depot shredder" and I'd be surprised if I could use a screwdriver to knock out its middle teeth.

What's always surprising in these suspicious cases is that the researchers cannot produce evidence to support their prior results. Does Ariely or his team not have any pictures of the shredders used? It's like they are their own best lawyers ("missing emails", "missing text messages" come to mind.) 

Over the years, many of Ariely's studies have been found to be fraudulent. I blogged about his work before when he responded to one of the controversies by claiming that he just now learned the importance of examining one's data before analyzing them - something he apparently did't learn in many years of teaching at top-tier universities and publishing in top journals.

***

The second objet d'art is the "bottomless scorpion bowl" by Brian Wansick, Cornell professor at the famous Ag school.

While I didn't hear about the Ariely's shredder until it's been exposed, I remember very well my immediate impression when a conference speaker retold Wansick's experiment on portion sizes of food. I thought it was genius! For years, I used this example in some of my talks.

The scorpion bowl - for my non-American readers - is described in Wikipedia as "a communally shared alcoholic tiki drink served in a large ceramic bowl traditionally decorated with wahine or hula-girl island scenes and meant to be drunk through long straws." It's something you find in college bars all over America. So, Wansick's idea is to make a bottomless scorpion bowl, one that is refilled from the bottom as people drink through straws.

In the experiment, half the college students were led to one room with the bottomless bowl, and the other half to another room with the non-refilling bowl. After the party is over, the students were asked how much they have drunk, and surprisingly, people in both rooms gave similar answers (the detail escaped me but it was portrayed as a slam dunk effect).

This research proved that humans have a poor sense of how much food we ingest, and large portion sizes served in American restaurants are a major cause of American obesity.

It was just the perfect research soundbyte - similar in nature to Ariely's shredder that revealed people's dishonesty.

Fast forward to today. Most of the research that came out of Wansick's lab have been discredited. In many cases, he said he couldn't find the data. Some researchers have questioned the existence of this bottomless bowl (link). It seems hard to make - especially because the device has to react to the rate of consumption. Nevertheless, Wansick has made various videos and pictures of bottomless bowls, and even relayed stories of failed attempts, so maybe the lab did make such a device.

It doesn't have to be so hard to prove that a study was performed. That's why Gelman's idea of a museum is a good one. It doesn't even have to be just for Scholarly Misconduct. It can be a Hall of Fame of ingenious experiments, with a Wall of Shame for retracted studies.

***

Even more amusing. The scorpion bowl apparently is a whole-cloth invention of the conference speaker from whom I heard this story. The bottomless bowl in the experiment contained Campbell tomato soup, not cocktails, so it's a soup bowl, not a scorpion bowl. Presumably, tomato soup is boring and scorpion bowls are fun.

Even a skeptic like myself is sometimes not skeptical enough. Given that Wansick most likely gave out credit to his students for participating in his lab's experiments, serving alcohol during that experiment would seem to violate college rules. Maybe the speaker said they served "virgin" drinks during the experiment - I don't quite recall.

All evidence are not created equal and some evidence are more equal than others

A casualty of the Covid-19 emergency is scientific rigor, and in particular, certain long-established statistical principles in evaluating evidence, and specifically, that the weight of evidence depends on the method of research, and how data are collected and analysed.

The media, and the scientists they employ as talking heads, rarely identify the nature of the evidence behind health policies. On air, they make little or no distinction between results obtained from randomized clinical trials (RCTs), observational studies, surveys, lab experiments, animal experiments, etc. Like a drumbeat, they promote the impression that all evidence are created equal, despite their disparate origins.

***

While such an attitude towards scientific evidence is not new, it became normalized during the pandemic. A great illustration is found in this recent article reporting the results of a clinical trial on colonoscopy (screening) for colon cancer (link).

In the U.S., the standard of care calls for all men above 45 years old to get a colonoscopy every 10 years. Colonoscopies are both expensive and invasive. In some other countries, less invasive alternatives like stool testing are preferred.

American doctors have pointed to strong evidence that colonoscopies work. According to the article:

Past research always showed that colonoscopy could put a huge dent, on the order of 70%, in the incidence and mortality from colon cancer.

In other words, we can reduce colon cancer deaths by 70% - purely by screening, and presumably catching cancer in its early stage. A researcher was quoted saying the profession has expected colon cancer to become "extinct" if every man were to take colonoscopy.

***

Chances are men were not informed that all past evidence about this practice has come from observational studies only.

Observational studies involve comparing the outcomes of people who have taken colonoscopies and those who haven't. However, because people self-select into these procedures, a direct comparison of the two groups is biased. The people who choose to do it are likely to be different from those who don't, beyond the decision to be screened. Analyzing observational data involves finding ways to remove biases.

Take a current example. It is obvious that when comparing the Covid-19 outcomes of vaccinated and unvaccinated people, these two groups are not identical because self-selection introduces biases. There's age bias, political bias, etc. etc. It's challenging enough to enumerate all sources of biases. The hard nut to crack is to learn the magnitude of such bias. Even if we know older people are more likely to be vaccinated, and more likely to suffer more severe consequences from Covid-19, how do we know the degree of bias? If we don't know the size of the bias, how can we tell if we have adjusted too much or too little?

It's like asking the tailor to adjust the length of a pair of pants without telling her the height of the person wearing those pants. It's just too long, the tailor is told.

Back to the observational studies of colonoscopies. Doctors have relied on a series of peer-reviwed observational studies that converge on the conclusion that colonoscopies are extremely effective. What can go wrong?

***

The profession ignored the nature of the evidence, the deficiencies of observational studies.

Finally, some researchers in Norway had the good sense to check these results by running the gold standard of medical evidence, a randomized clinical trial. In these trials, patients are randomly selected to get invitations to take colonoscopies. The random assignment of treatment is an automatic adjustment for known and unknown biases. (Read this previous blog for a primer.)

And... after 10 years of observation, the RCT showed that colonoscopies has essentially zero effect on mortality from colon cancer, and only about 20% reduction in incidence, both metrics coming in far below 70%, the efficacy found in the observational studies. I'll dig deeper into the results in a future post. For this post, I take these findings at face value.

***

What gave me feelings when reading the Stat article is the response from various experts who effectively advocate trashing the RCT result in favor of the observational studies. This reaction is shocking given the RCT's status as the gold standard of medical evidence.

This RCT is not some summer project by a student intern. It includes 85,000 people, with data collected over 10 years. And yet, the article ends with a quote that says "colonoscopy is still king".

The negative reaction reflects the belief that all evidence are created equal, and illustrates the skepticism toward counter-intuitive scientific findings. It's not just in medicine. Anyone who have run experiments in industry has encountered this attitude.

The arguments for rejecting the inconvenient results can be classified into the following types:

(a) Some other related evidence exists that contradicts the current study - e.g. a randomized trial of a related procedure has previously shown better results

The problem with this type of argument is that it says nothing about the current study. A Chinese term for this strategy is "raising East to beat West". This strategy works under the doctrine that all evidence are created equal. The other study is on a different procedure and it is brought into this conversation purely because it obtained the more "acceptable" result. Imagine this: if that trial has produced worse outcomes than the current study, would the expert have mentioned it to the reporter?

If we allow prior experiments to overrule new experiments if the latter's conclusion differs from the former, then what's the point of running the new experiments?

(b) The treatment worked better for a subset of the test subjects - e.g. if we restrict the analysis to the older population, the positive effect of the treatment is higher

Whenever the overall results of an experiment disappoint, the sponsor of the experiment almost always asks for "deep dive" analyses, which typically means, hunting for a subset of the population for which the treatment made a difference. What's wrong with this? Imagine, what if the researcher discovered a subset of patients for which the effect of colonoscopy on cancer diagnosis was not 20% reduction but 5% reduction? This finding will be dismissed and ignored because it adds "nothing new" to the overall conclusion. The only additional findings that can get the attention of the sponsor are those that validate the desired outcome.

Besides, any such deep dive analysis runs into statistical problems. The sample size of the subset is usually too small, harming the reliability of the result. The filtering effect of statistical significance means any such finding is likely to be an outlier for the experiment.

Worse of all, the sponsor may take the positive outcome for the subset, and create the impression that it works for everyone. This actually happens a lot more frequently than one thinks. Take a Covid-19 vaccine study that uses matching to remove biases. While the matching procedure increases our confidence that the two groups being compared are similar, many records are dropped from the analysis because they are unmatchable. For example, because most older people got vaccinated, and many younger people did not, the matched subpopulation has a different age profile than the full population. While the comparison of the matched groups is not biased, the extrapolation from matched groups to the general population is biased. Nevertheless, experts used the matching study to proclaim that the vaccine is effective for all, rather than that the vaccine is effective for the matchable subpopulation.

(c) Our hypotheses are better than your data - e.g. because treatment methods are improving over time, we believe the benefit will increase over time, therefore in five years' time, the outcome will be better

Everyone is entitled to their hypotheses but only those running the experiment has submitted their hypothesis to testing with real-world data. The experts offer no data to support the alternative hypotheses. The experts are also ignoring hypotheses that reinforce the study's findings. Imagine this: what about the possibility that someone would develop a new screening method that might dominate colonoscopies in the next five years?

(d) The data should be analyzed in some other way - e.g. we should use a per-protocol analysis

This reaction makes a mockery of pre-registering the study methods. We face the same problem  of ex-post hunting for a specific outcome. If the per-protocol analysis had yielded similar or "worse" results as the intention-to-treat analysis, the same expert would not have raised this critique.

***

Imagine this: the RCT for colonoscopy has confirmed the profession's longstanding view that the procedure cuts cancer deaths by 70% or more. What would the article look like? Would there be a line-up of experts raising concerns that the RCT outcome might be wrong?

All four types of counter-arguments could still have been raised. There is likely going to be at least one randomized trial of a related procedure that showed much lower efficacy. If the analysts did a deep-dive analysis, at least one subset of the patients probably experienced much lower benefits than the average. Various hypotheses could be proposed that suggest the RCT over-estimated the efficacy. Some other way of analysing the data would have yielded lower efficacy.

Nevertheless, we wouldn't be hearing these arguments.

***

Lost in this debate is the real question raised by the RCT; in fact, it hasn't even been asked. So I'm asking it here: What went wrong in those prior observational studies to have over-estimated colonoscopy's effectiveness to such a large extent? How did the machinery that "adjusts" for biases fail?

 

 

 

 

To prove inflation, get some inflated data

RippleMatch, a business that helps Gen Z finds jobs, was able to compile average undergraduate GPAs at Ivy League universities (link). The statistics may shock you:

Or, if you are aware of grade inflation, they don't.

Can these numbers be trusted? That depends on the data source. Clearly, these are not numbers found in press releases by the universities.

Here's the methodology:

RippleMatch is an AI-powered recruitment platform that matches early-career candidates with high-quality jobs and internships. When students sign up for our platform, they are prompted to fill out a profile which requests several different points of information, including their GPA. We have over 50,000 active student users from over 500 universities and determine the average GPA of each university based on the composite data from our users at each university.

In short, they come from self-selected respondents of a survey. We don't really know how large each university sample is - that depends on what kinds of people use the Ripplematch platform. On average, they have about 100 students per university but the Ivies can be either above or below that average.

Is the sample of students from, say, Brown University, who filled out the survey representative of the universe of Brown students and alumni whose academic performance the average GPA statistic purportedly estimates? In order to be representative, the sample would have to look like random. This would mean that the average user of Ripplematch is like a randomly selected student from any given college.

This condition is highly unlikely to hold. For one thing, this type of AI recruitment platform is favored more by the tech industry; some industries may not use it at all so there is an industry bias. The demand for an automated recruiting platform increases with the competitiveness of the industry, and the most qualified candidates tend to flock to the most competitive jobs.

Students with higher GPAs are more likely to submit their resumes to a platform like RippleMatch. If I had a low GPA, I don't think pushing my resume at online screening applications will result in interviews. 

I would not be surprised if those average GPAs used to prove grade inflation are themselves inflated.

 

P.S. This other webpage contains GPAs at some other schools. For example, Stanford is 3.66 right behind Brown. Vanderbilt (3.57), Duke (3.56), Amherst (3.53), WUSTL/Northwestern/Johns Hopkins (3.52).

 

P.S. Chapter 1 of Numbersense (link) covers the manipulation of statistics by many many law schools and colleges. Grade inflation is also grade compression. It's hard these days to get grades below 3.0 at most schools. Look at Brown's 3.71 versus Princeton's 3.49. That's two-thirds of a grade as the difference between A and A- is 0.3 GPA units.

The difference of 0.22 seems trivial. Emphatically not so! Just try to imagine how many Brown students earned GPAs above average but not perfect i.e. between 3.71 and 3.91. Let's say it's 30 percent. (That sounds high but remember that it's rare to get GPAs below 3.0 nowadays.) If we assume that the shapes of the grade distributions are identical between Brown and Princeton, except for the means, then Princeton's GPA distribution is just shifted down by 0.22 GPA units. So the equivalent 30 percent of above-average students at Princeton will have received GPAs between 3.49 and 3.69 (3.49 + (3.91-3.71)). This interval notably does not overlap with Brown's 3.71-3.91. 

Go here to get your copy of Numbersense