Big Data, Plainly Spoken (aka Numbers Rule Your World)

Know your data 36: the what and the why

In the unlikely place of a food blog (link), there appeared a feature article about restaurant data science, which provides a rare glimpse of the ins and outs of the data collection business.

The article describes pretty basic elements of data science that have been used by other industries for a while; evidently, restaurants have been reluctant to join this game until now.

***

Right from the start, the featured company is described as "stealthy." This description has a double meaning – it's a "stealth startup" although given the number of important restaurant groups mentioned in the article that already use its services, it's unclear whether this sense of "stealth" was what the reporter intended; the other meaning is that the company collects data "stealthily" and works behind the scenes with restauranteurs. I have never understood the secrecy in the data collection business when the industry leaders insist that they utilize the collected data strictly for the benefit of their customers – recall the recent fallout over Wendy's announcement to adopt adaptive pricing (link).

Perhaps these data collectors won't voice what they know, which is that raw data aren't all that useful. There is a question that lingers in my head as I read through the article: can we have at least one clear statement of the benefits arising from the collected data?

The article begins with what must be the elevator pitch of SevenRooms, which is portrayed as a leader in data collection and analytics in the restaurant space.

"Let's say you're dining at Lure Fishbar in Soho on a weeknight for dinner... kicking things off with a bottle of Champagne. As it turns out, without even going over to your table, owner... and managers know what you and and your tablemates are spending in real time, as you're spending it."

This leading paragraph is a bit anticlimactic. Restaurant managers have always known which table ordered what, and whether the table is a big spender – even without deliberate data collection. And the reporter didn't answer the question: How does SevenRooms utilize such knowledge? How do these actions benefit the diners?

"Based on your previous visits to any restaurant in the Mercer Street Hospitality group... they know your dietary restrictions and your wine preferences; how often you've visited...; how much you spend on average, per night, and per year."

This sentence also focuses on the what, but not the so what? Let's think this through.

What extra service do frequent diners get? Do the waiters come by more often because I'm a frequent patron? Compared to the prior state, does total service effort increase because more service is rendered to frequent patrons? Or is total service level fixed, implying that the waiters visit low-frequency diners less? If total service effort goes up, are waiters paid more? If so, does the manager expect that the higher service effort will induce frequent patrons to spend more money, thus paying for itself?

***

Next, let's consider the nature of data such as "wine preference". What SevenRooms actually possesses is the history of wine orders for each patron at the subset of restaurants that use SevenRooms, from which the diner's preference is deduced. The accuracy of this deduction depends on the individual. Some people may like only a single type of grapes but others may want a particular vintage from a particular winery of one type of grape if they order a particular dish at a particular restaurant but another vintage from another winery of a different grape when eating some other meal, and so on. If all those factors are included in the model of wine preferences, and we desire accurate predictions at the individual level, then we'd need quite a lot of data for each diner, beyond the amount that SevenRooms could have amassed.

From this discussion, you should realize that "wine preference" information is only useful if (a) the restaurant group is relatively large so it can capture most of the diner's wine drinking; and (b) the diner attains some threshold of loyalty to the restaurant group.

We have encountered this data science application before. Your Netflix movie recommendations won't hit the target if (a) you do a good chunk of your movie watching on other platforms; and (b) you've watched (and rated) only a small number of movies on Netflix.

***

Let's pretend that SevenRooms actually has a good model of people's "wine preference".

For a one-grape patron, it's easy to get "wine preference" right, and I'm sure the managers would have gotten it right, with or without data collection.

In the other extreme, a good wine list may have many wines that fit with a diner's history, and should the restauranteur impress the diner by sending the top recommendation - in lieu of the wine list - to the table? "Here's your wine for the evening. (And don't you dare say no)." If the restaurant doesn't send the wine directly, but arms the sommelier with the recommendation as an ice-breaker, is this materially different from the status quo?

***

The reporter gets more and more confused about the benefits of SevenRooms's data as she issued more words. The data could seemingly solve world hunger. The following miracles were mentioned:

"data can be key in the battle for luring diners"

"using customer data to tailor menus"

"should you choose to splurge on that $3,000 bottle... all hands on deck, with every manager getting an alert"

"can parse seating arrangements so that some tables are earmarked for Resy, some for OpenTable, ..."

"steps up service for VIPs"

"using it to plug and play among various reservation platforms quickly and on the fly"

"stitching together historical information on... needs and guidances"

"plotting tables on the floor plan to optimize the flow of service"

"tracking a table's stage in their dining experience"

"quote wait time"

"automated email campaigns"

keep track of which restaurant reservation channels are responsible for sending restaurants what numbers of diners

let restaurants send automated Priority Alerts to their most valuable guests about availability

"helps restaurants reward their most valuable guests to get them through their doors most often"

run a loyalty program "behind the scenes"

"improves customers' experience"

"and much more"

For each of these presumed benefits, one should consider how it changes the revenues and costs of the restaurants, whether the collected data really provide reliable insights to drive these actions, and how will the restauranteurs learn whether they yield the expected incremental gains.

Know your data 35: Incognito data collection

If data collection is for the benefit of us, as we are often told, the lingering question remains why companies are so secretive in collecting our data?

The recent settlement by Google is another case in point (link). Google admitted that its so-called "incognito" mode on the Chrome browser isn't "private". Based on what is said in the article, third-party cookies were not possible to turn off even when a user was "incognito". Third-party cookies are a crucial part of Google's advertising business. In addition, the tech company has retained huge datasets of people's browsing history while in "incognito" mode, which it has now "promised" to "destroy". (Not "sticking around in browser history" does not mean Google does not store browsing history on its servers.)

I put those two words in quotes because I'm convinced that is an empty promise. It just can't be done. By the way big data analytics work, any data are replicated lots of times, across many platforms and servers, and possibly have landed also in laptops and devices of analysts. Further, the data may even have been shared with partners and customers of Google.

***

Such lawsuits always provide nice fodder as internal emails are unearthed. For example, a marketing executive wrote: "“We are limited in how strongly we can market Incognito because it’s not truly private, thus requiring really fuzzy, hedging language that is almost more damaging."

She was totally right. Even before this statement, Chrome incognito mode came with a suspicious disclaimer.

 

 

 

 

From grand theft to grand retreat

 

Last year, I wrote about the expected downside of deploying self-checkout technology in retail stores (link). Since then, many retail chains have announced cutbacks in self-checkout tech.

The latest to join the exodus is Dollar General (link). DG operates 20,000 stores. They have installed self-checkout registers in 14,000 stores. Because of rampant theft, the retailer is admitting "there is no substitute for an employee presence".

Here is the new policy: In 300 stores, the self-checkout option will be removed entirely. In 9,000 stores (64%), self-checkout will be relegated to an overflow option during rush hours. In the other 5,000 stores, self-checkout will be limited to five items or fewer.

These changes are very telling. The theft is so widespread that the changes affect every store. Limiting usage to five items caps the potential loss.

Target, Walmart, Five Below, Costco, etc. have all announced changes to self-checkout operations in the last few months.

***

This is not the first time removing humans backfires. I see the same problem every week in the NYC subway. Some travelers jump over turnstiles, enter through emergency exits, or use various tricks to evade the fare. Since switching to self-serve ticket machines, most stations have zero staff presence. No one is around to stop fare-evaders, not even to look them in their eyes. (Fare evaders are not limited to the stereotypes; I have seen amongst them families with kids, professionals in suits, and people of all colors and ages.)

I'm guessing that something similar is happening with automated toll collection on highways. Just like self-checkout, there are obvious ways to fool the technology (e.g. remove license plates, cover them, or fake the plate numbers). Like most businesses that deliver a service and collect money later, I'm guessing highway operators face a "non-payment" problem, which requires an expensive collections operation. Contrast this to the past: few drivers ran through toll booths with human toll collectors without paying; tolls were collected before the service was rendered.

For sure, these technologies bring benefits as well. The hidden cost is unfair distribution of costs. The NYC subway is constantly in grave financial trouble. Eventually, they must raise prices. The people paying higher prices are the ones paying the fare! In effect, they are paying for their own tickets, plus the tickets of the fare evaders. For self-checkout, the ones not stealing will eventually pay for those abusing self-checkout. Those opting not to use self-checkout also will pay for the groceries of the self-checkout thieves.

This state of affairs may last for a while. Over time, more and more customers will migrate to the cheating segment, as it is the rational thing to do, given the rules of the game. The more people cheating, the smaller the remaining group of customers bearing the cost, the greater the incentive for them to cross over.

***

There is one other thing that these retail executives are going to discover soon, something that is just as predictable as self-checkout theft. This problem will be particularly acute in those stores that choose to combat theft by deplaying staff as monitors. The reintroduction of humans has transformed their roles from servicing with a smile to policing with a scowl. Customers find themselves between machines and police, not the most inviting environment. (The machines will likely also be equipped with surveillance tools.)

As some of these retail chains have learned, there is still a place for human-rendered services.

Is Waze a ruse?

Waze, Google Maps, and other GPS navigators presents themselves as solving an optimization problem that has been studied for decades: the shortest-path problem of navigating from a given starting location to a given ending location, while staying on legal roads on a given map, with the goal of reaching the destination in the shortest possible time.

The simplest version of this problem admits a static solution, that is to say, it gives the same route regardless of time of day. In this setting, traffic conditions are ignored, and speed of driving is assumed the same for all drivers and at all times. In practice, this static solution sets a benchmark, and a lower bound for the shortest possible time.

A more sophisticated solution removes those restrictions, which is possible only if some kind of sensor data is available, via sensors buried in the ground, embedded in mobile phones, or roadside cameras. With the ingestion of "real-time" data, the solution involves a new element - forecasting.

That's because knowing the traffic where the driver is currently is useless; it's dead on arrival, so to speak. What the navigator needs ideally is the perfect crystal ball - the future traffic conditions between the current location and the destination.

***

If you're a regular Waze user, you may have noticed Waze doesn't quite have the crystal ball. I have suspected this for a while, and a recent trip by a friend from Brooklyn to Manhattan on a Sunday morning motivated me to blog about this.

When he started the journey at 11:02 a.m., Waze estimated an arrival time of 11:32 a.m., so a travel time of 30 minutes. My friend sent me a link that allowed me to track his progress, with a note: "Waze shows only 30 minutes, doubt it... Let's see."

Needless to say, the 30-minute projection was wrong. Not slightly off, but way off.

At 11:17 a.m., about half way through the original schedule, the arrival time has shifted back 4 minutes to 11:36 a.m. Not bad -- except that it kept sliding. Ten minutes later, at 11:27 a.m., Waze now showed an estimated arrival time of 11:50 a.m. Wait, the new estimated travel time was almost 20 minutes more than the original estimate of 30 minutes. My friend texted over a photo of the traffic on the road.

Twelve minutes later, at 11:39 a.m., the estimated arrival time shifted further back to 11:54. So the passage of 12 minutes added another 4 minutes of delay. At 11:46 a.m., seven minutes later, the new arrival time was 11:56; thus, another 2 minutes of delay upon passage of 7 minutes. Another three minutes later, the estimated arrival time moved again, this time, it moved earlier by 1 minute. Oops, two minutes after saying he'd get there one minute faster, the app recomputed the arrival time again, adding two minutes, more than reversing the previous update.

In the end, he got there in 60 minutes, which is double the estimated travel time of 30 minutes at the start of the journey. At 11:50 a.m., 10 minutes before actual arrival, Waze gave an estimate of 5 minutes left so the actual remaining travel time was still double the estimated time towards the end of the trip. Pretty embarrassing stuff.

Note that this was a Sunday morning trip, not a particularly busy or unpredictable time. And the delay wasn't due to traffic accidents. My friend knew it was wrong at first sight.

***

My suspicion is that these navigators are not solving any kind of mathematical problem. It is not making any forecasts. It is purely reactive, responding to what's directly ahead of them.

Ever followed the progress bar on a file download of a large file? You'd probably notice that the estimated completion time fluctuates wildly, both up and down. This fluctuating phenomenon is most apparent at the start of many downloads, when there is a burst of speed. The progress bar will give an estimate, e.g. 5 minutes left. This number soon shots up to 8, 10, 15, 20 minutes... as the download speed slows down.

I have long suspected that the remaining time as displayed is estimated by a simple calculation: remaining amount of data to download divided by the current speed of download. Thus, if the download speed increases, the remaining completion time is shortened, and if the download speed slows, the remaining time is increased.

In this setting, there is no "forecasting" per se. The computation assumes that what's happening now is what will happen in the future. The current download speed will continue into the future until the file is completely downloaded.

Constant speed is a very poor assumption because the variability of the displayed remaining time is a direct result of variable speeds. If the download speed were constant, as assumed, then the remaining time would peel off at a linear rate consistent with real time. In other words, the estimated completion time would not change. In other words, the projection based on constant speed played out as expected.

***

The feeling I got from observing my friend's Waze on his recent trip reminds me of watching large files download onto my laptop. Could it be that Waze implements a similar, simple computation? The estimated arrival time is based on the remaining distance to the destination (assuming the driver follows the prescribed route) divided by the current speed of the driver?

It must be a little better than that. If it really used the instantaneous driving speed, then every time the car stopped, the estimated arrival time would jump to infiinty. Instead of the current speed, perhaps Waze takes the average speed of some duration, say 5 minutes.

Regardless, it appears that Waze's assumed driving speed does not match reality well at all.

Read that sentence again. Then, ask yourself: did Waze give you the shortest route to your destination at the start of the trip? Could it do so without accurate projections of your driving speed on that route?

 

 

 

Know your data 34: think before you scan

Quick, when was the last time you scanned a QR code? How frequently do you scan QR codes?

Source: Wikipedia

In recent years, QR codes have become omnipresent. They have been accepted by the public as a source of convenience. It's an elegant concept which led to an inspired piece of engineering. The camera on our mobile phone detects the QR code image, it converts the image into numbers then to text. The text is in the form of a hyperlink (https://..) The phone then calls that web link, the server on the other side sends the contents, e.g. the menu of a restaurant. In the meantime, the server collects data such as the owner of the phone, the location of the phone, etc. Advertisers and third-party trackers can insert themselves into this process.

A QR code is just a two-dimensional bar code. Usually, what's being encoded into the two-dimensional bar code is a hyperlink.

So, scanning a QR code is equivalent to clicking on a hyperlink.

Remember what cybersecurity people say about unsolicited emails with hyperlinks? They say never to click on such hyperlinks unless we inspect carefully the link itself. Something that starts with "https://www.citibank.com/" might be legitimate but if it looks like "https://citibank.bankingcenter.info/", that might be a pretender, and clicking on it will open up a communication channel with a server operated by a scammer.

***

What are we doing when we scan a QR code using our cell phone? We're clicking on a hyperlink - sight unseen.

Because the text is scrambled as a two-dimensional bar code, we can't inspect the link. This convenient piece of engineering opens us up to the kinds of scams that cybersecurity people have warned us about.

This scam is super easy to pull off. The scammer just needs to post a QR code outside a restaurant, with the words "Scan me to see the menu". Alternatively, the scammer can print their own QR code and place it on top of a legitimate one.

Think before you scan.

***
Here's more from the FTC warning (link).

P esteem, bosom peril, flag to clamor, and anti-tech tech

What drug am I on this morning? Let me tell you, it's the anti-tech tech drug.

As you may have guessed, many new technologies are prone to abuse e.g. using generative AI to produce deep fakes, or what I learned earlier this week, QR code scams (link). The tech consensus is that we don't see any problems because one can use technology to combat such risks. I call this the anti-tech tech drug.

In the academic community, the digitization of knowledge has had profound impacts. One such impact is the creation of "metrics" - a whole field called bibliometrics - that purportedly measure the influence of specific researchers and research papers. What did I say about metrics? See this post from long time ago called "The inevitable perversion of metrics", which later got expanded into Chapters 1-2 of Numbersense (link).

Someone decided it would be a good idea to measure research contribution just like we measure the value of online advertising. Both count things like citations, clicks, downloads, views, etc. Citations are like hyperlinks. Google, whose initial algorithm was entirely based on counting hyperlinks, quickly discovered that hyperlinking is easily gamed. From very early on, humans invented multiple schemes to defeat it as a measure of importance, e.g. link exchanges, barters, buying links, selling links.

 

Not surprisingly, smart researchers have used myriad schemes to game bibliometrics. According to this recent article in Significance magazine, some less-than-honest researchers are flooding the zone with "fake" papers, written by bots. Just imagine having generative AI generate large numbers of "new" papers by feeding it already published articles.

Some anti-tech tech companies say don't worry, we have technology to combat such abuse. What is this tech? It's a plagiarism detection tool that can find copied contents.

What happens? The tricksters adapted.

***

Plagiarism detectors look for segments of text that have been seen before. What kind of text will evade detection? Unusual phrases, what we might call statistically unlikely phrases. Well, someone already wrote software that injects unusual phrases to replace common ones. The tricksters use such software to continue gaming the metrics.

Thus, these "rephrasing tools" have invented "P esteem" instead of "p value", "bosom peril" in lieu of "breast cancer", "flag to clamor" which means "signal to noise".

***

Even so, the authors of the Significance article were too polite to say out loud what's really broken about the peer review process, which has failed to reject many such papers. They reported that more than a thousand papers retrieved from PubMed contain the words "statically significant" instead of "statistically significant" in its title or abstract.

The authors respond to this enormous red flag by musing that "somehow, these goobledegook sentences also pass through peer review."

I can think of two likely scenarios:

1. some peer reviewers did not even open the papers they were supposed to opine upon
2. some editors colluded with authors in a publishing scam

 

Getting serious about AI

Finally, we have a serious report on the potential harms of AI that is recommended reading (link, paywall).

***

Here are five things about AI harms that I think everyone should know.

1. The press is typically focused on the wrong harm

Perhaps misdirected by industry heavyweights, the press frequently waxes about the potential for AI to "kill us all." But it is mundane harms, such as being robbed $1,000 by an anonymous person across the ocean, that most of us will likely be victims of.

2. Don't forget the term "deepfake"

A few years ago, the press warned us about the harms of "deepfakes." These days, not so much. In fact, one has to dig pretty deep into the Financial Times article to find a mention of "deepfake." At least they used the term, which is a pleasant surprise. What the industry is now calling "generative AI" is the same thing as what was labeled "deepfake".

The name change switches our attention from the harmful use cases to the beneficial ones. But a name change does not reduce the potential for AI abuse. Rather than ameliorating it, the potential for harm is scaling up as quickly as the underlying technology is developing.

 

3. It's two sides of the same coin

It's wonderful to imagine a world in which we can deploy AI that gives us benefits without also inducing harms. But it's a mytical world that won't exist.

Take the simple example of 1-click shopping, for which our credit card details must be stored. For sure, these technologies make it much more convenient for online shoppers, and delight retailers as they reduce the friction of buying. But they also open people to hacking, stolen identities and scams.

If our account details weren't saved on the servers of these online retailers, then they could not be stolen. Not saving such data also destroys the value of 1-click shopping.

 

4. Reducing AI harms through more AI is a dead end

I laugh every time someone claims that the solution to AI abuse is more AI. This idea requires AI to predict accurately which outputs came from AI.

An example would be technology that tells professors which students submitted essays written by ChatGPT. The AI harm reduction technology reads an essay and computes the probability that it was written by an AI not a human.

But... we are looking at one side of the coin. What's on the other side? People - e.g. newsrooms, banks - are looking to use the same AI technology as the students to write articles, replacing human writers. The more successful are these use cases, the less predictive is the AI harm reduction technology.

Remember that the ultimate goal of AI is to equal or exceed human ability. Let's pretend we have an AI that is effectively equivalent to humans. How is it possible for another AI to predict which output came from AI and which from humans?

 

5. AI developers can reduce AI harms today (but they don't)

Try subscribing to ChatGPT. You will find that you can't do so without disclosing your identity. They won't accept a Google Voice number, they absolutely want your mobile number, which completely identifies you to them.

The main reasons why ChatGPT wants your personal data are commercial.

However, we know that they know who we are. Now, consider the professor who wants to know if someone used ChatGPT to write an essay.

You don't need any AI to make a prediction. ChatGPT knows for sure because ChatGPT has the personal data of the student. So if ChatGPT is serious about reducing this type of AI harm, it can absolutely do it. This method works even if the AI evolves to the point where it generates essays indistinguishable from those by human writers.

Why won't they pursue this path? I'd let you all ponder this.

On the evidence raised by SAT distractors

After reading my prior post, long-time reader Mark (link) alerted me to a series of posts he wrote 10 years ago on the anti-SAT movement. This is a topic that bubbles up every few years, and it appears that the current generation of college administrators may go where their predecessors didn't. Some colleges have started to drop the SAT requirement for admissions.

Post 1, Post 2, Post 3, Post 4, Post 5

Let me summarize Mark's various posts and add some color commentary.

***

In his first post (link), Mark explores the relationship between SAT scores and college GPAs. This exercise in analyzing correlation is meaningful to both sides of the debate. The distractors detractors argue that "there already is something that accurately mirrors the grades a student gets in school. Namely: the grades a student gets in school" while proponents say SAT scores are the best predictors of college success.

For example, when MIT reinstated the SAT requirement after the Covid-19 pandemic (link), the dean of admissions stated:

Our research has shown that, in most cases, we cannot reliably predict students will do well at MIT unless we consider standardized test results [especially the math component] alongside grades, coursework, and other factors.

Mark points out that the right measure of any predictor is incremental insight – what additional nuance it adds to the collection of predictors. This explains why the SAT looks more like a general intelligence test than a high-school exam. Otherwise, the SAT score would be an unnecessary duplicate of high-school GPA. (Why the previous sentence is stated forwards but not backwards is another sign of weak analytics in this space.)

Mark's point is self-evident: assume that high-school GPAs predict college performance just as well as the SAT, as the distractors detractors suggest, then SAT scores should mirror high-school GPAs, and if they did, the two metrics would result in identical rankings of the students; if so, what's the cause for complaint?

***

The second post (link) examines various reasons SAT critics have given for dumping the test. Mark is unimpressed by the quality of the evidence, and in many cases, its absence.

The weirdest argument is that according to Google's description of its own hiring practices, Google does not consider standardized test scores or college ranking. Mark remarked that "Google doesn't talk much about SATs at least in part because they've largely maxed out the metric."

Let's say we tally up the background data of Google hires; we are likely to find a good number of Ivy League graduates with high SAT scores. Can Google legitimately claim their hiring does not consider test scores or college ranking?

Sure, someone is going to say "Those people weren't hired because they graduated from an Ivy League school, or that they had high SAT scores. They just happen to have those characteristics. Correlation is not causation!"

Well, admitted students to top colleges not only have high SAT scores, but they also have high GPAs, and other great things on their applications so did they get accepted because of the SAT scores, or the high GPAs?

***

The third post presents a series of math questions from actual SAT exams because Mark realized that many critics of the SAT attacked an imaginary scarecrow version of the SAT test, rather than what the test really is. Seeing these real questions helps ground the debate.

He also touched on one virtue of the SAT test, which is its transparency. Past exams with answers are readily available. By contrast, transparency is definitely missing in high-school GPAs. Nevertheless, in real life, transparency frequently backfires. Where are these SAT critics when it comes to high-school GPAs? Let's compile GPAs and split them by race - I'd be surprised if they didn't find a race gap. The reason that the race gap in GPAs isn't in the news is that schools don't release such data.

***

The fourth post is a must-read. Here is Andrew's recap of what Mark wrote.

Mark trashed those critics who grumbled about the point deduction for wrong answers in the SAT. Mark explained why the point deduction scheme in fact simplifies life for test-takers, because the penalty makes it counterproductive to do "random guessing" (what Mark describes as "guessing without first reading the question"). Tests that don't deduct points for wrong answers are more annoying because smart test-takers should guess on certain questions based on their level of confidence in knowing the answer.

The unspoken point behind Mark's post is that the SAT is rigorously designed, and continuously improved, by lots of statisticians, and therefore it is the best instrument we have. In Numbers Rule Your World (link), half of Chapter 3 is devoted to the statistical work behind a niche issue in SAT tests, work that reduces specific biases due to race and other characteristics.

The biggest irony in the anti-SAT movement is that on the major complaints about the SAT, alternative instruments such as high school GPAs and reference letters are far worse.

***

In the last post of this series (link), Mark turned our intuition upside down. He argued that the students most likely to benefit from the SAT admission requirement are those good students who go to public schools – or if we extend this line of reasoning, good students from underprivileged backgrounds. That is to say, the SAT requirement benefits precisely those people whom its distractors detractors say they care the most about, with the caveat that the benefit accrues to the cream of that crop.

Ten years later, the MIT dean of admissions echoed Mark's point when he said that MIT uses the SAT score to identify disadvantaged students who are most likely to succeed at MIT.

unlike some other inequalities — like access to fancy internships or expensive extracurriculars — our empirical research shows the SAT/ACT actually do help us figure out if someone will do well at MIT.

***

This last discussion leads us to some general topics in the area of statistical measures. No metric is perfect, most metrics are correlated with each other, all metrics promote perverse behavior, alternatives often appear better only because of lack of data or analyses. See Chapter 2 of Numbersense (link) for more on measurement, where I dig into the anti-BMI movement. BMI is the widely accepted metric for assessing obesity; like SAT scores, the metric has attracted a small loud group of distractors detractors.

 

Ryanair fights with its retailers

This story (link) about Ryanair's rift with several online travel websites caught my eye. The outline of the story reads like a typical business dispute, the likes of which U.S. cable TV consumers have regularly experienced, such as when Disney channels were taken off the Charter network (link). In Ryanair's case, the back story is fascinating, bringing out the contradictions and tensions in the modern tech industry, and that's what I want to discuss today.

***

Ryanair is a budget airline based out of Ireland, one of the largest airlines in Europe. A few online travel sites (Booking, Kayak, Kiwi) have recently suspended selling Ryanair tickets. This action escalates an ongoing feud between the two companies. Ryanair is unhappy that online travel sites "trick" customers using an impossibly low fare, and later tag on extra fees. In addition, Ryanair won an injunction to stop a third party called Flightbox from scraping pricing data from its website. Flightbox supplies the scraped pricing data to online travel sites. Ryanair is also annoyed that these online vendors hide customer email addresses from the airline, and has countered by launching a "Know Your Passenger" initiative.

The removal of Ryanair's listings may have negatively affected Ryanair's sales during this past holiday season, although only slightly, while the dispute is not expected to negatively impact Ryanair's profit during the quarter.

That's a lot of grievances to unpack.

***

In the air travel industry, airlines are the "manufacturers", online travel websites are the "retailers", and passengers are customers. Online websites collect commissions for selling air tickets from the airlines to the customers. The airlines also sell directly to consumers, setting up competition with online travel websites. Such competition is routine in digital commerce; many manufacturers have direct presence on Amazon, for example.

Online travel websites such as Travelocity, Kayak and Expedia have been around for many years. At first, they pitched themselves as "disintermediating" the industry. Travelers would be better off buying plane tickets online, cutting off the "intermediaries", who were travel agents relying on airline commissions. In the late 1990s, Expedia started out as a do-it-yourself travel planning tool inside Microsoft that refused to serve travel agents.

Eventually, it emerged that these startups did not so much remove intermediaries as take their business. These online travel websites are now the ones earning commissions from airlines. When we purchase these sites, we aren't buying directly from manufacturers. Similarly, Amazon has displaced big-box retailers by becoming the mega-big-box retailer. Square or Apple or Google has not disintermediated payment processors; they just substituted previous players, and worse, in some cases, as when a website processes Visa transactions through Square, the new players inserted another middle layer that collects even more intermediary fees. Many of these websites also now serve their former adversaries: some brick-and-mortar travel agents use these websites to serve their customers. Many real estate agents list their short-term rental properties on AirBnB.

The disintermediation story never made any sense since the intermediate layer fulfills some functions, e.g. comparison shopping, and search tools, that aren't possible when a customer buys directly from an airline's website. Running websites that provide such services costs money.

***

These intermediary businesses go to unsavory lengths in pursuit of revenues. Some years ago, mom-and-pop restaurant owners that use Grubhub were surprised to find "fake" websites and phone numbers created in their names by Grubhub. When potential diners search for their favorite restaurants, Google sends them to these Grubhub websites, so their orders are effectively intercepted and rerouted, allowing Grubhub to charge restauranteurs tolls. (More on this here.)

Typically, these websites justify their fees (as high as 30%) by claiming they generate sales leads for restaurants. They claim credit for every transaction conducted on their websites. Such credit attribution policies confuse correlation with causation. It produces a perverse incentive: the easiest way to make a buck is to redirect traffic from the manufacturers to themselves.

Of course, some of the transactions would not have happened without Grubhub but what proportion?

The Ryanair affair has shown that the influence of such intermediaries is much weaker than claimed. After Ryanair listings disappeared from those travel websites, Ryanair's bookings only dropped by 1 or 2 percent. In other words, almost all of their customers found their way to Ryanair's website and made their purchases there.

Since those diners searched specifically for their favorite restaurants on Google, most of them would have ordered directly had they not been diverted to Grubhub.

***

The other two complaints are salvoes in an ongoing information warfare. I previously wrote about the web scraping business here.

Flightbox is a company that scrapes data on behalf of its clients, such as Booking.com. Since web scraping is in a legal gray zone, large or public companies may attempt to shirk legal liability by outsourcing such work to third parties.

An Achilles heel of any online business is that they expose their pricing data to the entire world. Previously, one has to be physically present in a store with pen and paper or mobile devices to collect pricing data – it was very expensive, even impractical, to get complete data on all stores and all products. That all changed.

Any economist can tell you information has value: in any negotiation, or trade, or power play, the side with the better information has better odds of winning. Businesses know this well, which is why they suck up any bit of information they can find. Imagine the advantages Target can gain against Walmart if Target knows Walmart's pricing on everything.

Almost all online retailers pronounce that scraping data from their websites is against the rules. You can typically find such provisions in the terms and conditions which they know only a select few read. Flightbox and other web scrapers consciously ignore such provisions. Thus, Ryanair went to the court to obtain an injunction. Not surprising because the online travel websites are using the scraped data to compete with Ryanair's direct sales efforts.

I won't take sides in this dispute as I'd be shocked if Ryanair's data scientists aren't scraping data "illegally" as well. As you may have heard, ChatGPT is being sued also for "illegally" obtaining data to train their large language models. Further, unless the courts severely penalize web scraping, the online travel sites will simply hire something else to continue scraping.

The tussle over email addresses is also part of this information warfare. I'll address it in a future post.

Building a model of grade point averages

In the prior post, I featured the following chart:

This is a simulation of the distribution of GPAs of Yale graduates based on the just-published 2023 distribution of course grades.

What Yale released is a distribution of grades across all courses and departments during the last decade. The press picked up the 2023 grade distribution in particular, and I also used that as a basis for the simulation.

The published information is a distribution of grades, which is not the same as a distribution of GPAs. The GPA is an average over 36 courses, which is the requirement for graduation at Yale. In order to bridge the gap between course grades and student GPAs, we need to know how grades are distributed within students.

Since this information is missing, a statistician will leverage a "model" to fill in the blank.

The most naive model is that all students are alike, everyone is like the average student. In this case, everyone has the same distribution of grades as the average student. This means 22 As, 7 A-s, 4 B+s and 3 B or lower, for the 36 courses taken. This model replicates the average GPA of 3.70. In fact, every student has a GPA of 3.70.

An obvious improvement is to insert more variability so that different students have different GPAs while keeping the average GPA at 3.70.

***

The model that underlies the chart above is called "multinomial". It assumes all courses are like the average course. Any student enrolling in any course faces the following probabilities: 60% chance of getting an A, 20% chance of getting an A-, 10% B+, and 10% C+. (The last category C+ is a catch-all category: in real life, it stands for a grade B or below; based on the released information, the average of all grades in this category is 2.30, which equates to a C+.)

Each student takes 36 courses, and for each course, the student can obtain one of those four grades, with those stated probabilities. It turns out there are about 9,100 different grade combinations, starting with 36 As (perfect) and ending with 36 C+s. Each combination leads to a GPA value that ranges between 2.30 and 4.00.

Those 9,100 combinations reduce to under 200 distinct GPA values, if we round all GPAs to two decimal places. This is because there are many ways to obtain the same GPA. For example, anyone with 35 As and 1 A-s has a 3.99 GPA. But the one A- can come from the 1st, .... or 36th course taken.

For each value of GPA (e.g. 4.00, 3.99), I aggregate all the different grade combinations, and the probabilities of a student getting those combinations. Those probabilities are what you see in the chart above. The probabilities are cumulative meaning that they represent the probability of getting a GPA at or below some value.

For example, the probability of getting a GPA of 3.70 or below is 50%. This is often interpreted as half the students get 3.70 or below. The probability of getting a GPA at 3.87 or above (i.e. between 3.87 and 4.00) is 1 percent so the top 1 percent of graduates earn GPAs at 3.87 or above.

Notice how tightly the GPAs cluster. Half the student body is bunched between 3.64 and 3.75. Ninety-percent of the GPAs are between 3.48 and 3.87.

Since each GPA is an average of 36 grades, classic statistical theory predicts that the distribution of GPAs is substantially less variable than the distribution of grades. While 10 percent of grades are B or below, it is almost impossible to get a GPA of B+ or below.

***

The multinomial model itself uses many assumptions.

We assume every student faces the same odds. In fact, the grading distribution depends on which department, which professor, which course, etc. We also make many assumptions of independence. We assume that all students are independent of each other - this would be violated if some students collaborate on homework, or form study groups, for example. We assume that all courses are independent of each other. An obvious violation of this assumption is that some departments have stricter grading policies than others. As pointed out in my first post, it is also likely that a student who gets an A in one class tends to get an A in another class.

I also assume that the grade distribution at the course level is constant through time. There is disclosure in Yale's report that this assumption is not realistic - they talk about a Covid-19 effect that causes another inter-year jump in grades. Note that the first 9 courses taken by any graduating senior were from four years ago, and so I should have used the course grade distribution from that year, rather than that from 2023. This is an assumption of convenience, which is different from the structural assumptions described in the prior paragraph. I can easily fix this assumption by making the model more complex, and utilizing the additional data released by Yale. I just chose not to bother with the extra complexity for a blog post.

The independence assumptions, however, cannot be simply removed. I'd have to gather details like course departments and professors, and study groups, and so on. None of those data are simple to obtain.

On a high level, every assumption papers over some missing information. Many modern statisticians, especially those of the Bayesian variety, are highly comfortable making assumptions. Doing so is akin to clearing hurdles. Once past, the terrain is much smoother. However, it is very easy to forget about these assumptions, some of which may represent reality poorly.