Big Data, Plainly Spoken (aka Numbers Rule Your World)

Exploring text summarization using LLM tools, part 3

On social media, one comes across much casual commentary on outputs of AI tools like ChatGPT of the type “Look, it wrote this beautiful poem” or “Look, it generated this amazing summary of the document”. Recently, I took a forensic look at examples of text summarization in order to provide a balanced view of the strengths and improvement directions for such tools.

The blog post is broken up into four parts, which are released all today so you can binge if you want.

Part 1: Exploring the length of summary, and selection of contents
Part 2: Exploring selection of contents, grammar, and word choices
Part 3: Exploring word choices, and hallucinations
Part 4: Exploring hallucinations

This post is Part 3 of the series.

***

WORD CHOICES

At the end of the previous post, I noticed that the three LLM platforms treated the phrase "next year" differently. In some summaries, the phrase was directly quoted while in others, the LLM inferred what year "next year" is. But the inference was sometimes wrong.

The one summary that correctly substituted "next year" with 2024 came from ChatGPT. Score one for ChatGPT?

It’s hard to say. Here’s why. I didn’t include the date of the article in the prompt, so it’s not clear how any LLM could figure out what year “next year” is. I dug into the article, and literally found just one clue in the 1,416 words - a sentence that reads “ESPN’s operating income for fiscal 2023 fell 1.7% to $2.8 billion”.

Strictly speaking, this little clue is insufficient to conclude that “next year” is 2024. Because it’s “fiscal year”, not “calendar year”, it’s possible that “fiscal 2023” runs from December 2023 to December 2024. In other words, “fiscal 2023” does not rule out next year being 2025.

Further, we’ve subtly assumed that the article is being read in year 2023. Imagine that someone reads this same article in 2050. The only foolproof way of determining what “next year” means is to know the date the article was written, which is not found in the text itself.

This is an example of why sometimes it’s better to leave the original words alone.

***

Another word change found in most of these summaries is turning “annualized savings” into “annual savings”. As any financial analyst can tell you, they are not the same things; if so, we’d not need two different words!

Ditto with swapping “operating income” with “profit”.

One of the summaries substituted “board seats” with “board influence” when discussing what the activist investor was seeking. There are many ways to influence the board of directors, only one of which involves pushing for board seats.

Claude got creative when dealing with the reasons for better results in the Experiences segment. In one of the summaries, it uses “travel demand” instead of “demand for in-person entertainment experiences” in the original. In the other summary, it states that “demand rebounds for in person entertainment” while the original sentence said “Disney… invested heavily … in the hopes of capitalizing on rising demand for in-person…” One can call this a mild form of “hallucination”.

HALLUCINATIONS

In the LLM community, the term “hallucination” is used to describe “fake” details invented by the LLM tool that are not true. In the last section, I pointed to a case in which the article said Disney expected a rising demand for in-person entertainment experiences, which isn’t the same thing as observing a rebound in demand. It’s a subtle issue of expectation vs observation but extremely important to any reader who cares about the accuracy of the summarization task.

The good news is that by and large, the problem of hallucination is not evident in my test. No numbers were invented, and the numbers seem to have survived paraphrasing.

For example, ChatGPT printed this:

Disney has made strides in its streaming sector, reducing losses from $1.47 billion to $387 million in the recent quarter,

instead of the original text:

The business, which also includes Hulu and ESPN+, lost $387 million in the most recent quarter, down from $1.47 billion a year earlier.

That's nice.

With respect to hallucinations, I found a few sentences to explore further. I'll cover these in the next post of the series.

Exploring text summarization using LLM tools, part 2

On social media, one comes across much casual commentary on outputs of AI tools like ChatGPT of the type “Look, it wrote this beautiful poem” or “Look, it generated this amazing summary of the document”. Recently, I took a forensic look at examples of text summarization in order to provide a balanced view of the strengths and improvement directions for such tools.

The blog post is broken up into four parts, which are released all today so you can binge if you want.

Part 1: Exploring the length of summary, and selection of contents
Part 2: Exploring selection of contents, grammar, and word choices
Part 3: Exploring word choices, and hallucinations
Part 4: Exploring hallucinations

This post is Part 2 of the series.

***

In the previous post, I classified ideas from the original Wall Street Journal article into three groups: Essential ideas, Primary ideas and Secondary ideas. Primary ideas are those considered important by a specific platform, showing up in both runs, but not essential, meaning that at least one platform excluded them in both runs.

Because Mistral uses fewer words, its summaries must have ignored some of the primary ideas retained by Claude and ChatGPT. What are these?

• The entertainment giant said on Wednesday in its earnings report that it is seeking $7.5 billion in annualized cost efficiencies, up from the $5.5 billion it targeted at the beginning of this year.
• Disney […] generated sales of $21.2 billion for its fiscal fourth quarter
• The company’s main streaming service, Disney+, added 6.9 million “core” subscribers…
• Overall, Disney+ ended the quarter with 150.2 million global subscribers,
• ESPN’s operating income for fiscal 2023 fell
• Excluding ESPN, Disney’s traditional TV networks saw revenue fall

Mistral omissions relative to the other platforms fall into three types: overall performance indicators (revenues, operating profit, etc.), specific business divisions (ESPN, TV networks), further details on key business divisions (Disney+ subscriber numbers). Of these, Mistral treated overall performance indicators as "secondary" while completely omitting news about ESPN, TV networks or Disney+ subscriber nambers.

Ponder the first bullet for a second. Five of the six summaries did not mention “earnings report” or “Wednesday,” that is to say, they excluded the context of the article. The one summary that did came from ChatGPT, in which it said, in its last sentence, “Disney’s stock rose after hours following the earnings report, even as it deals with possible new activist campaigns for board influence.” I’m giving ChatGPT a merit, even though I suspect the inclusion of “earnings report” is more accidental than intentional.

Score one for Mistral for being the only platform that considered narrowing streaming losses as a primary idea. Both Claude and ChatGPT dropped what was described in the opening line as one of the “two areas … crucial to the company’s future”. While all summaries addressed the reduction of losses and the projected breakeven in the streaming business at some point, only Mistral elevated the streaming business challenges as a key area of concern.

GRAMMAR

One of the most exciting areas of progress in text processing is the computer’s ability to write grammatical sentences. The six summaries were largely free of grammatical mistakes. Here are several minor mistakes:

ChatGPT #1 wrote:

“Disney+ lost subscribers in India and is considering selling its unit there after failing to secure cricket streaming rights.”

In the original paragraph, it’s clear that Disney is the entity selling the service in India while Disney+ is the entity that lost subscribers.

In compound sentences, ChatGPT appears to have trouble keeping track of the subjects. Another example is

“The company is also seeking strategic partners for ESPN as it transitions to a direct-to-consumer platform.”

What is the “it”? In reality, the company is transitioning ESPN to something. (Besides, the article did not ever associate the strategic partner search with the direct-to-consumer transition.)

ChatGPT #2 also got off rail when talking about the India service. It included

“Despite losing subscribers in India due to lost cricket streaming rights, Disney is considering selling its India unit.”

Instead of “despite”, it should have read “After” or “As a result of a subscriber attrition in India due to …”

An inference mistake was also found in one of the Mistral summaries, which included

“The company's Experiences segment, which includes theme parks, cruise ships, and merchandise licensing, also saw strong growth.”

Note the erroneous insertion of “also”. The previous two sentences were about the losses in the subscription business so “also” sounds out of place. Either omit it or use something like “however”.

WORD CHOICES

Another dimension in which the platforms differ is the degree of paraphrasing (vs direct citation). Mistral summaries tend to lift entire phrases out of the original article, with few changes to wording and phrasing. At the other extreme, ChatGPT likes to rewrite as much as it can.

I favor quotation as word changes often create subtle (and sometimes unwanted) shifts in meaning. However, there are real concerns about plagiarism and copyright that must be addressed with this type of writing aid.

Let me just give a colorful example of word changes to illustrate the risks.

Recall one of the essential ideas of the article:

“The company […] reiterated that it believes streaming will break even by September of next year.”

The Mistral summaries pretty much print this line verbatim. Both Claude’s summaries proactively substituted “next year” with “2023”. But that is the wrong inference! Next year is 2024.

ChatGPT gives different results in each summary: in the first run, it retains “next year” like Mistral; in the second run, it substitutes “next year” like Claude. However, ChatGPT got the year right.

Score one for ChatGPT?

Not so fast, as I'll explain in the next post of the series.

Exploring text summarization using LLM tools

On social media, one comes across much casual commentary on outputs of AI tools like ChatGPT of the type “Look, it wrote this beautiful poem” or “Look, it generated this amazing summary of the document”. Recently, I took a forensic look at examples of text summarization in order to provide a balanced view of the strengths and improvement directions for such tools.

The blog post is broken up into X parts, which are released all today so you can binge if you want.

Part 1: Exploring the length of summary, and selection of contents
Part 2: Exploring selection of contents, grammar, and word choices
Part 3: Exploring word choices, and hallucinations
Part 4: Exploring hallucinations

This is part 1 of the series.

***

The running example in this series of blog posts concerns this Wall Street Journal article on Disney’s latest financial results (link). I asked the LLM tools (large language models) to

“Please summarize the below, and be concise and accurate.”

I imagine myself to be a developer considering building an app using the outputs of the LLM tool, and therefore, I’m seeking a level of quality that rises above the casual observer using it as a toy.

I asked four different LLM platforms to summarize the same article. On each platform, I made two runs, as these LLM tools incorporate pseudo-random behavior designed to generate different outputs each time. The four platforms tested are ChatGPT (from OpenAI), Claude (from Anthropic), Zephyr (from Huggingface) and Mistral.

[Note: You should consider these blog posts as exploratory research. I'm using a small sample. At this initial stage, it's important to look into details, rather than in aggregate.] [To be specific, ChatGPT and Claude results were obtained from their web client. Zephyr was 7B Beta, and Mistral was 7B Instruct v1 AWQ.]

The platform that makes the biggest first impression is Zephyr, in a head-scratching way. Instead of a summary, Zephyr sent back an invented Q&A script, like this:

According to the article, how did Disney's core streaming service, Disney+, perform during the latest quarter? What factors contributed to its success?

Answer: During the latest quarter, Disney's core streaming service, Disney+, added 6.9 million "core" subscribers (excluding India), bringing its total number of global subscribers to 150.2 million. This was more than double what analysts had predicted, and Disney highlighted the popularity of recent movies like "Pixar's Elemental," "Guardians of the Galaxy Vol. 3," and "The Little Mermaid." Average monthly fees for Disney+ increased slightly compared to the previous year. These factors contributed to Disney+'s success. However, the article also mentions that Disney is exploring selling its India unit due to subscriber losses following a bid for cricket league streaming rights.

This might be because this LLM platform hasn’t been trained to do text summarization but it has been trained to produce Q&A scripts from a document. For the rest of this blog post, I exclude Zephyr.

***

The other three platforms dutifully produced summaries of the WSJ article. At first glance, they all look pretty good. They certainly beat my expectation of what a computer could generate. Here is one of the Mistral summaries:

Disney has announced a more ambitious plan for cost cuts and has made progress in its streaming business, narrowing its losses. The company's streaming business, which includes Disney+, Hulu, and ESPN+, lost $387 million in the most recent quarter, down from $1.47 billion a year earlier. Disney has reiterated that it believes streaming will break even by September of next year. The company's Experiences segment, which includes theme parks, cruise ships, and merchandise licensing, also saw strong growth. Disney shares rose 4.65% in after-hours trading.

[If interested, you can download all six summaries here.]

Of the six summaries I’m comparing, the within-platform differences (two runs) are much smaller than the between-platform differences (three platforms). In most of what follows, I focus on between-platform differences.

LENGTH OF SUMMARY

The most noticeable difference between platforms is the length of the summary. From the most concise to the most verbose: Mistral > Claude > ChatGPT.

Since I did not ask for a specific length, each platform decides the number of words to use. This is an important metric because the more words in the summary, the more details are included, the less loss of meaning. The original article has 1,416 words (which includes a small amount of stray text insertions like captions and so on). The Mistral summary #1 shown above uses 85 words, which gives a compression rate of 95%. The ChatGPT summary #2, the longest of the six, uses 284 words, more than three times as long as Mistral #1, and the compression rate was 80%.

Of course, longer summaries are not as summarized, and potentially less useful. In the future, I will run another test specifying different word limits.

SELECTION OF CONTENT

The differing lengths of the generated summaries implies that the different platforms selectively retained different parts of the article. I divided the contents into three groups: “Essential,” if an idea is found on all six summaries; “Primary,” which are ideas present in both runs of the same platform but not present in all platforms; and “Secondary,” which includes ideas that appear in only one of two runs from the same platform.

Essential ideas are present in all six summaries.

Primary ideas are ideas that each LLM platform considers important enough to include in both runs - but they are not essential, meaning that at least one platform chose not to mention them at all.

The long summaries generated by ChatGPT included many ideas that are not found in the Mistral summaries.

Secondary ideas are ideas that pop up in only one of two runs, by platform.

Note: all words in the above table are direct, unaltered quotations from the Wall Street Journal article. As discussed in a future post, certain LLM platforms tend to paraphrase rather than quote directly.

***

I’m concerned that the proliferation of secondary ideas found in just two runs of each platform. This suggests that these platforms do not have a consistent model of what’s important and what’s not in a given article. Of the essential ideas, I’m not convinced by the inclusion of the Experiences segment.

In the next post of this series, I'll examine whether Mistral should have included some of the primary ideas from the other two platforms.

 

 

Know your data 34: the best defense is offense

There really is not much more to say than to print the first sentence of this Bloomberg artcile (link):

Facebook owner Meta Platforms Inc. for years paid a contractor to scrape data from other websites while publicly condemning the practice and suing companies that pulled data from its own social-media platforms.

***

Web scraping is a very strange industry that operates in a shadow. At the end of this post, I hope you'll appreciate why the scrapers are hiding.

Let's trace how data land on websites. I'm going to use an example of baseball statistics in Baseball-Reference.com. Here is how one set of data looks like to human eyes:

The browser does not see it this way. This is what it seems (HTML):

Each line is a row of data and these lines extend far into the right. Here is the next part:

You start to see the numbers embedded among the HTML tags here (49, 27.3, ....)

Not surprisingly, the HTML format is not efficient for storing data. The raw data are stored in databases, and there is a process that extracts the data out of these databases, and transform them to HTML format. The transformation essentially adds formatting to let the browser know how to display the data.

The act of "web scraping" is to grab the HTML-formatted files, then reverse the above process - i.e. strip out the formatting, and reduce the information back to something that looks like an Excel spreadsheet.

***

The mystery is why this reverse process is needed.

What the scraper wants is the original dataset without the formatting. You'd think they can just ask for it directly.

In fact, some websites encourage others to use their data, so they create tools to distribute them. For example, Baseball-Reference.com has a tab called "Share & Export", with these options:

The key point is that all these options place the data directly into the users' hands. There is no scraping, no stripping of tags, etc.

This means that if scraping is necessary, it's usually against the website's wishes. Many websites (apparently including Meta/Facebook) want to "own" their data. These websites typically put up lots of barriers to impede scrapers. The process is not as smooth as the one I described above. For example, they can easily differentiate between a reader browsing the site and a scraper pulling down every page of every table - and they block the latter. They may use a variety of tools to hide the information from the scrapers. (You may have come across some webpages that make it impossible to copy data out of the tables - using Ctrl-C.) They may restrict the number of pages that can be viewed within a time window.

There is a bit of an arms race between scrapers and these data hoarders.

Pretty much all the big tech companies (Meta, Google, Microsoft, LInkedin, Amazon, etc.) don't want scrapers. Twitter was an exception although that is quickly changing. That's why scrapers are operating in a gray area. It may also be seen as rebellious - upholding a belief that data should be free, or that data should be owned by the people who create them.

***

From a technical perspective, data analysts should prefer to take the data directly out of databases, rather than processing scraped files. The reason is that the insertion of formatting is not always a reversible process. Take for example the following excerpt from the table of pitching statistics:

Notice that on line 9 (Chase Anderson), there is a blank space under the column "GmScA." In a proper database, missing values are not blanks but usually represented by symbols like "." or NA. When the data are ingested to the HTML table, those missing indicators are removed and replaced by blanks, which looks better on the browser.

However, the existence of such blank spaces may confuse the scraping tool. As a result, the data might be shifted, as if the blank cell did not exist. It appears easy to spot when I highlight a specific instance like this. But larger datasets may be spread out over dozens if not hundreds of tables, and the missing values can appear anywhere. Another example of an annoying feature of HTML tables is footnote symbols printed next to the data, e.g. 45.6^#. Such formatting is for browser presentation and would not exist in the underlying database.

Other than working around obstacles planted by websites, the back and forth transformation has little value and may introduce impurities.

 

 

Sloppy language misleads

One of the most frustrating aspects of reading Covid-19 related publications is reading the conclusions. A summary of results is supposed to provide readers with the minimally necessary information to understand the full contents of the paper - but what we encounter is "story time": almost always, these conclusions are stated in the most general sense, ignoring all the caveats, the exclusions, the imperfections of the methodologies and analyses. They are "stories" that go far beyond the limited evidence presented.

I've been reading the Clalit study that was a major force behind the approval of the first booster shots back at the end of 2021/start of 2022. This paper made the following one-line conclusion:

Participants who received a booster at least 5 months after a second dose of BNT162b2 [i.e. Pfizer] had 90% lower mortality due to Covid-19 than participants who did not receive a booster.

Is it possible to misread this sentence? Yes, if you literally read the sentence.

***

If we know something about vaccine efficacy/effectiveness, we immediately recognize that these authors are selling the same story - that the vaccine has VE 90%, the only difference here being the outcomes are deaths attributed to Covid-19, rather than reported cases. As such, this VE is driven by the relative ratio of the death rates of the booster group vs the no booster group. They didn't use the term VE as it typically connotes cases not deaths.

A death rate is a fraction composed of two parts: people who are counted as part of the comparison group, and people within that group who died during the study period. In this post, I'll pay attention to both these parts.

From my last post, we already know one issue with the above statement. If we take any participant at the start of the study and ask whether this person is counted as part of the booster group or the no-booster group during the study, the answer, for 90% of the participants, is both. 

Since at the start of the study, 100 percent of the participants had not received the booster, and by the end of the study period (a mere 54 days), 90 percent of them got the booster, the researchers could not have divided up the study population into two distinct groups of people - each person (except the 10% who didn't get the booster during the study period) first appeared in the no-booster group, and then transitioned to the booster group.

But the concluding statement reads like the study enrolled two distinct groups of participants. This misreading is even more likely for knowledgable readers because ordinary studies, such as well-designed clinical trials, use distinct groups.

Note that I'm not criticizing the methodology up to now, I'm describing what it is, and criticizing how the language in the summary statements sends readers the wrong message.

***

The above detail may sound trivial but it has major implications. Think about the regression adjustments. As with all observational studies, researchers claim that by adding standard covariates to their regression models (age, gender, ethnicity, prior health conditions, etc.), they have corrected substantively all biases. For example, if older participants are more likely than younger ones to want the booster, rather than being assigned a treatment at random as in a randomized clinical trial, then it's possible that the age distribution in the booster group does not mirror that in the no booster group, i.e. an age bias exists.

Look at the setup here: Ninety percent of the participants appear once in the treated group and once in the not treated group. By the design of this study, there is little variation across the comparison groups in the distribution of everything, including age. If we check "covariate balance", it's balanced because the two groups are by and large identical. So this type of studies underestimates the variability of everything - which leads to p-values that are too small, and regression adjustments that are meaningless.

***

In the paper's conclusion, the phrase "participants who did not receive a booster" is incorrect in more ways. We already know almost all of them ended up getting boosters before the end of the study period, which was fewer than two months long. I have previously said that a participant transitions into the booster group when s/he takes the booster shot.

This is in fact not true. That's because this study also imposes a "case counting window," under the claim that the booster shot takes 7 days to become effective. In other words, if someone takes a booster shot, and falls ill with Covid-19 within the first 7 days, this case is not counted as a case in the booster group.

Negating this case is one thing - but this study (and many others) took another step: the case is not negated but it is counted as a case in the no-booster group!

How do they pull this off? The study design says not to transition the participants to the booster group until the 8th day after the booster shot.

So, let's review this. Each participant starts off in the no-booster group, then the person took the booster shot; this person is kept in the "no-booster" group for an additional 7 days before being moved into the "booster" group. During these 7 days, if the person gets infected or died from Covid-19, the case or death is counted as "no booster".

Remember that we can only talk about exposure time for this type of study since almost everyone is eventually exposed to both conditions. The exposure time of the "booster" group therefore excludes the first seven days after the booster while the exposure time of the "no booster" group includes the first seven days after the booster.

This is an instance of the additional bias that Lataster described in his comment on our Three Biases paper.

***

As the story goes, even the above description is inaccurate. We haven't dealt with the following puzzling disclosure:

participants who received the booster and had a confirmed case of Covid-19 within 3 days before the effective-booster date (defined as 7 days after the booster was administered) were excluded

The analysts further subdivided the seven days after the booster shot is administered into two parts: Days 1-4 and Days 5-7. Let's follow a participiant who did not have the booster at the start of the study. During the study period, this person took the booster shot. If the person got sick during Days 1-4, the person is counted as a member of the no-booster group with infection as outcome. However, if the person got sick during Days 5-7, this person is excluded from the study, i.e. not counted as part of the no-booster group, and not counted as a case. Finally, if the person did not get sick by Day 7, the person transitions to the booster group.

There is a subtle but important difference. The Days 1-4 treatment is applied to every participant, regardless of outcome. If they got the booster, they continued to be called "no booster" for seven additional days. The Days 5-7 treatment is an exclusion based on the outcome. If the person got sick, the person is removed while if the person did not get sick, the person is retained. For me, it is never acceptable to re-define treatment groups after observing the outcomes. And yet, this paper (and many others) engages in this practice.

To understand the impact of these data processing procedures, you can use the same device we used in the Three Biases paper: imagine a vaccine that is ineffective, apply the analysis, does it result in no effect, or something else?

This is a great example of why researchers must disclose the code used to process the data. It's hard to believe these special adjustments did not get the attention of reviewers. But I can think of no other way to interpret those words. (In fact, the waterfall chart in the paper confirms the exclusions.)

***

Now that you have heard how they measure outcomes and define "no booster", it's time to revisit the conclusion:

Participants who received a booster at least 5 months after a second dose of BNT162b2 [i.e. Pfizer] had 90% lower mortality due to Covid-19 than participants who did not receive a booster.

Does this sentence correctly summarize the study?

 

 

Uninsightful by design

I saw the following chart in an article about the ordering of author names in research papers. This is a topic of great interest to researchers looking to become famous. Memorably, the article was written by two European researchers with the same first and last names.

 

The article investigates conventions of sequencing authors in scientific papers with more than one authors. The two most popular conventions are: order by significance of contribution ("importance"), or order alphabetically by first letter of last name.

It appears that the following analysis was performed to make the above chart: take all papers with exactly two authors, group by first character of first author's last name, then compute the probability that the last names are listed in alphabetical order, given the first character of the first author's last name.

There is a natural downward slope to this line: if the first author has a last name starting A, the proportion of papers for which the second author has a last name starting B, C, D,... or Z should be close to 100%. If the first author's last name begins with Z, there will be an overwhelming chance that the second author's last name starts with a letter before Z, which means that the pair of names would not be alphabetical.

This chart caught my attention because it's one of those charts that may be uninsightful by design.

***

Why can't it be insightful?

Let's start by finding a statistical model for the observed data (whether the pair of names are alphabetical). The order convention (by importance or by alphabet) is unobservable.

Assume we have a pair of authors. For a specific research paper, the pair is either listed alphabetically or not. If the authors are listed non-alphabetically, then we are sure that the order convention is not alphabetical - which for the purpose of this analysis, we assume to be by importance. By contrast, if the observed order is alphabetical, then either order convention could have generated it -- names being listed alphabetically does not exclude the possibility that they were ordered by importance!

In the article, the authors draw the following conclusion from the chart:

Statistics [green line] exceeds this baseline [bold black line] the most, indicating that many authors intentionally ordered their surnames alphabetically. By contrast, psychology [blue line] appears to be the discipline where the list of authors is mostly ordered by the maount of work contributed (i.e., the percentages are close to the baseline probabilities).

They see the signal in the gap between the value of a particular discipline and the "baseline" probability. If the proportion of papers with alphabetical ordering is higher, then these authors conclude that that discipline is more likely to prefer alphabetical ordering.

Must it be so?

***

Let's take an extreme case as a thought experiment. Assume that no publication uses alphabetical ordering. All authors are listed by importance of their contributions.

If we create the above chart, computing the proportion of alphabetically ordered names given the first letter of the first author's surname, it would look exactly the same - except that the horizontal axis should be relabeled "First letter of the primary author's last name" (instead of "First letter of the first author").

However, the gap between these lines gives us zero information about the preference for alphabetical ordering because all names are listed by importance, and some just happen to also be alphabetical. In this experiment, the gap is purely driven by the (chance?) pairing of authors.

Either the research doesn't accomplish what the researchers think it did, or I'm missing the point. What do you think?

Is it worth it to flout parking rules?

Significance magazine (link) has a fun article about parking in front of fire hydrants in New York City. It's great data journalism.

The researchers conducted a "field experiment" in which they observed "six routes through Manhattan streets" on July 11, 2022 from 1-5 pm, during which they observed 138 violations. They then merged the hand-collected data with NYC's open data recording all tickets, matching license-plate numbers to learn that only 7 of those drivers have been served tickets.

Based on that, they deduced that most drivers who park in front of hydrants get away with the violation. They computed a detection rate of 5% (= 7/138). During that period of time, the traffic cops issued 423 tickets, and therefore, they deduced that the total violations to be 423/5% = 8,460. This number includes both the 423 tickets and a further 8,037 instances for which no tickets were issued.

They now model the cat-and-mouse game between the rule-bending drivers and the cops as an "exponential game". A timer is started each time some driver parks at a hydrant. The game is to see whether the cop shows up to flag the violation first, or the driver returns to pick up the car first. The model assumes that the time until the next check is exponentially distributed. An exponential distribution is where most values are small, but a few values can be very large. Such a distribution is fully described by its average value, which is estimated to be 6 hours. On average, a cop checks a given hydrant once every 6 hours. (That's the average interval, but each interval has a different value.)

They apply the same exponential model structure for the rule-bending drivers. In this case, the time until the driver returns to a given hydrant - i.e. the duration of violation -  is exponentially distributed, with a mean of 18 minutes (0.3 hours). Comparing the two exponential distributions, we can already see that drivers who leave their cars for a short period of time (say under 60 minutes) have a low chance of getting ticketed but if they park their cars illegally for hours at a time, their risk of detection is much higher. The authors did these calculations: the probability of detection is 16% for one hour, and increases to 99% for 24 hours.

Given the above model - and we assume also that the cops and the drivers act independently, the probability that the cop arrives before the driver returns is the key statistic we care about. That's the chance of the driver receiving a ticket, which as we know from above, is 5% based on the observed data. A consequence of the exponential game model is that this number is the average time until a cop arrives divided by the sum of the average times of a cop arriving and of the driver returning. Thus, 0.05 = 0.3/(0.3+6). [This verifies that the above calculation is correct.]

***

This type of model has been deployed to analyze gambling. If one buys lottery tickets every day, what is the chance that one hits the jackpot before one goes broke?

***

What's left unexplained is how they got the 6-hour estimate for the time until detection. This is a clever bit of work. Remember the researchers matched a subset of the observed violations to actual tickets issued. For these violations, the researchers determined the duration between observing the violation and issuing the ticket to be 6 hours on average.

But... that duration is counted from the moment the researchers found the illegally-parked car, not from the moment the driver parked illegally. What's unknown is the passage of time before the researchers documented the violation.

Conveniently, that unknown duration can be ignored. This convenience comes with using the exponential model. In the exponential world, the future does not depend on the past. Let's say the car has been illegally parked for 10 minutes already. How much time will elapse before a cop shows up? The answer is the same as before: it's exponentially distributed with average of 6 hours.

***

Based on the math above, the researchers found that illegally parking makes sense. The penalty for getting caught is $115. Returning within 5 minutes limits the expected penalty to under $2, which is similar to the cost of feeding a parking meter.

 

P.S. [7/10/2023] Corrected "flout" in the title of the post. Thanks to the reader who alerted me to this.

Were the U.S. polls accurate?

Andrew chimes in to defend the polling profession (link). He's responding to a piece in the London Review of Books that claims that the polls for the U.S. 2022 mid-term elections predicted a red wave that did not materialize.

He pointed to the Economist model that is an aggregation of polls that did not insert punditry type assumptions. This model is different from, say, Nate Silver's models for FiveThirtyEight, which incorporates some subjective judgement.

The poll aggregation model did remarkably well in predicting the overall outcomes in both the House and the Senate. In the House, the model predicted 225 Republican seats (208-244) compared to the eventual outcome of 222. In the Senate, the model predicted 51 Republican seats (46-55) compared to the eventual outcome of 49.

The poll aggregation model did not predict a red wave, as the interval estimates of the number of seats for both parties substantially overlap. In other words, it predicted a close election, which it turned out to be.

***
Andrew speculated that the LRB writer thought the polls predicted a red wave because he was "sticking to an existing narrative." The journalist regurgitates the "conventional wisdom" which happens to be wrong.

Andrew distinguishes between poll aggregation and poll forecasting, the latter combining poll data with punditry. He also focuses on the aggregate outcome (number of races predicted correctly), not the outcomes of individual races (e.g., did the polls get in right in state X?).

***

Andrew's post triggered a few thoughts of mine.

I dislike the conventional metrics of accuracy that treat every race as equally hard to predict. Given that most races in all U.S. elections are uncompetitive - as in, even a chimpanzee can predict the winner - a serious metric of accuracy should only consider competitive races. This issue is similar to that of grade inflation in education: the nominal scale (A-F) is misleading when all grades are compressed into the A-B range.

Worse than sticking to an existing narrative, the journalist is reinforcing a false narrative. I see the world as comprising two types of people in their relationship to data: data-first versus story-first (link). People who are story-first starts with the story and finds data that support the story. In the case of the 2022 elections, the Republican mouthpieces were everywhere pushing the narrative of the red wave. The polling data were an inconvenience that didn't matter.

Democrats can also be story-first. In fact, some Democrats might have allowed the false narrative to persist in the hope that fearing the red wave would drive their supporters to the polls.

When I came up with the data-first/story-first framework, I wanted something that is non-judgmental. Story-first is not always wrong; indeed, the data can be consistent with the story. In my experience, story-first is the default human instinct, even though being a data person, I wish the opposite. I'm definitely data-first but data-first isn't right for every decision either, especially when the data are inadequate, contradictory, or inappropriate.

The recent U.S. elections have been so close that it's virtually impossible to predict the result accurately. Statistical models are calibrated to say exactly that. The interval estimates shown above overlap substantially. This is a situation in which the statistician would be unwilling to put a strong bet on either side. For example, in a Bayesian sense, the Republicans were found to win control of the Senate in about 60% of simulations. This means the Democrats won control in the other 40% of simulations. Real life is just one simulation!

The model hitting the right ranges of results was a matter of skill in collecting and analyzing data. The model coming quite close to real life was a matter of luck. People find it frustrating when statisticians don't give a clear-cut answer. The real issue is not the statistics, it's the difference between story-first and data-first thinking. Story-first thinking is typically black and white; data-first thinking often involves shades of gray - as Andrew likes to say, we "embrace uncertainty".

 

P.S. [4/28/2023] Reader Mark P. has some further thoughts on this topic. His point about the inutility of election forecasting reminds me of this past post.

Long story behind simple metrics

In a recent post on the dataviz blog, I featured a project by FiveThirtyEight on the impact of potential state-specific abortion bans. For comments on the visualization, please see the other blog post. This post discusses the analytics behind the visualization.

The data journalists measure the impact on women wanting abortions along two dimensions: "distance" and "congestion". Distance is the distance to travel to the nearest abortion clinic, and congestion is represented by a proxy, which is the service capacity of the said abortion clinics. (To be precise, the authors said they counted "the number of reproductive-age women that each clinic serves, which serves as a rough stand-in for appointment availability.")

I'd rather they call a spade a spade. I don't think much is lost by calling the second metric "capacity". Congestion is a result of matching supply and demand while they only have data on supply.

***

The smoothness and niceness of the dataviz hide some of its underlying assumptions (and thus biases). As consumers of dataviz, we must recognize them.

Look at the following pair of map excerpts. They both feature Arizona and its neighbors. The left panel shows the current state while the right panel is the projected state if Arizona should ban abortions. I have highlighted Pima county in which there is an abortion clinic.

In this example, the county went from the lower left corner of the color legend to the upper right. Women who have easy access to a clinic today would have to travel much further (266 miles) to find one in the future. In addition, that substitute clinic in New Mexico has a much lower capacity than the one in Pima county.

It appears that they are plotting the absolute values of distance and capacity. How about plotting the relative changes? How much further would women have to travel relative to the current state? What's the difference in capacity between the current clinic and the substitute clinic?

***

Clicking around, I dwelled on Coconino county, which showed something odd.

Notice that the color for Coconino county went from dark green (current) to lighter green (post-ban). It appears as if Coconino residents would benefit from an abortion ban. How could it be?

Digging into this reveals assumptions that the dataviz designer must make when producing such a chart. Clark county on the left panel is rated as low congestion (i.e. high capacity) while Maricopa county is rated as medium capacity. Both counties are rated low distance from Coconino county. And yet, the algo assumes that women in Coconino prefer the facility in Maricopa.

We don't have detailed data on every woman seeking service at these clinics. And if we do, we'll find that they don't all go to the same clinic. The designer is aiming for the average behavior here. Someone else may decide to connect Coconino to Clark county in the current state. These two designers would have justified their algos by arguing their assumptions.

In the case of FiveThirtyEight, there can be a number of reasons why Maricopa is preferred: state loyalty, insurer preference for in-state providers, the drive to Maricopa may be nicer, safer, and/or faster than Clark county, the capacity in Maricopa may be sufficient, the types of services offered by the two clinics are not the same, etc. etc.

It would be interesting to learn what proportions of the counties have the same predicted outcome as Coconino, i.e. the women would have benefited from an Arizona ban, at least according to the two criteria being considered.

***

There is another assumption revealed when comparing the pre- and post-ban Arizona. Notice that the colors of all surrounding states do not change at all.

This highlights what I pointed out before. "Congestion" is not congestion. It is "capacity". If it is congestion, then the fact that a whole county's women from Coconino are now expected to show up at Clark county's clinic would have caused more congestion in Clark.

***

In short, in order to make this data visualization, the designer must make assumptions about people's behavioral choices. There is an underlying optimization model that decides what women would do depending on where they live, and which states have bans. Not having such a model creates so many possibilities that the graphic would become too complex.

By how much did observational studies over-estimate Covid vaccine effectiveness

I'm happy to announce that our paper on limitations of observational (i.e. "real-world") studies of Covid-19 vaccines has been published in the Journal of Evaluation in Clinical Practice (link). I'd like to thank my co-authors (Peter and Mark) and reviewers for feedback that greatly improved the presentation, and the journal editors for the courage to print it.

In the paper, we offer three examples in which standard methods used to analyze observational data produce vaccine effectiveness (VE) estimates that are wildly off the mark. Specifically, we show that a hypothetical vaccine with 0% efficacy will be rated as 50 to 70 percent effective. The magnitude of the biases surprised us, and should surprise you too. That said, we are not claiming that Covid-19 vaccines are ineffective - as I explain below, the device of a 0% effective vaccine allows us to work around the lack of data disclosure.

The motivation for writing this paper is to encourage better science. Observational studies are hard, and the quality of Covid-19 related observational studies has been mediocre.

Two major obstacles to writing this paper were (a) the pathetic level of data disclosure for major studies that have had tremendous impact on public health policies and (b) the lack of a ground truth - no one knows the real vaccine efficacy. We found a way to work around these challenges.

For (b), we analyzed the case of the 100% ineffective vaccine. This device is very useful because if the vaccine is ineffective, it is ineffective for all subgroups of people, at all times, etc. If we assumed any other level of vaccine efficacy, say 50%, we'd have had to make a huge list of other assumptions as well. For example, there is no rationale to assume that if VE is 50% in aggregate, then it is 50% for every age group.

For (a), we opportunistically found papers that disclosed just enough data to prove the existence of specific biases. For example, to study the effect of background infection rate bias, we require data on weekly infection rates for certain subpopulations within an analysis window, which we found in a Danish study but are not typically disclosed by researchers. We didn't find a single paper that disclosed all the data required to study all three sources of bias, thus for each bias, we used data from a different reference.

In each example, we show that if the ineffective vaccine were tested in a randomized clinical trial, the typical analysis applied to RCTs would conclude correctly that VE is 0%; however, if the ineffective vaccine were analyzed using methods that were popular in early 2021, the resulting VE would erroneousy land in the 50-70% range! The magnitude of the over-estimation surprised us.

The full paper is here. Spread the word!