Big Data, Plainly Spoken (aka Numbers Rule Your World)

Exploring text summarization using LLM tools, part 4

On social media, one comes across much casual commentary on outputs of AI tools like ChatGPT of the type “Look, it wrote this beautiful poem” or “Look, it generated this amazing summary of the document”. Recently, I took a forensic look at examples of text summarization in order to provide a balanced view of the strengths and improvement directions for such tools.

The blog post is broken up into four parts, which are released all today so you can binge if you want.

Part 1: Exploring the length of summary, and selection of contents
Part 2: Exploring selection of contents, grammar, and word choices
Part 3: Exploring word choices, and hallucinations
Part 4: Exploring hallucinations

This post is Part 4 of the series.

***

HALLUCINATIONS

An acknowledged problem with LLM tools is invention of untrue facts, usually referred to as "hallucinations". I found a few examples of mild hallucinations in the six summaries I reviewed.

Claude turned these lines:

Disney has begun reporting more detailed results from its ESPN sports network as it seeks strategic partners to invest in the flagship sports network’s future.

ESPN’s operating income for fiscal 2023 fell 1.7% to $2.8 billion, while revenue rose 2% to $16.4 billion. Disney owns 80% of ESPN through a joint venture with Hearst, and Iger has said the company is working to transform the network into a fully direct-to-consumer platform, with live sports and other sports content streamed to consumers outside the cable bundle.

Excluding ESPN, Disney’s traditional TV networks saw revenue fall 9.1% for the quarter to $2.62 billion. Operating income from the networks was flat at $805 million. A weeklong standoff in September between Disney and cable provider Charter Communications over carriage rates for Disney’s cable channels and whether or not Charter’s 15 million customers should get free access to Disney+ raised concerns about the future of the cable television model.

into

Traditional TV networks like ESPN saw declines in revenue and income as the cable TV model faces challenges. Iger aims to transform ESPN into a direct-to-consumer streaming platform.

This is the first mention of ESPN by Claude in the entire summary. It was never explained that ESPN is a division of Disney. The first sentence reads like it is about the overall cable industry which includes ESPN as another key player, independent of Disney. (ChatGPT suffers from a similar issue.)

Besides, in the original paragraph, ESPN’s operating income fell but revenue rose so it is incorrect to say both revenue and income declined.

Claude also dropped the word “fully” from direct-to-consumer streaming platform, but that qualifier is crucial in this context. (Intriguingly, the one ChatGPT summary that mentions this point also omitted “fully”.)

***

In a summarization task, “hallucinations” are not just about made-up facts not found in the text, but also about true information not found in the text (the LLM may have been trained on other documents, which may give it prior knowledge that it inserts into the summary).

A recurring summarization technique used by LLMs is to reduce a long list by dropping some elements. How does it decide which elements are important to retain? I suspect that the LLMs might appeal to prior knowledge (not present in the article); if not, it may be arbitrarily dropping elements.

Let’s look at this example:

Other bright spots included Disney’s Experiences segment, which includes theme parks, cruise ships, a family-adventure travel-guide business and merchandise licensing. The unit’s operating income rose 31% from the year-earlier quarter, to $1.76 billion. Disney has raised prices at its theme parks and invested heavily in its cruise ship business in the hopes of capitalizing on rising demand for in-person entertainment experiences.

All three platforms decided that the growth at the Experiences segment is an “essential” idea. In the WSJ article, this unit is described as comprising four things, apparently too many to an LLM’s liking.

Claude renamed the unit as “theme parks and experiences segment/business”, and omitted cruise ships, family-adventure-travel-guide, and merchandise licensing from its summaries.

ChatGPT renamed the unit as “theme parks and related experiences” in one case, while omitting the other items. In the second example, ChatGPT kept the Experiences segment name but in its description, only included theme parks and cruise ships, grouping the rest into “related”.

Mistral typically makes few word changes so it’s not surprising that it retained the Experiences segment name. However, like ChatGPT, it also abridged the description, by dropping “family-adventure-travel-guide business” completely.

What is driving the inclusion/exclusion decision? Could it be the order of the items? That might explain Claude and ChatGPT but not Mistral.

***

I ran a further test. I altered the text by reordering the list of items: from “theme parks, cruise ships, a family-adventure travel-guide business and merchandise licensing” to “a family-adventure travel-guide business, merchandise licensing, theme parks, and cruise ships”. The item most dropped is now listed in front while the most well-known business of Disney (theme parks) appears in position three of four. How did the LLM tools react?

With the original order of items, in the two previous runs, ChatGPT always retained theme parks, always dropped travel guides and merchandise licensing, and sometimes included cruise ships. In the test using the new order of items, ChatGPT printed

“The company's Experiences segment, encompassing travel guides, merchandise, theme parks, and cruise ships”

so it kept all four items but reduced “family-adventure travel-guide business” to “travel guides” and “merchandise licensing” to “merchandise”. The latter contraction is troubling.

This suggests that the LLM tool considers an item listed earlier as more important. It’s still not clear whether ChatGPT has learned from other training data that theme parks and cruise ships are major revenue sources for Disney, or learned within this article as those categories have been mentioned elsewhere.

Mistral, on the other hand, responded with more or less the same output regardless of the order of those items.

“The company's Experiences segment, which includes theme parks, cruise ships, and merchandise licensing, saw operating income rise 31%...”

This suggests Mistral does not interpret the sequence of items within the list as informative. Weirdly, it reordered the items to show theme parks and cruise ships first. Mistral may be biased against long descriptors, or it uses external information to determine how to order and cull longer lists.

Finally, Claude somehow decided to not mention the Experiences section at all. This is a bit worrying since the only thing I changed in the text is the order of the items in this list, which should not affect the LLM’s opinion of whether the growth at the Experiences segment is a significant idea in the WSJ article!

CONCLUSION

LLM tools show promise in the text summarization task. They generally produce highly readable, grammatical summaries that cover most of the essential ideas. Numbers seem to have survived even when the logical structure has changed. Each LLM platform has an internally consistent idea of how much compression should be in the summary although the lengths differ significantly across platforms. Severe hallucination is absent but a milder form exists, leading to subtle changes of meaning due to word changes, incorrect inference of relationships, or turning speculative statements into observed facts. More work is needed to understand whether these instances are caused by external knowledge resident in the LLMs, or by randomness built into their structures.

Given a fixed document, each LLM platform is moderately consistent across runs, but only about two-thirds of the ideas recur while the other third appear as if at random. The last test of altering the order of four items presented as a list suggests that the outputs from these LLM tools are not robust, in the sense that small changes to the input may lead to material changes in the output.

 

Exploring text summarization using LLM tools, part 3

On social media, one comes across much casual commentary on outputs of AI tools like ChatGPT of the type “Look, it wrote this beautiful poem” or “Look, it generated this amazing summary of the document”. Recently, I took a forensic look at examples of text summarization in order to provide a balanced view of the strengths and improvement directions for such tools.

The blog post is broken up into four parts, which are released all today so you can binge if you want.

Part 1: Exploring the length of summary, and selection of contents
Part 2: Exploring selection of contents, grammar, and word choices
Part 3: Exploring word choices, and hallucinations
Part 4: Exploring hallucinations

This post is Part 3 of the series.

***

WORD CHOICES

At the end of the previous post, I noticed that the three LLM platforms treated the phrase "next year" differently. In some summaries, the phrase was directly quoted while in others, the LLM inferred what year "next year" is. But the inference was sometimes wrong.

The one summary that correctly substituted "next year" with 2024 came from ChatGPT. Score one for ChatGPT?

It’s hard to say. Here’s why. I didn’t include the date of the article in the prompt, so it’s not clear how any LLM could figure out what year “next year” is. I dug into the article, and literally found just one clue in the 1,416 words - a sentence that reads “ESPN’s operating income for fiscal 2023 fell 1.7% to $2.8 billion”.

Strictly speaking, this little clue is insufficient to conclude that “next year” is 2024. Because it’s “fiscal year”, not “calendar year”, it’s possible that “fiscal 2023” runs from December 2023 to December 2024. In other words, “fiscal 2023” does not rule out next year being 2025.

Further, we’ve subtly assumed that the article is being read in year 2023. Imagine that someone reads this same article in 2050. The only foolproof way of determining what “next year” means is to know the date the article was written, which is not found in the text itself.

This is an example of why sometimes it’s better to leave the original words alone.

***

Another word change found in most of these summaries is turning “annualized savings” into “annual savings”. As any financial analyst can tell you, they are not the same things; if so, we’d not need two different words!

Ditto with swapping “operating income” with “profit”.

One of the summaries substituted “board seats” with “board influence” when discussing what the activist investor was seeking. There are many ways to influence the board of directors, only one of which involves pushing for board seats.

Claude got creative when dealing with the reasons for better results in the Experiences segment. In one of the summaries, it uses “travel demand” instead of “demand for in-person entertainment experiences” in the original. In the other summary, it states that “demand rebounds for in person entertainment” while the original sentence said “Disney… invested heavily … in the hopes of capitalizing on rising demand for in-person…” One can call this a mild form of “hallucination”.

HALLUCINATIONS

In the LLM community, the term “hallucination” is used to describe “fake” details invented by the LLM tool that are not true. In the last section, I pointed to a case in which the article said Disney expected a rising demand for in-person entertainment experiences, which isn’t the same thing as observing a rebound in demand. It’s a subtle issue of expectation vs observation but extremely important to any reader who cares about the accuracy of the summarization task.

The good news is that by and large, the problem of hallucination is not evident in my test. No numbers were invented, and the numbers seem to have survived paraphrasing.

For example, ChatGPT printed this:

Disney has made strides in its streaming sector, reducing losses from $1.47 billion to $387 million in the recent quarter,

instead of the original text:

The business, which also includes Hulu and ESPN+, lost $387 million in the most recent quarter, down from $1.47 billion a year earlier.

That's nice.

With respect to hallucinations, I found a few sentences to explore further. I'll cover these in the next post of the series.

Exploring text summarization using LLM tools, part 2

On social media, one comes across much casual commentary on outputs of AI tools like ChatGPT of the type “Look, it wrote this beautiful poem” or “Look, it generated this amazing summary of the document”. Recently, I took a forensic look at examples of text summarization in order to provide a balanced view of the strengths and improvement directions for such tools.

The blog post is broken up into four parts, which are released all today so you can binge if you want.

Part 1: Exploring the length of summary, and selection of contents
Part 2: Exploring selection of contents, grammar, and word choices
Part 3: Exploring word choices, and hallucinations
Part 4: Exploring hallucinations

This post is Part 2 of the series.

***

In the previous post, I classified ideas from the original Wall Street Journal article into three groups: Essential ideas, Primary ideas and Secondary ideas. Primary ideas are those considered important by a specific platform, showing up in both runs, but not essential, meaning that at least one platform excluded them in both runs.

Because Mistral uses fewer words, its summaries must have ignored some of the primary ideas retained by Claude and ChatGPT. What are these?

• The entertainment giant said on Wednesday in its earnings report that it is seeking $7.5 billion in annualized cost efficiencies, up from the $5.5 billion it targeted at the beginning of this year.
• Disney […] generated sales of $21.2 billion for its fiscal fourth quarter
• The company’s main streaming service, Disney+, added 6.9 million “core” subscribers…
• Overall, Disney+ ended the quarter with 150.2 million global subscribers,
• ESPN’s operating income for fiscal 2023 fell
• Excluding ESPN, Disney’s traditional TV networks saw revenue fall

Mistral omissions relative to the other platforms fall into three types: overall performance indicators (revenues, operating profit, etc.), specific business divisions (ESPN, TV networks), further details on key business divisions (Disney+ subscriber numbers). Of these, Mistral treated overall performance indicators as "secondary" while completely omitting news about ESPN, TV networks or Disney+ subscriber nambers.

Ponder the first bullet for a second. Five of the six summaries did not mention “earnings report” or “Wednesday,” that is to say, they excluded the context of the article. The one summary that did came from ChatGPT, in which it said, in its last sentence, “Disney’s stock rose after hours following the earnings report, even as it deals with possible new activist campaigns for board influence.” I’m giving ChatGPT a merit, even though I suspect the inclusion of “earnings report” is more accidental than intentional.

Score one for Mistral for being the only platform that considered narrowing streaming losses as a primary idea. Both Claude and ChatGPT dropped what was described in the opening line as one of the “two areas … crucial to the company’s future”. While all summaries addressed the reduction of losses and the projected breakeven in the streaming business at some point, only Mistral elevated the streaming business challenges as a key area of concern.

GRAMMAR

One of the most exciting areas of progress in text processing is the computer’s ability to write grammatical sentences. The six summaries were largely free of grammatical mistakes. Here are several minor mistakes:

ChatGPT #1 wrote:

“Disney+ lost subscribers in India and is considering selling its unit there after failing to secure cricket streaming rights.”

In the original paragraph, it’s clear that Disney is the entity selling the service in India while Disney+ is the entity that lost subscribers.

In compound sentences, ChatGPT appears to have trouble keeping track of the subjects. Another example is

“The company is also seeking strategic partners for ESPN as it transitions to a direct-to-consumer platform.”

What is the “it”? In reality, the company is transitioning ESPN to something. (Besides, the article did not ever associate the strategic partner search with the direct-to-consumer transition.)

ChatGPT #2 also got off rail when talking about the India service. It included

“Despite losing subscribers in India due to lost cricket streaming rights, Disney is considering selling its India unit.”

Instead of “despite”, it should have read “After” or “As a result of a subscriber attrition in India due to …”

An inference mistake was also found in one of the Mistral summaries, which included

“The company's Experiences segment, which includes theme parks, cruise ships, and merchandise licensing, also saw strong growth.”

Note the erroneous insertion of “also”. The previous two sentences were about the losses in the subscription business so “also” sounds out of place. Either omit it or use something like “however”.

WORD CHOICES

Another dimension in which the platforms differ is the degree of paraphrasing (vs direct citation). Mistral summaries tend to lift entire phrases out of the original article, with few changes to wording and phrasing. At the other extreme, ChatGPT likes to rewrite as much as it can.

I favor quotation as word changes often create subtle (and sometimes unwanted) shifts in meaning. However, there are real concerns about plagiarism and copyright that must be addressed with this type of writing aid.

Let me just give a colorful example of word changes to illustrate the risks.

Recall one of the essential ideas of the article:

“The company […] reiterated that it believes streaming will break even by September of next year.”

The Mistral summaries pretty much print this line verbatim. Both Claude’s summaries proactively substituted “next year” with “2023”. But that is the wrong inference! Next year is 2024.

ChatGPT gives different results in each summary: in the first run, it retains “next year” like Mistral; in the second run, it substitutes “next year” like Claude. However, ChatGPT got the year right.

Score one for ChatGPT?

Not so fast, as I'll explain in the next post of the series.

Exploring text summarization using LLM tools

On social media, one comes across much casual commentary on outputs of AI tools like ChatGPT of the type “Look, it wrote this beautiful poem” or “Look, it generated this amazing summary of the document”. Recently, I took a forensic look at examples of text summarization in order to provide a balanced view of the strengths and improvement directions for such tools.

The blog post is broken up into X parts, which are released all today so you can binge if you want.

Part 1: Exploring the length of summary, and selection of contents
Part 2: Exploring selection of contents, grammar, and word choices
Part 3: Exploring word choices, and hallucinations
Part 4: Exploring hallucinations

This is part 1 of the series.

***

The running example in this series of blog posts concerns this Wall Street Journal article on Disney’s latest financial results (link). I asked the LLM tools (large language models) to

“Please summarize the below, and be concise and accurate.”

I imagine myself to be a developer considering building an app using the outputs of the LLM tool, and therefore, I’m seeking a level of quality that rises above the casual observer using it as a toy.

I asked four different LLM platforms to summarize the same article. On each platform, I made two runs, as these LLM tools incorporate pseudo-random behavior designed to generate different outputs each time. The four platforms tested are ChatGPT (from OpenAI), Claude (from Anthropic), Zephyr (from Huggingface) and Mistral.

[Note: You should consider these blog posts as exploratory research. I'm using a small sample. At this initial stage, it's important to look into details, rather than in aggregate.] [To be specific, ChatGPT and Claude results were obtained from their web client. Zephyr was 7B Beta, and Mistral was 7B Instruct v1 AWQ.]

The platform that makes the biggest first impression is Zephyr, in a head-scratching way. Instead of a summary, Zephyr sent back an invented Q&A script, like this:

According to the article, how did Disney's core streaming service, Disney+, perform during the latest quarter? What factors contributed to its success?

Answer: During the latest quarter, Disney's core streaming service, Disney+, added 6.9 million "core" subscribers (excluding India), bringing its total number of global subscribers to 150.2 million. This was more than double what analysts had predicted, and Disney highlighted the popularity of recent movies like "Pixar's Elemental," "Guardians of the Galaxy Vol. 3," and "The Little Mermaid." Average monthly fees for Disney+ increased slightly compared to the previous year. These factors contributed to Disney+'s success. However, the article also mentions that Disney is exploring selling its India unit due to subscriber losses following a bid for cricket league streaming rights.

This might be because this LLM platform hasn’t been trained to do text summarization but it has been trained to produce Q&A scripts from a document. For the rest of this blog post, I exclude Zephyr.

***

The other three platforms dutifully produced summaries of the WSJ article. At first glance, they all look pretty good. They certainly beat my expectation of what a computer could generate. Here is one of the Mistral summaries:

Disney has announced a more ambitious plan for cost cuts and has made progress in its streaming business, narrowing its losses. The company's streaming business, which includes Disney+, Hulu, and ESPN+, lost $387 million in the most recent quarter, down from $1.47 billion a year earlier. Disney has reiterated that it believes streaming will break even by September of next year. The company's Experiences segment, which includes theme parks, cruise ships, and merchandise licensing, also saw strong growth. Disney shares rose 4.65% in after-hours trading.

[If interested, you can download all six summaries here.]

Of the six summaries I’m comparing, the within-platform differences (two runs) are much smaller than the between-platform differences (three platforms). In most of what follows, I focus on between-platform differences.

LENGTH OF SUMMARY

The most noticeable difference between platforms is the length of the summary. From the most concise to the most verbose: Mistral > Claude > ChatGPT.

Since I did not ask for a specific length, each platform decides the number of words to use. This is an important metric because the more words in the summary, the more details are included, the less loss of meaning. The original article has 1,416 words (which includes a small amount of stray text insertions like captions and so on). The Mistral summary #1 shown above uses 85 words, which gives a compression rate of 95%. The ChatGPT summary #2, the longest of the six, uses 284 words, more than three times as long as Mistral #1, and the compression rate was 80%.

Of course, longer summaries are not as summarized, and potentially less useful. In the future, I will run another test specifying different word limits.

SELECTION OF CONTENT

The differing lengths of the generated summaries implies that the different platforms selectively retained different parts of the article. I divided the contents into three groups: “Essential,” if an idea is found on all six summaries; “Primary,” which are ideas present in both runs of the same platform but not present in all platforms; and “Secondary,” which includes ideas that appear in only one of two runs from the same platform.

Essential ideas are present in all six summaries.

Primary ideas are ideas that each LLM platform considers important enough to include in both runs - but they are not essential, meaning that at least one platform chose not to mention them at all.

The long summaries generated by ChatGPT included many ideas that are not found in the Mistral summaries.

Secondary ideas are ideas that pop up in only one of two runs, by platform.

Note: all words in the above table are direct, unaltered quotations from the Wall Street Journal article. As discussed in a future post, certain LLM platforms tend to paraphrase rather than quote directly.

***

I’m concerned that the proliferation of secondary ideas found in just two runs of each platform. This suggests that these platforms do not have a consistent model of what’s important and what’s not in a given article. Of the essential ideas, I’m not convinced by the inclusion of the Experiences segment.

In the next post of this series, I'll examine whether Mistral should have included some of the primary ideas from the other two platforms.