Big Data, Plainly Spoken (aka Numbers Rule Your World)

Causal skimming

The Wall Street Journal reports that all the rich people in New York City are moving downtown. The headline claims "New York's Wealthiest All Want to Live Downtown Now". (link) I'm not surprised by this statement, I want to believe it, but as a data scientist, I'd like to know how they prove it.

I suppose they can run a survey and interview a bunch of rich people. I imagine that might be a tall order, as rich people might not be very responsive. Perhaps they found their way to a wonderful dataset that shows the movement of rich people. That's not out of the realm of the possible. We know that someone was able to trace private planes around the world (link). I imagine the dataset would be quite small, as uptown/downtown NYC are not the only desirable addresses for rich people. Maybe they have good data on housing inventory. Let's find out.

There are two classes of relevant evidence. First, I'm looking for evidence of the trend. Second, for any causal explanation they provide, I'd like to see evidence of those linkages.

***

Let's skim through this article, and I'll mark down each piece of evidence I encounter.

A financier sold a West Village apartment for $60 million recently, which the couple had purchased for $29 million nearly a decade ago. This is a "record" for downtown Manhattan.

This is presented as evidence for the trend. What kind of evidence is this?

It's an anecdote. It's an extreme value of an anecdote. This anecdote uses house prices as a proxy for people movement - it assumes that if house price increases, then a wealthy person has moved from uptown to downtown. In fact, the $60 million record doesn't mean anything unless we also know that the $29 million was not extreme.

Let's keep going.

The buyer worked at Jane Street Capital, a quant trading firm, which has recently expanded its headquarters located nearby.

Already, the journalist has made a causal claim - that the downtown migration trend is due to rich folks wanting to live close to their work. The evidence for that is another anecdote. It'd have been more convincing had Jane Street moved its HQ from uptown but that's not what happened.

Downtown real estate prices are starting to "rival the city's most expensive uptown enclaves".

This assertion is not supported by any statistic. We don't know how wide the gap was, and how much it changed. Should there be data, we'd have an idea which districts are considered uptown and which downtown.

"As finance and tech firms have migrated downtown—alongside new retail, parks, and cultural institutions—a new wave of luxury condo development along the West Side Highway is luring wealthy buyers to neighborhoods like the West Village, Tribeca and Chelsea."

I left the above as a direct quote. This one citation contains a few more causal assertions, with zero data. Notably, the anecdote of Jane Street Capital should not even count because Jane Street did not "migrate" downtown. In subsequent paragraphs, they mention the names of some of these new developments but they never named any new retail, parks and cultural institutions.

Next:

A real-estate agent says that wealthy people are "buying up the West Side Highway".

Expert testimony. It's possible that these experts know what they are talking about, I'll give them that. Still no data or statistic in sight.

"There were more $30 million-plus home sales below 34th Street in the past five years than in the previous decade, according to data from Corcoran Sunshine Marketing Group. Since 2023, the area has seen more than $1 billion worth of home sales above $20 million."

Now arrives the first bit of data. The first sentence compares downtown sales across two periods of time. This can't prove a preference for downtown since we don't know the trend in uptown neighborhoods for those time periods - it's totally reasonable to believe that uptown prices may also have inflated a lot during the last 15 years. The second sentence gives a snapshot value of home sales in one time period, which does little to support the claimed downtown migration trend.

Since we are talking about economic value stretching back 15 years, those figures ought to have been inflation-adjusted. Another problem is the varying definition of downtown - two paragraphs back, a different sentence mentions "sales and listings below 14th Street".

Three more sales transactions are cited, two with just the current sales price, and one with both current and prior sales prices.

This raises the count of anecdotes to four. Since each transaction has a buyer as well as a seller, we'd also need to know where the seller moved to. If buying a home suggests preference for the neighborhood, does selling a home indicate preference for some other district?

A set of current listings with sky-high listing prices are mentioned.

Assuming these homes eventually find buyers at the listing prices, they would add to the anecdotal evidence.

"Downtown has long drawn wealthy buyers".

I'd label this anti-evidence. This statement undermines the claim of a recent trend.

The next few paragraphs run down a list of financial and tech firms that have set up shop in Hudson Yards. 

The alleged link between employment and residence is questionable because we are talking about $50 million homes, not $5 million homes. Sure, Google is "one of the weathliest corporations" in the world but how many Google employees can afford a $50 million home?

The supply of homes for the ultrawealthy is limited downtown. New developments are making larger homes with luxe amenities comparable to those uptown.

This also works as counter-evidence. Are prices downtown increasing because of shifting patterns of demand, or is it because these homes are larger and have better amenities? This section further muddies the causal picture. 

***

Let's summarize. The entire article contains one paragraph that cites two statistics, there are a bunch of anecdotes, and there are lots of unsupported story-telling.

Regarding the existence of a downtown migration trend, all the included evidence is anecdotal. We only learned that some rich people paid a lot of money to buy downtown homes. We don't know where they moved from. We don't know where the sellers moved to. We don't know the trend in real-estate prices uptown, with the unspoken assumption that they did not rise, or did not move as drastically. In the end, we're asked to trust the experts.

Regarding why rich people are preferring downtown, the main causal explanation is tech and financial firms relocating downtown. It would have been useful to know, for example, what proportion of these workers own $50 million homes. Several other causes are also mentioned, such as larger homes, homes with better amenities, new parks and cultural institutions. All evidence is anecdotal, and we again must just trust the experts.

 

 

 

 

 

 

 

 

Running out of matches

There is one other curious thing about the matching procedure used to pre-process observational data in the paper analyzing the effect of the Omicron booster from my previous post (link).

As noted, in the table showing pre-matching statistics, there were 3.8 million people who didn't take the booster against 588,000 who did. So, there were about 6.5 non-boosted people for each boosted person. The matching task is to find one non-boosted person to pair up with each boosted person. We therefore expect the vast majority of the 3.8 million non-boosted people to be dropped from the study population.

Now, in the article itself, it said that 511,352 unique control participants were matched. This implies that over 10% of these matched non-boosted people were matched to multiple boosted persons. This usually happens when there aren't enough untreated persons available for matching.

However, they had on average 6.5 untreated available to match to each treated. So there didn't run out of unique untreated people. The next possibility is that the treated and untreated groups were sufficiently different along certain dimensions that the same untreated person must be used multiple times. For example, if there were 10 women aged 95+ with certain set of preexisting conditions in the boosted group, and there were only 2 such people in the non-boosted group, then the same two non-boosted women would appear on the other side of the pair for all 10 matched pairs. In other words, the more re-use there is, the stronger the suggestion that the two groups had some major differences.

In such circumstances, I'd prefer to drop the treated person from analysis rather than use an untreated person multiple times.

Think about this scenario. Let's say there are a particular demographic segment with 5 treated and 1 untreated. The one untreated person will be part of all five pairs. Now, assume that the untreated subject died. That death would have been replicated five times after matching. The matching turned a single death into a 100% death rate in that segment.

Alternatively, assume the untreated person did not get Covid during the observation window, but 2 of the five treated did. Replicated five times, the estimate for the untreated group will be zero cases, out of five.

Matching sounds simple, but details matter. Most published studies don't disclose nearly enough for an outsider to figure out the quality of matching.

Loose lips sink hearts

A reader sent me a new study from the Veterans Health Administration about the Omicron Covid boosters, introduced in September 2023 (link). These boosters were one of the first of many “boosters” that were reformulated vaccines targeted at new coronavirus variants, all of which were approved by the FDA without conducting clinical trials. The VHA researchers used real-world data to measure the vaccine effectiveness (VE) of these boosters. In short, they found these boosters lacking, with negative VE against cases, and low VE against hospitalization or death, topping out in the 30% range but waning to below 10% within the first six months.

Reading this report brings back unpleasant memories because they contain some of the same loose language that encourages sloppy thinking, misleads people, and hides iffy science. I’ll be liberally quoting from the above linked paper but the same type of writing can be found in many similar papers I have read during and after the Covid pandemic.

First, a quick refresher on observational data. Researchers pull health records, which include people’s vaccination and medical history. Without a clinical trial, people self-selected into treatment, that is to say, they decided on their own to take the booster shot (or not). Therefore, any observed difference in outcomes could be partly due to inherent differences in the comparison groups (technically called confounding). The random assignment of treatment, as implemented in a clinical trial, ensures that the comparison groups are statistically identical, removing this complication. A good analysis of observational data starts with a reasonable model of the self-selection of treatment, followed by effective adjustments to deal with these confounders.

***

Now, I’ll point out three of the worst cases of loose language in these Covid studies:

“Exact matching”

One of the more popular techniques for dealing with confounding is “matching”, a procedure designed to artificially create pairs of treated and not treated subjects, with the hope that they might behave as if treatment were randomly assigned. To match pairs of subjects requires a set of matching variables, typically variables such as gender, age, and preexisting health conditions.

There are many matching methods. One such method – popular because it is easy to describe – is known as “exact matching”. For example, if we have a vaccinated subject who is a 59-year-old male diabetic, we want to pair him up with an unvaccinated subject who is also a 59-year-old male diabetic. Each matching variable must match exactly, age to age, gender to gender, diabetes condition to same.

But … in the cited paper, what is tagged as “exact” is not actually “exact”. For example, matching on age is done using broad age groups, rather than single-year age. Only three age groups were used: 18-64, 65-74, and 75+. So someone who is 30 can be matched to someone who is 60, which is not what one can reasonably describe as an exact match.

Similarly, another matching variable is called “CAN score”, which is a predicted probability of dying within the next year. This probability score is also grouped into three broad buckets (0-50, 51-89, >=90) before matching so that someone who has 51% chance of dying can be matched to someone with 80% chance of dying. In this paper, that’s an “exact” match.

“Unvaccinated”

Every Covid study compares a “vaccinated” group to an “unvaccinated” group but in almost every case, the unvaccinated have taken at least one Covid vaccine shot – they just haven’t taken the particular vaccine shot that is the subject of the study.

In the above linked paper, they studied the Omicron vaccine shot. But to be eligible for inclusion, a person must have “received at least one documented Covid-19 vaccine in the VA healthcare system at any time in the past”. Therefore, what they label as “unvaccinated” are vaccinated people who decided not to get the Omicron booster. What’s wrong with calling the treated group “Had Omicron booster” and the untreated group “No Omicron booster”?

Take a look at this table header from the article’s Supplement:

If you haven’t read all the details in the paper, you’d assume that they are comparing one group of people who have taken an updated Covid-19 vaccine against another group of people who have had “no vaccination”. But then you’d have misread the study.

The conclusion

Almost all conclusions of these Covid studies mislead through over-generalization.

In this paper, the conclusion stated:

“COVID-19 vaccines targeting the XBB.1.5 variant of Omicron were not effective in preventing infection and had relatively low VE against hospitalization and death, which declined rapidly over time.”

This conclusion only applies to (a) the people in the VHA system who (b) meet the eligibility criteria who (c) look like someone in the matched groups.

If a clinical trial were done by the VHA with the same eligibility criteria, then the (a) and (b) restrictions would still hold while (c) could be dropped because there would have been randomization in place of matching.

One key fact that is always glossed over is that the matched subpopulation does *not* look like the population. In matching, each Omicron vacinee is matched to someone who didn’t take the booster. So after matching, the no-booster group looks just like the booster group. But, the no-booster group post-matching does not look like the no-booster group pre-matching! The magic of matching comes from dropping no-booster people who have no counterpart in the booster group.

All of the above is reflected in the statistics included in the paper.

The following table (relegated to the Supplement) displays the pre-matching statistics.

One immediately learns that the no-booster subpopulation is 4 times as large as the Omicron subpopulation, that self-selected Omicron subjects are much older, less likely to be female, much more likely to have pre-existing comordibities, more likely to have many recent primary care encounters, and also more likely to have gotten a Covid vaccine within the last year.

What happens after matching? The two groups now have the same number of subjects. The column showing the Omicron group doesn’t change while the no-booster column looks almost the same as the Omicron group. (This is found in a Table in the paper.)

Take age groups for example, the post-match, no-booster distribution is 27%, 33%, 40% for 18-64, 65-74, and 75+, which is exactly what it is for the pre-match, Omicron group. But the pre-match, no-booster group’s age distribution is 48%, 23%, 29%! Evidently, those dropped from the analysis who didn't take the booster don't look like those who were matched.

Importantly, the result of this analysis of matched pairs can shed light only on the effect of the Omicron booster on those who choose to take it, which is only about 25% of the entire population (under study). Matching is needed precisely because the no-booster group have different characteristics than the booster group (as seen in the pre-matching statistics); therefore, whoever is in the subgroup of no-booster who look like those in the booster group cannot be generalized to the unmatched no-booster people.

I'm not trashing the matching methodology. It's a very useful technique for analyzing observational data. It just doesn't generalize in the way that many researchers imply.

***

It's sad that paper after paper pushes misunderstanding by using loose language. We can and should do better.

 

From elevator music to causal thinking

Today, I'm going to be the person who believes everything shown to me in the office building's elevator.

According to some study, they claimed that drinking sparkling water can help people reduce weight - but only if drinking it is combined with having sufficient exercise. (This went by fleetingly so I'm missing part of the paragraph.)

The scientific claims can be stated as:

• Drinking sparkling water by itself does not lead to weight loss
• Drinking sparkling water in addition to exercising leads to weight loss

So, if we trust that they did a proper study, then we should be stocking up on sparkling water while also renewing our gym membership.

That sounds like a reasonable action plan that takes advantage of the study's findings.

Wait, but we are ignoring another plan that is also fully consistent with those two points. Namely, that we just renew our gym membership.

That's because we don't have information on the following:

Does exercising alone lead to weight loss? (Presumably yes). Does exercising plus sparkling water lead to more weight loss than exercising alone?

***

We can set up the following grid:

The elevlator blurb provides information about the top left cell. It's saying the total effect of both exercise and water is positive in a meaningful way.

Our first question concerns the bottom left cell - is the contribution by exercising positive in a meaningful way? If yes, then we'd like to know what proportion of the total effect it accounts for.

If exercising has no effect, then the total effect is just the effect of drinking sparkling water. If exercising has a negative effect, then the total effect under-estimates the effect of water alone. Both these cases can be excluded since we were told that water alone is insufficient.

The two claims lead to x(w) is close to zero, and x(e) + x(w) is dominated by x(e).

If most or all of the total effect is due to the effect of exercising, then the claims above are "true lies". While it is true that the total effect is important, most of all of the effect is due to exercise, and not drinking water.

***

The grid above uses a very simple model typically found in many medical studies but it's a overly simple model. Let's look at a more realistic grid:

The key change is the quantity inside the top left cell. Previously, it's the sum of the individual effects. Now, the sum is modified by an additional term called the "interaction effect", which depends on the values of the individual effects.

The interaction term enables synergies, positive or negative. The sum of 2+2 can now equal 5 or 3 whereas in the first model, 2+2 = 4 always.

For positive synergies, the total effect is larger than the sum of the individual effects; for negative synergies, combining the two effects results in a smaller total.

***

Let's consider this possibility: drinking sparkling water by itself is useless but drinking together with exercising triggers some special synergy, indicated by the fact that the total effect is larger than the effect of exercising alone. This scenario is not admissible in the first, simple model described above.

Let's clarify why the second model can capture reality better. Let's fix a model of the interaction effect, x(e,w) = k x(e) * x(w). This says that the modification term depends on the product of the individual effects, plus a factor k (which can be any number or fraction).

Under that model, we can see that the total effect can be written as x(e) + x(w) [1 + k x(e)]. This means that while the effect of water alone is x(w), the effect of water in the presence of exercise is x(w) modified by a factor, and that factor's value depends on the effect of exercise. Simplified, we're proposing that the effect of water varies based on the effect of exercise.

Similarly, we can write the total effect as x(w) + x(e)[1+ kx(w)], and confirm that the effect of exercise varies based on the effect of water.

In the first model, the effect of water is always x(w) regardless of exercise, and the effect of exercise is always x(e) regardless of water.

 

 

 

Wishful thinking using statistics

In this piece (link), John Burn-Murdoch at the Financial Times (he's the guy who made some nice Covid-19 graphs) advances a theory of "peak obesity" saying that the new weight-loss drugs will reverse the trend of increasing obesity around the world, pointing to the recent CDC release showing a slight dip in U.S. obesity rate as evidence.

Here is a funny quote from his article:

More likely than not, this will prove another case of “where the US leads, others will follow”. In Denmark, home of Ozempic and Wegovy creator Novo Nordisk, 3 per cent of adults were using the new drugs by the end of 2023. The decades-long climb in the obesity rate slowed to a crawl that same year, and declined among several age groups.

When only 3 percent of Nerwegians are using these drugs, the impact on the population-wide number is highly limited. I don't find this argument convincing.

***

Imagine one of these patients whose use of these drugs has directly lowered the obesity rate in Norway. The person has to be obese, and is taking the drug to treat obesity (rather than treating diabetes, its original use). The person has been taking the drug for long enough time for the effect to become detectable, probably a year, depending on how the original studies measured impact. Because the flattening of the obesity rate line happened during 2023, this person must have started the treatment before 2023, or in the early part of 2023.

The person must not have stopped taking the drug. The person must have lost enough weight so as to cross the boundary from obese to not obese (BMI = 30). In other words, the more obese the person is, the more unlikely this person's improvement can move the average obesity rate down because as far as I know, Novo Nordisk is claiming around 15% weight loss, and not claiming that an extremely obese person could become non-obese after a year of Ozempic or Wegovy. A severely obese person is defined as having BMI over 40. A weight drop of 15% reduces BMI of 40 to 34, which is still above the obesity threshold of 30. For simplicity, we can think of someone with BMI 30-35 as most likely to help lower the obesity rate if their BMI drops below 30 as a result of taking the drugs.

Further, as with all drugs, only a portion of those who take them would realize benefits because no drug is 100% effective.

What we know is 3 percent of all adults were taking the drugs at year end 2023. Is that a large enough slice of the population to affect the obesity rate? I doubt it.

***

Last but not least, there is a hidden model behind the FT claim - which explains the change in obesity rate using a single factor of weight-loss drugs. This model assumes that the people who successfully lost weight after taking these drugs did not undertake other activities (such as exercise or dieting) which also may lead to weight loss. Since this is data analysis, not a designed study, I don't think we know what other measures people have taken.

Aging well, book-wise

2024 is a year that has brought me quite a bit of satisfaction. More precisely, world events have elevated the relevance of my two books, which is energizing. When I developed the themes of these books, I consciously sought topics that have enduring value.

The second chapter of Numbers Rule Your World (link) describes one of the grand successes of modern statistical science: the investigation of foodborne disease outbreaks. Toward the end of 2024, the U.S. has seen a spate of food recalls due to outbreaks of E.coli, salmonella, listeria, etc. (cucumbers, frozen waffles, frozen chicken, frozen cookie dough, carrots, deli meat, ...), the most famous of which was the McDonald’s E.coli outbreak of November 2024. Investigators believed that the outbreak was caused by contaminated onions served at McDonald’s stores (link). How epidemiologists make such determination is the subject of (half of) Chapter 2. The spinach outbreak featured in the chapter represents a best-case scenario, in which the investigators were able to confirm their cause-effect hypothesis by finding contaminated spinach at its source that tested positive for the exact strain of E.coli that caused the outbreak. As I wrote on this post (link), the McDonald’s investigation was not as lucky. The cost-benefit of food recalls deserves more scrutiny.

During the hurricane season, I revisited a key question of Chapter 3 of Numbers Rule Your World (link): is hurricane risk insurable? The insurance market is a game of musical chairs, in which players shift the risk around, and once in a while, the music stops, and a few get scorched. Many large insurers have already exited the scene, while small startups keep popping up, take incentives from the government, get run into the ground, and then bailed out by the government. See this blog post for more.

Numbers Rule Your World (link)’s Chapter 4, half of which concerns steroids testing in professional sports, became relevant again this year, as the top tennis player of either gender has become embroiled in controversies due to positive tests (link; link). At the time of writing, my take on steroids testing, that false negative rather than false positive is the Achilles heel of the testing programs, was controversial; later, Lance Armstrong got snitched and subsequently confessed, which laid waste to hundreds of “negative” tests over his long career (hello Holger Rune). Both tennis stars got a slap on the wrist because the governing bodies accepted their alternative explanation of contamination. Accidental contamination, and a myriad of other excuses, have always followed positive tests; it’s one of those applications that reveal the limit of statistical evidence: we can prove only the likelihood of foul play; to really nail a perpetrator, we need a confession, or physical evidence (e.g. caught with blood bags). My post about the tennis controversy is here.

***

Inflation continues to run unimpeded in New York City, and probably also where you live. The place where I used to get croissants (cornetti, to be geographically correct) now charges $7; it was $6 a couple of months ago, $5 a few years ago, and $4 when they opened about 10 years ago. (Those prices don’t include tax and tip, which adds at least 25%.) If you do the math, an increase from $4 to $7 in 10 years implies an average growth rate of 6% per year; and if you look up official statistics (link), the CPI index moved from 237 in Nov 2014 to 316 in Nov 2024, suggesting a growth rate of 3% per year. Chapter 7 of Numbersense (link) explains why the official statistic deviates so dramatically from lived experience. It demonstrates a very important key to success in data analytics: the analyst must know inside-out how numbers have come into being.

Chapter 6 of Numbersense (link) addresses the other big economic statistic, unemployment, in which, again, the official statistic seems to misrepresent reality.

Obesity drugs are all the rage, being pitched as the ultimate solution to the obesity crisis in the U.S. Chapter 2 of Numbersense (link) looked at the measurement of obesity, and treatments ranging from diets to surgery. CDC recently announced that the obesity rate of Americans has apparently plateaued, with the latest number showing a (statistically insignificant) slight decline (link). People were quick to attribute this to the rise of obesity drugs; thus, they are making the case that obesity drugs are causing a fall in average BMI, which is the standard metric used to measure obesity. I wonder how many of these people complained about using BMI to measure obesity during an earlier time when most treatments appear to be ineffective. The jury is still out on whether these drugs lead to long-term weight loss, which is one of the toughest tests of obesity treatments.

***

If you're looking for a gift this holiday season, get Numbers Rule Your World (link), get Numbersense (link), or get both!

Contaminated science

 

 

In one sense, the E.coli outbreak attributed to McDonald's has ended, as reported cases have dwindled to a drip. In another sense, it hasn't ended, because the number of cases is still growing.

This is the challenge of foodborne disease outbreak investigations. It is still possible that the theory that onions caused the outbreak is a false lead. In the most successful investigations, scientists were able to isolate samples of contaminated foods which contain the specific strain of the bacteria causing illnesses. An example of a very successful case is the spinach investigation featured in Numbers Rule Your World (link).

***

Late last week, CDC put out a new press release (link), updating the public about the current investigation. But it manages to unnerve me with terrible messaging.

The communication on November 13, 2024 carries the following conclusion:

"Epidemiologic and traceback data show that fresh, slivered onions served at McDonald’s were the likely source of this outbreak. Affected onions served at McDonald's locations were recalled and are past shelf life."

The sentences point to two types of evidence: epidemiological data come from questionnaires sent to reported cases asking for information such as what foods were consumed prior to getting sick from E.coli; and traceback data require salvaging potentially contaminated ingredients at source and testing them in labs for the specific strain of E.coli associated with the current outbreak.

We already knew about the epi data. What's new in this press release is the traceback data. Epidemiological data always come first, and were used to point the finger at McDonald's, then Quarter Pounders at McDonald's, then fresh, slivered onions in Quarter Pounders at McDonald's.

Scroll to the bottom of the press release, and this is the entire paragraph published about traceback:

FDA tested recalled onions and one sample tested positive for non-O157:H7 Shiga toxin-producing E. coli. The E.coli found in onions was not the outbreak strain and no human illnesses have been linked to this strain. Additional testing results will be released when available by FDA.

It's a word salad worthy of U.S. presidential candidates.

Translation: the investigators did not find the specific strain of E.coli associated with this outbreak in the onion samples they collected from the farms. We don't know how many samples they tested but only one of them tested positive for any kind of E.coli, and unfortunately for the investigation, it wasn't the right strain. Thus, this positive result does not strengthen the evidence against onions from those farms, and their customer, McDonald's.

And yet, whoever wrote the summary paragraph inserted the false claim that epi and traceback data were consistent, and complementary.

As a scientist, I find that this type of communciations undermines my confidence in our public health organizations. One can say this is an instance of "story time". They lull us in with some data, and then spin a story ostensibly supported by the data but not really.

Indeed, most media outlets have swept the uncertainty under the rug, referring to this outbreak as the McDonald's outbreak, or the onions outbreak. The CDC press release actually only used the term the "likely" source.

***

For giggles, I asked ChatGPT whether the lab test results added to the evidence that onions were the cause of the outbreak, and here is its answer:

No, the lab testing did not directly add to the evidence that the onions were the cause of the E. coli outbreak. The epidemiological evidence was already strong, with 99% of the interviewed individuals reporting eating at McDonald's, and 84% specifically eating menu items containing fresh slivered onions. The lab testing of recalled onions found a non-O157:H7 strain of E. coli, which was not linked to human illnesses and did not match the outbreak strain. Therefore, while the testing confirmed that the onions were contaminated with E. coli, it did not provide direct confirmation that the specific strain responsible for the outbreak was linked to the onions. The epidemiological data (such as the high proportion of sick individuals who ate the onions) was the primary evidence supporting the conclusion that the onions were the source.

I like this answer!

***

Note: It's possible that the lab testers knew something else that links those tested onions to the outbreak but nothing about the lab test results as disclosed in the press release supports its summary paragraph.

Note 2: In the meantime, the economic damage of food recalls is real. McDonald's is reportedly spending $100 million to recover from the fallout (link).

 

Nate Silver says no to early voting data

Nate Silver, now writing his own substack, warned about reading too much into early voting tabulations (link; preview). He's right.

It is tempting to think that when you have millions of early votes cast, that's a large enough sample size that overcomes any problems. It's not.

In statistical sampling, there are two types of errors, bias and variance, or what could be called systematic vs random errors. Larger sample sizes will reduce variance or random errors but will not overcome bias which is a symptom of the un-representativeness of the observed sample.

When we compute optimal sample sizes to run experiments or A/B tests, the computation assumes zero bias, and the sample size only cures sampling variance due to random errors.

The key to removing bias is not sample size but getting a representative sample of the population of interest.

***

The biggest issue with early voting data is that citizens choose whether to vote early or not. And the decision to vote early is correlated with whom the person is voting for.

Remember that in certain states in the last election, Trump was ahead by a huge margin, after they counted the in-person votes, then as the precincts started counting the early votes, Biden came roaring back and ultimately won. That's because in those states, Democrats were much more likely to vote early than Republicans.

A less clear version of this plays itself out in states with big cities, even before the age of early voting. The counties with big cities will take more time to count all the votes while those rural counties in the same states will finish counting quickly; thus, early results during election day will be biased towards smaller counties. Rural counties tend to favor Republicans while big cities usualy vote Democrat.

Aside from timing effects, the biases in early voting are also driven by partisan or demographic or behavioral factors. For example, an interesting segment consists of those voters who would not have voted if early voting was unavailable.

Plus, those using early voting data to make predictions are making the assumption that voters vote according to party affiliation (since voting is confidential). In this election, that assumption is highly suspect. One of the major talking points of Democrats is that certain prominent Republicans are defecting to vote blue. According to reports (link, link), they have been running ads encouraging wives to lie to husbands, and husbands to lie to buddies, if they were going to go blue.

Besides, Republicans are also hoping to convince some Democrats to switch sides, e.g. minorities.

What Nate Silver says is that including early voting data (of which there aren't much) in his models doesn't help because by themselves, such data have low predictive value.

The science behind food recalls

McDonalds is having a moment. It's got our attention when Trump used it to make a political statement, then it got implicated in a food recall. The FDA thinks that at least 75 people have gotten sick and one person died after consuming burgers, and the culprit is thought to be the raw onions inside the burgers. (link)

***
In Chapter 2 of Numbers Rule Your World (link), I praised the science of tracing food outbreaks but readers also learn that these investigations are far from infallible.

The challenge arises because people eat a variety of foods, and when only a few people are getting sick, it's hard to tell where the E.coli has come from. The initial investigation is done using questionnaires to collect as much information as possible about what people consumed. Even this is hard, because of timing issues. Once the data are collected, then some math is used to infer which foods are most likely the culprit.

According to this CDC release, they were able to interview 42 of the sick people, and all of them have eaten at McDonalds prior (how long ago?) before getting ill. Further 86% of them ate a quarter pounder. That's all they have released so far. It sounds very damning and definitive. Yet it is misleading.

We are missing information on any control subjects i.e. similar people who didn't get sick. It's possible that similar people also have high probability of eating at McDonalds, and that a high proportion of McDonalds patrons order quarter pounders. We need a case-control study, and the press release only mentions the cases.

In a perfect world, everyone who got sick ate one common thing and everyone who didn't get sick didn't. But we don't live in a perfect world. So, the result of the investigation is an educated guess.

For example, as far as I could tell, no reporter has asked the hard questions yet: they were first blaming quarter pounders, then they said it's the raw onions inside the quarter pounders... but, does this imply that McDonalds have different supplies of raw onions, and they only use Taylor Farms on quarter pounders, and use a different source of onions on other items?

Also, according to the article, Taylor Farms also supply onions to other fast-food chains, so why is the current outbreak limited to McDonalds? How come the people who ate the same onions supplied to Burger King didn't get ill?

***

Sometimes, these investigations chance upon very good leads, and it is possible to trace the contamination back to a specific batch from a specific plot of land. Sometimes, they might even be able to find the source of E.coli at the farm.

This investigation isn't that kind, though. It seems like it's so far mostly guessing based on scarce data.

***

What else do we know about food recalls from Chapter 2?

Food recalls are very expensive because everyone is throwing stuff away based on "an abundance of caution".

The benefits of food recalls are hard to nail down.

Most contamination events are temporary, and affect only certain batches of produce during a limited time window. Thus, whether or not there is a recall, these perishable goods will leave the food supply eventually, and new cases will vanish. It's very difficult to ascertain if the recall prevented any cases. Indeed, given the time it takes to investigate, and organize recalls, it's quite likely that the crisis is over before the recalls even start.

That's not to say food recalls have zero benefits. It's just not as obvious as one thinks.

 

Nobody likes seeing the dentist

Nobody likes seeing the dentist, and it isn't for the reason you have in mind.

Yes, having a dentist wield scrapers, drills and unnamable tools messing up your gums is a painful experience. But the real reason for not liking dentistry is to be found in this article (link): they recommend procedures that have no basis in evidence, and may even do more harm than good.

In recent years, my dentists claim that more frequent x-rays of the whole set of teeth are now recommended; one even suggests doing it more than once a year. According to Ars Technica's investigation, the American Dental Association does not recommend once-a-year x-rays.

Other procedures that are not supported by data include preventative extraction of wisdom teeth, treatment of white spot lesions, twice-annual cleanings, etc.

The reporter somehow fails to mention several root causes of over-treatment and over-diagnosis: profit motive and action bias. Every x-ray taken generates income for the dentists (and various other parties); in addition, patients mistakenly think that doing something (anything) is better than not doing, especially if they have an "insurance" plan in which costs can be passed to others.