Big Data, Plainly Spoken (aka Numbers Rule Your World)

The science of counting more people using fewer people

In the previous post, I explored how the U.S. government counts the homeless population. We learn that they don’t try to count everyone year-round but estimate the count on a single day of the year. But they don’t really count every homeless person everywhere on one day, either. In this post, I look into how they handle the sampling dimension of geography.

Each jurisdiction (dubbed CoC in the HUD Guide) divides its territory into sub-regions. The sub-regions are labeled high- or low-density based on the anticipated number of homeless. The sub-regions are “sampled”: not all sub-regions are counted; those sub-regions without actual counts are estimated. All high-density sub-regions are chosen, plus a selection of low-density regions. This sampling scheme makes a lot of sense, as it expends the most effort in places where most homeless reside.

Elsewhere, the HUD Guide discloses that some regions are completely excluded, e.g. suburban sub-regions that have not reported any homeless people for a few years might be removed in subsequent years. Thus, there are really three sampling “strata”: high-density sub-regions are 100% sampled, zero-density sub-regions are 0% sampled, and the low-density sub-regions are partially sampled (the counts in those low-density areas that are not sampled will be subsequently estimated).

Unfortunately, the authors of the HUD Guide call the above sampling scheme a “Random Sample of Areas”. For example, on page 12, the guide misrepresents the sampling scheme as:

An example of a random geographic sample is a CoC [what I’ve been calling a jurisdiction] that selects some geographic areas (e.g. according to Census tracts or, city blocks) to represent the entire CoC geography. The data collected from the homeless people counted and interviewed in these regions would then be adjusted to represent the overall homeless population.

Without reading the details of the sampling scheme, one is misled to believe that every sub-region has equal chance of being selected, which isn’t an appropriate way of sampling these sub-regions, nor is this the recommended sampling scheme described later in the Guide.

Now, your numbersense might have detected something I slipped past you a few paragraphs ago... how is it that they know which sub-regions have high or low numbers of homeless before they start counting? Seems like there is a chicken-and-egg issue. This is indeed the case: in order to reduce the workload, we need to send more workers to high-density areas but if we know nothing about the homeless population, how can we designate some sub-regions as high-density?

The truth is any statistical procedure involves some guesswork. In this case, because HUD isn't dealing with homeless for the time time ever in 2024, they can use historical data as a guide; in addition, experts working at organizations that work daily with homeless people have valuable insights. Imagine a sub-region that has experienced a recent surge in homeless. The historical data would be misleading in this case but the expert knowledge would likely be informative.

***

Now, consider what happens inside a sub-region. We ask the same question as before: do we just walk around the entire area and count (and survey) every homeless person we encounter? In the HUD Guide, this method of counting is called “complete coverage count”.

Most large jurisdictions probably practice something called “known locations count” method. This involves compiling a list of all locations that homeless people tend to stay at overnight, and visiting those locations. This labor-saving tactic, just like the other one we’ve seen, carries the risk of under-counting. They also must work with experts to figure out all the spaces where the homeless reside.

The Guide also allows something called “service-based” count. This is another fascinating concept: the staff visits social service centers, such as food banks, that homeless people might visit during the day. This count is materially different from the other counts in two ways: the counting happens during the day (not at night); the counting does not happen on the night of the counting day but during a seven-day window that starts after the night of counting.

This "service-based" count is a remedial count; it is used to correct the initial night count. The assumption is that some homeless will be missed on the night of counting but they might show up at social service centers during the day. Deduplication is really important here. They must make sure the same person isn't counted twice. This is the reason why the service-based count doesn't commence until the main counting is finished.

***

If you still think counting people is easy, I'll leave you with one further complication.

Since the unsheltered homeless usually aren't found at their sleeping locations during the day, the counting takes place during the designated night, but the window of counting spans midnight. So, if left to our own devices, some of us will write in one date while the rest of us will write in the other date as the day of the count.

The Guide explicitly stipulates that every worker should use the date of the start of the counting window as the date of the count.

***

I was expecting the whole exercise of counting the homeless to be a mess but what I found instead was a well-written, quite comprehensive documentation of the methodology used by HUD workers to measure the homeless problem.

 

 

How hard can counting be?

Since the pandemic, the homeless problem has become visible in some major cities. The U.S. government just reported an 18% increase in the homeless population in 2024. This news brings forth an intriguing statistical question: how do they count the homeless?

Do they conduct a poll with some kind of random digit dialing, and compute the proportion of respondents who admit they are homeless? (Just kidding…)

HUD (Department of Housing and Urban Development) reports the homeless statistic as a count: 772K people are flagged as homeless, which is 18% higher than the count in 2023. How do they know this? Do they go around and find every homeless?

A moment’s thought quickly dispels that as impractical, well nigh impossible. To start with, the number of homeless people changes every moment in time. Is the 772K a snapshot or some kind of average? According to the Bloomberg article, it’s an “annual point-in-time estimate.” So, the bean-counters conduct a homeless tally once a year, on some specific date.

Even that should be impractical. Do they run around every neighborhood in all 50 states to find every homeless person hanging out on the streets that day? To learn more, I found my way to this HUD guide (link).

Reading this guide is like peeling an onion. Each answer leads to more questions; to their credit, the guide covers most of what we need in order to understand the count.

First, they provide a clear definition of homeless. Only people who meet the inclusion criteria are counted.

A homeless person can be “sheltered” or “unsheltered”. Sheltered homeless people are those who checked into shelters temporarily while unsheltered are those sleeping outside, at train stations, etc.

Notably, if some community offers permanent housing to the homeless, they are no longer regarded as homeless. So hypothetically, the homeless problem could be “solved” by giving away free housing, i.e. by affluence. (Before you jump all over this, I put “solved” in quotes for a reason.)

Sheltered homeless is easier to count as most organizations that offer temporary shelters to homeless receive public funding, and thus must satisfy reporting requirements. To obtain the sizes of these populations, it is possible to just extract data from a database. There are still potential problems, such as incomplete or inaccurate reporting.

Unsheltered homeless is counted in a more labor-intensive way by having workers approach them to conduct a survey. The survey isn’t strictly necessary if all we want is a count but since the government also wants to study different subgroups of homeless (e.g. homeless youth), the workers attempt to interview the homeless people they encounter. The guide also discloses that each interviewee receives “incentives” to participate – not shocking as many businesses also pay people to fill out questionnaires.

***

Having reduced the problem to counting the homeless who are sleeping outside, we’re still stuck with the prospect of running around whole regions to find every such person! This is still impractical for any sizable jurisdiction. We therefore need statistics. Counting everyone is called a “census”. Instead of counting everyone, we use statistical sampling to arrive at an estimate of the census tally.

Sampling means we don’t try to count everyone. Instead, we count some of the population, and then use math to extrapolate. How does HUD decide what to count?

This topic is unexpectedly rather complicated. The first sampling dimension is time. We want to describe the extent of homelessness for the whole year of 2024 but we certainly won’t be counting people every day of the year. As mentioned above, HUD only counts unsheltered homeless on a single day of 2024. Naturally, we should wonder why that single day (in late January) is selected as representative.

The HUD guide answers this question directly. It turns out they make no claims that the snapshot tally is “representative”. Even if it’s not the elusive number we’re chasing after, this estimate is still useful – imagine that the homeless condition has worsened generally across the U.S., however you want to define it, we’d expect that deterioration to be reflected in the single-day tally as well.

Further, HUD makes this amusing claim: they picked the last week of January because they expect to find the smallest unsheltered homeless population during that time of the year. Is this a brazen attempt to under-report the total? Here’s how they explain it: In many regions, the cold winter weather drives homeless to check into shelters, rather than sleeping outside. That’s why they expect the relative proportion of unsheltered homeless to reach a seasonal low. Because HUD staff believe that counting sheltered homeless is easier, and less prone to inaccuracy, compared to counting unsheltered homeless, they conduct the counting exercise in January each year.

The other benefit is labor saving. Counting the unsheltered homeless is labor intensive.

By restricting the counting to a single day, we have reduced the workload by several hundred times. But any labor saving comes with a cost – if someone wants to extrapolate this snapshot to the entire year, much more work is involved.

[A further nuance: it’s too hard to coordinate all jurisdictions to count on a single night, or for larger communities, to finish counting on one night. In practice, there is a week during which the counting takes place. Moving from one night to multiple nights creates the risk of duplicate counting as the homeless population is fluid. HUD spends quite a bit of energy to make sure the same person isn’t counted multiple times.]

The task at hand is still too daunting. Just imagine having to run around every corner of Manhattan to count all homeless people on a single night. Thus, the task is further condensed by sampling along a second dimension: geography. I’ll discuss this topic in the next post.

Pining for less precise forecasting

Mark also wonders about weather forecasts on his blog. He says:

If we have a prediction that is inaccurate, displays sudden swings, and makes forecasts wildly divergent from its competitors, that raises some questions.

As many of us have discovered, the websites that carry weather forecasts seem to be wrong most of the time. But "wrong" is hard to define. In fact, even divining the meaning of the probability of rainfall is hard, as I discussed before.

My other friend, with whom I mused about the accuracy of Waze's estimated driving time (link), is on the verge of giving up on Waze completely. Last week, he told me that Waze is not just messing up the time estimation, but it's also getting the routing wrong more and more frequently.

***

For weather, it seems that forecasts can be quite accurate when they are expressed using ranges of time. An example of such a forecast is a noreaster might hit the Northeast within the next five days, bringing temperatures down to around freezing point. In my experience, that statement has a high chance of being correct. But... if the forecast was stated as the temperature will drop to freezing point in NYC on Wednesday afternoon between 2 and 3 pm because of a noreaster, my experience is it's almost always wrong. The temperature plunge would indeed come around but the timing would be off.

The above forecaster thinks temperature will drop below freezing this coming Sunday. The timing of that forecast is probably wrong. If we look at that, and modify the forecast to Sunday plus/minus 2 days, i.e. some time between Friday and Tuesday, then we'd be more confident in its accuracy. Less precision is better!

***

Then comes Mark's issue. What if the forecaster continuously updates the forecast, so by Friday morning, the temperature drop is pushed back to Monday 4 pm, then by Monday 2 pm, it's further delayed to 6 pm, and by 5 pm, it's now slated for Tuesday 9 am, etc. etc. This is indeed what these weather websites do. Eventually, their forecast is "correct".

Because weather is a "continuous" phenomenon mostly, that is to say, the noreaster isn't going to emerge instantaneously on top of NYC, such a continuously-updated prediction methodology is guaranteed to become accurate eventually.

Aging well, book-wise

2024 is a year that has brought me quite a bit of satisfaction. More precisely, world events have elevated the relevance of my two books, which is energizing. When I developed the themes of these books, I consciously sought topics that have enduring value.

The second chapter of Numbers Rule Your World (link) describes one of the grand successes of modern statistical science: the investigation of foodborne disease outbreaks. Toward the end of 2024, the U.S. has seen a spate of food recalls due to outbreaks of E.coli, salmonella, listeria, etc. (cucumbers, frozen waffles, frozen chicken, frozen cookie dough, carrots, deli meat, ...), the most famous of which was the McDonald’s E.coli outbreak of November 2024. Investigators believed that the outbreak was caused by contaminated onions served at McDonald’s stores (link). How epidemiologists make such determination is the subject of (half of) Chapter 2. The spinach outbreak featured in the chapter represents a best-case scenario, in which the investigators were able to confirm their cause-effect hypothesis by finding contaminated spinach at its source that tested positive for the exact strain of E.coli that caused the outbreak. As I wrote on this post (link), the McDonald’s investigation was not as lucky. The cost-benefit of food recalls deserves more scrutiny.

During the hurricane season, I revisited a key question of Chapter 3 of Numbers Rule Your World (link): is hurricane risk insurable? The insurance market is a game of musical chairs, in which players shift the risk around, and once in a while, the music stops, and a few get scorched. Many large insurers have already exited the scene, while small startups keep popping up, take incentives from the government, get run into the ground, and then bailed out by the government. See this blog post for more.

Numbers Rule Your World (link)’s Chapter 4, half of which concerns steroids testing in professional sports, became relevant again this year, as the top tennis player of either gender has become embroiled in controversies due to positive tests (link; link). At the time of writing, my take on steroids testing, that false negative rather than false positive is the Achilles heel of the testing programs, was controversial; later, Lance Armstrong got snitched and subsequently confessed, which laid waste to hundreds of “negative” tests over his long career (hello Holger Rune). Both tennis stars got a slap on the wrist because the governing bodies accepted their alternative explanation of contamination. Accidental contamination, and a myriad of other excuses, have always followed positive tests; it’s one of those applications that reveal the limit of statistical evidence: we can prove only the likelihood of foul play; to really nail a perpetrator, we need a confession, or physical evidence (e.g. caught with blood bags). My post about the tennis controversy is here.

***

Inflation continues to run unimpeded in New York City, and probably also where you live. The place where I used to get croissants (cornetti, to be geographically correct) now charges $7; it was $6 a couple of months ago, $5 a few years ago, and $4 when they opened about 10 years ago. (Those prices don’t include tax and tip, which adds at least 25%.) If you do the math, an increase from $4 to $7 in 10 years implies an average growth rate of 6% per year; and if you look up official statistics (link), the CPI index moved from 237 in Nov 2014 to 316 in Nov 2024, suggesting a growth rate of 3% per year. Chapter 7 of Numbersense (link) explains why the official statistic deviates so dramatically from lived experience. It demonstrates a very important key to success in data analytics: the analyst must know inside-out how numbers have come into being.

Chapter 6 of Numbersense (link) addresses the other big economic statistic, unemployment, in which, again, the official statistic seems to misrepresent reality.

Obesity drugs are all the rage, being pitched as the ultimate solution to the obesity crisis in the U.S. Chapter 2 of Numbersense (link) looked at the measurement of obesity, and treatments ranging from diets to surgery. CDC recently announced that the obesity rate of Americans has apparently plateaued, with the latest number showing a (statistically insignificant) slight decline (link). People were quick to attribute this to the rise of obesity drugs; thus, they are making the case that obesity drugs are causing a fall in average BMI, which is the standard metric used to measure obesity. I wonder how many of these people complained about using BMI to measure obesity during an earlier time when most treatments appear to be ineffective. The jury is still out on whether these drugs lead to long-term weight loss, which is one of the toughest tests of obesity treatments.

***

If you're looking for a gift this holiday season, get Numbers Rule Your World (link), get Numbersense (link), or get both!

Contaminated science

 

 

In one sense, the E.coli outbreak attributed to McDonald's has ended, as reported cases have dwindled to a drip. In another sense, it hasn't ended, because the number of cases is still growing.

This is the challenge of foodborne disease outbreak investigations. It is still possible that the theory that onions caused the outbreak is a false lead. In the most successful investigations, scientists were able to isolate samples of contaminated foods which contain the specific strain of the bacteria causing illnesses. An example of a very successful case is the spinach investigation featured in Numbers Rule Your World (link).

***

Late last week, CDC put out a new press release (link), updating the public about the current investigation. But it manages to unnerve me with terrible messaging.

The communication on November 13, 2024 carries the following conclusion:

"Epidemiologic and traceback data show that fresh, slivered onions served at McDonald’s were the likely source of this outbreak. Affected onions served at McDonald's locations were recalled and are past shelf life."

The sentences point to two types of evidence: epidemiological data come from questionnaires sent to reported cases asking for information such as what foods were consumed prior to getting sick from E.coli; and traceback data require salvaging potentially contaminated ingredients at source and testing them in labs for the specific strain of E.coli associated with the current outbreak.

We already knew about the epi data. What's new in this press release is the traceback data. Epidemiological data always come first, and were used to point the finger at McDonald's, then Quarter Pounders at McDonald's, then fresh, slivered onions in Quarter Pounders at McDonald's.

Scroll to the bottom of the press release, and this is the entire paragraph published about traceback:

FDA tested recalled onions and one sample tested positive for non-O157:H7 Shiga toxin-producing E. coli. The E.coli found in onions was not the outbreak strain and no human illnesses have been linked to this strain. Additional testing results will be released when available by FDA.

It's a word salad worthy of U.S. presidential candidates.

Translation: the investigators did not find the specific strain of E.coli associated with this outbreak in the onion samples they collected from the farms. We don't know how many samples they tested but only one of them tested positive for any kind of E.coli, and unfortunately for the investigation, it wasn't the right strain. Thus, this positive result does not strengthen the evidence against onions from those farms, and their customer, McDonald's.

And yet, whoever wrote the summary paragraph inserted the false claim that epi and traceback data were consistent, and complementary.

As a scientist, I find that this type of communciations undermines my confidence in our public health organizations. One can say this is an instance of "story time". They lull us in with some data, and then spin a story ostensibly supported by the data but not really.

Indeed, most media outlets have swept the uncertainty under the rug, referring to this outbreak as the McDonald's outbreak, or the onions outbreak. The CDC press release actually only used the term the "likely" source.

***

For giggles, I asked ChatGPT whether the lab test results added to the evidence that onions were the cause of the outbreak, and here is its answer:

No, the lab testing did not directly add to the evidence that the onions were the cause of the E. coli outbreak. The epidemiological evidence was already strong, with 99% of the interviewed individuals reporting eating at McDonald's, and 84% specifically eating menu items containing fresh slivered onions. The lab testing of recalled onions found a non-O157:H7 strain of E. coli, which was not linked to human illnesses and did not match the outbreak strain. Therefore, while the testing confirmed that the onions were contaminated with E. coli, it did not provide direct confirmation that the specific strain responsible for the outbreak was linked to the onions. The epidemiological data (such as the high proportion of sick individuals who ate the onions) was the primary evidence supporting the conclusion that the onions were the source.

I like this answer!

***

Note: It's possible that the lab testers knew something else that links those tested onions to the outbreak but nothing about the lab test results as disclosed in the press release supports its summary paragraph.

Note 2: In the meantime, the economic damage of food recalls is real. McDonald's is reportedly spending $100 million to recover from the fallout (link).

 

The election could have been called much earlier

(Source: St Stev, Flickr)

The talking heads on TV talk shows like MSNBC and Fox News are notorious for their innumeracy. But on election night last week, they impressed me by demonstrating numbersense.

Viewers have been warned not to expect a result on the day of the election, as all the reputable forecasters such as Nate Silver, FiveThirtyEight, and the Economist, had proclaimed the 2024 edition to be the most tightly contested ever. Silver famously announced that he ran 80,000 simulations to discover that the 2024 presidential election would be a toss-up (50/50 with Kamala getting the edge by the tiniest of decimal points). Close races will require counting of all votes, including early and mail-in votes, and some states accept mailed votes that arrive after election day if they were postmarked prior. It sounded like we were predestined for another cliffhanger, like in 2020.

Various channels stayed with that narrative faithfully throughout the broadcasts, as the hosts (including some data scientists) dove into selective counties to report how Trump was performing against his 2016 or 2020 margins, and whether Kamala was over- or under-performing Biden's 2020 numbers.

Meanwhile, a red wave was rolling over the whole map, as we eventually learned. The early taking of Florida was a harbinger. Then, Trump started leading in all seven so-called "swing" states. Even now, the talking heads told viewers the election was "too close to call". It was not until the next morning that most of these networks eventually confirmed Trump the victor.

Nevertheless, some time before midnight, the tone of the election broadcasts changed. The hosts ceased pretending that Kamala had any chance left, even as they perfunctorily stated that their networks had yet to call the overall winner.

Those were the telling moments at which the talking heads gave in to their numbersense. Their data science colleagues were telling them to shush for the sake of good "science"; the business leaders didn't want them to ruin the watch parties. And yet, the numbersense in their heads insisted the race was over.

The hosts tip-toed into post-mortem analyses, but checked themselves, acting like unruly drivers who cut over multiple lanes to make a turn, all the time apologizing to other drivers while doing the naughty. Later, they threw caution to the wind, and earnestly examined the bruised blue corpse of the election.

***

The mood turned on the concept of "too close to call". What does it mean?

In the U.S., the news outlets follow the presidential election as a collection of state elections because of the electoral college. The focus is on the "swing states", of which there are seven in 2024. These are the (only) states for which the outcomes are somewhat uncertain.

The media outlets have been endowed the responsibility of "calling" the election. For some reason, each state election is reported as a horse race, with vote counts continuously updated after the polls close. As a result, while people in the Western states are still voting, the results in some other states have already been publicized. Furthermore, early results don't necessarily forecast final results since the sequence of vote counting is far from random. The non-random order of counting affects close races even more.

All the swing states were tagged "too close to call" on election night. What calculations happen to draw this conclusion?

Let's take a look at the situation in North Carolina at 10:47 pm ET, as shown in this CNN screenshot:

Trump was leading Kamala 51% to 48%, a gap of 135,000 votes. The race was labeled "too close to call".

The key number is displayed at the bottom: the current result reflected 84% of the total votes. That means 16% of the votes were still to be counted. Since 4.8 million votes have been counted, they were expecting roughly 910,000 votes. (I'm assuming no third-party votes, which is roughly correct as the Kamala and Trump percentages added up to 98.6%.)

The gap of 135,000 is 15% of 910,000. Thus, in order for Harris to get even, she would have needed to grab 15%+48% = 63% of the remaining votes while Trump gets 51%-15% = 36%. If she were to overtake Trump, she would have needed to win over 63% of the remaining votes.

In the initial 84% of votes, she claimed 48% to Trump's 51%; in the last 16%, she had to win 63% to 36%. In my view, that was an impossible feat so the race could have been called at that point (or even earlier). While the last 16% of the votes could have been skewed Democratic, it's not likely because the vast majority had already been counted, and counting happened simultaneously across all counties.

The data scientists behind the scenes would have access to even more data, at the county-level. The remaining votes were going to be found in a subset of the counties. They can repeat the same analysis obtaining a county-level required vote split for Kamala to get even or overtake Trump. On election night, the hosts worked around the election maps and highlighted some of the counties that have higher populations, especially Democratic strongholds where Kamala could have found additional votes. It didn't convince me that Kamala could cover the gap.

***

Now, the networks would defend their non-call by a criterion of "mathematical impossibility". That essentially means that they wouldn't call the race unless and until Trump's vote tally exceeded half the total votes, making it "mathematically impossible" for Kamala to win, regardless of uncounted votes.

This defense rings hollow because the same networks were extremely quick to call states like California and Wyoming - typically within a minute of the polls closing. At the time they called those "easy" states, they clearly did not see Trump's tally exceed half. So they were using different criteria for different states.

The whole business of having media outlets "call" the election results boggles the mind. These outlets have an incentive to hold out as long as possible because of viewership and ratings.

 

Nate Silver says no to early voting data

Nate Silver, now writing his own substack, warned about reading too much into early voting tabulations (link; preview). He's right.

It is tempting to think that when you have millions of early votes cast, that's a large enough sample size that overcomes any problems. It's not.

In statistical sampling, there are two types of errors, bias and variance, or what could be called systematic vs random errors. Larger sample sizes will reduce variance or random errors but will not overcome bias which is a symptom of the un-representativeness of the observed sample.

When we compute optimal sample sizes to run experiments or A/B tests, the computation assumes zero bias, and the sample size only cures sampling variance due to random errors.

The key to removing bias is not sample size but getting a representative sample of the population of interest.

***

The biggest issue with early voting data is that citizens choose whether to vote early or not. And the decision to vote early is correlated with whom the person is voting for.

Remember that in certain states in the last election, Trump was ahead by a huge margin, after they counted the in-person votes, then as the precincts started counting the early votes, Biden came roaring back and ultimately won. That's because in those states, Democrats were much more likely to vote early than Republicans.

A less clear version of this plays itself out in states with big cities, even before the age of early voting. The counties with big cities will take more time to count all the votes while those rural counties in the same states will finish counting quickly; thus, early results during election day will be biased towards smaller counties. Rural counties tend to favor Republicans while big cities usualy vote Democrat.

Aside from timing effects, the biases in early voting are also driven by partisan or demographic or behavioral factors. For example, an interesting segment consists of those voters who would not have voted if early voting was unavailable.

Plus, those using early voting data to make predictions are making the assumption that voters vote according to party affiliation (since voting is confidential). In this election, that assumption is highly suspect. One of the major talking points of Democrats is that certain prominent Republicans are defecting to vote blue. According to reports (link, link), they have been running ads encouraging wives to lie to husbands, and husbands to lie to buddies, if they were going to go blue.

Besides, Republicans are also hoping to convince some Democrats to switch sides, e.g. minorities.

What Nate Silver says is that including early voting data (of which there aren't much) in his models doesn't help because by themselves, such data have low predictive value.

The science behind food recalls

McDonalds is having a moment. It's got our attention when Trump used it to make a political statement, then it got implicated in a food recall. The FDA thinks that at least 75 people have gotten sick and one person died after consuming burgers, and the culprit is thought to be the raw onions inside the burgers. (link)

***
In Chapter 2 of Numbers Rule Your World (link), I praised the science of tracing food outbreaks but readers also learn that these investigations are far from infallible.

The challenge arises because people eat a variety of foods, and when only a few people are getting sick, it's hard to tell where the E.coli has come from. The initial investigation is done using questionnaires to collect as much information as possible about what people consumed. Even this is hard, because of timing issues. Once the data are collected, then some math is used to infer which foods are most likely the culprit.

According to this CDC release, they were able to interview 42 of the sick people, and all of them have eaten at McDonalds prior (how long ago?) before getting ill. Further 86% of them ate a quarter pounder. That's all they have released so far. It sounds very damning and definitive. Yet it is misleading.

We are missing information on any control subjects i.e. similar people who didn't get sick. It's possible that similar people also have high probability of eating at McDonalds, and that a high proportion of McDonalds patrons order quarter pounders. We need a case-control study, and the press release only mentions the cases.

In a perfect world, everyone who got sick ate one common thing and everyone who didn't get sick didn't. But we don't live in a perfect world. So, the result of the investigation is an educated guess.

For example, as far as I could tell, no reporter has asked the hard questions yet: they were first blaming quarter pounders, then they said it's the raw onions inside the quarter pounders... but, does this imply that McDonalds have different supplies of raw onions, and they only use Taylor Farms on quarter pounders, and use a different source of onions on other items?

Also, according to the article, Taylor Farms also supply onions to other fast-food chains, so why is the current outbreak limited to McDonalds? How come the people who ate the same onions supplied to Burger King didn't get ill?

***

Sometimes, these investigations chance upon very good leads, and it is possible to trace the contamination back to a specific batch from a specific plot of land. Sometimes, they might even be able to find the source of E.coli at the farm.

This investigation isn't that kind, though. It seems like it's so far mostly guessing based on scarce data.

***

What else do we know about food recalls from Chapter 2?

Food recalls are very expensive because everyone is throwing stuff away based on "an abundance of caution".

The benefits of food recalls are hard to nail down.

Most contamination events are temporary, and affect only certain batches of produce during a limited time window. Thus, whether or not there is a recall, these perishable goods will leave the food supply eventually, and new cases will vanish. It's very difficult to ascertain if the recall prevented any cases. Indeed, given the time it takes to investigate, and organize recalls, it's quite likely that the crisis is over before the recalls even start.

That's not to say food recalls have zero benefits. It's just not as obvious as one thinks.

 

Doping pops up in the news

Thanks to the Olympics, doping has once again popped up in the news.

Some American athletes (and journalists) harshly criticized WADA (link), the international anti-doping authority, accusing it of covering up about 20 positive drug tests of swimmers from China who were subsequently exonerated and continue to compete; the Chinese investigators submitted evidence that these athletes unknowingly ingested the banned substances at a hotel. These accusations were flying around for months prior to the Paris Olympics, and created a situation in which any Chinese medalist in swimming would be viewed suspiciously, regardless of whether the athlete failed a drug test during these Olympics.

To someone who's not familiar with the landscape or history of doping in sports, the facts looked damning. But if you know anything about this subject, or have read Chapter 4 of Numbers Rule Your World (link), you should know better. The WADA process as described by these complainers is exactly how it has always been handled, even when the accused are U.S. athletes.

The vast majority of athletes who have tested positive for performance enhancing drugs have denied doping, and used a variety of excuses to explain away the test findings. There are even official "therapeutic use exceptions" (link) that permit athletes to use banned drugs if they suffer from some conditions. TUEs should be controversial. It has been previously noted that many swimmers have been granted exceptions due to asthma, making them much, much more likely than the general population to suffer from the condition.

Eventually, the Chinese authorities struck back, pointing to the case of an American track athlete who, you guessed it, failed a drug test, and was allowed to continue to compete after the American investigators submitted evidence that he ingested the substance unknowingly from contaminated meat (link).

This post was prompted by long-time reader Antonio R. who sent me an article about Jannik Sinner, the current world No. 1 tennis player from Italy, who has recently failed drug tests, and, you guessed it, was allowed to continue to compete after the Italian investigators accepted evidence that suggested that the banned substance could have entered his body through a spray used by his physiotherapist (link).

***

As outside observers, it's impossible for us to judge who's lying and who's not. In my book, I made some observations about the anti-doping program:

1. It is far more likely that dopers are getting away with false negatives than innocent athletes who are wrongfully flagged with false positives. (Hence, the reference to "timid testers" in the title of Chapter 4.)
   
   
   
2. To come up with your own judgment of the effectiveness of antidoping, you can start by guessing what percentage of Olympics athletes might be doping. Is it 30%, 10%, 1%? Then, all you need to do is to look up what percent of tests have come back positive. When I did this exercise, the proportion of positives is way below the proportion of athletes who could be doping, which tells you that most of the dopers have been tested and tested negative.
   
   
3. The book was published before the fall of famous American cyclist Lance Armstrong, who, prior to admitting guilt, had always insisted that the hundreds of negative test results during his career represented a mountain of evidence that he was not a doper.
   
   
4. We should distinguish between a "chemical positive" and a "real positive". The chemical positive reveals that a banned substance was found to exceed the allowable level in the test. Crucially, the test finding itself cannot prove that the accused athlete doped, even if we want to believe it. Such a conclusion would require the assumption that the only way by which such substance could have entered the athlete's sample is via intentional doping.
   
   The history of athletes submitting alternative explanations for their positive tests is really very long. It ranges from the relatively boring such as eating contaminated meats to the extremely creative such as "the vanishing twin" (link).
   
   To nail down a "real positive," investigators have to locate actual evidence of doping, such as drug supplies or paraphernalia. Needless to say, it's very hard to prove such a thing. Armstrong was ultimately brought down by one of his former lieutenants.
   
   
   
5. It is equally hard for the accused athlete to prove that s/he has eaten a contaminated piece of meat. At best, they could show that meat purchased from some vendor contains the forbidden substance. But that tested meat isn't the meat that was eaten.
   
   It appears that the anti-doping agencies are quite willing to accept these explanations, and whether it's China, US, or Italy, they have been exonerating positive results based on these theories. Note that these athletes have not proven that the chemical positive was a false positive; by their response, they concede that the test was indeed positive, they just argue that it's not a "real positive".

***
This topic fascinates me as it relates to statistical thinking. There are the measures (via tests) used as indicators of hypothesized causes. Then there is the causal gap, because the indicators themselves don't prove a cause.

For further thoughts on this topic, see Chapter 4 of Numbers Rule Your World (link).

How would we know if Google engineers really improved traffic flow at intersections

 

Mark P. and I have had discussions about evidence of efficacy related to Google Maps navigation: it is mighty hard to come up with data that prove that software like Google Maps or Waze actually delivers on the promise of reducing travel time. See this blog post for previous coverage.

These real-time optimization products often leave me with the feeling that they are executing "local" optimization, that is to say, they find the best solution within a highly constrained set of facts. A locally optimal solution is in general not optimal.

Since everyone has downloaded files from some web server, it's a good example to chew on. We have all seen the pop-up window that traces the estimated remaining time to finish downloading a large file. We also know that the prediction algorithm is so naive as to be laughable. It appears to perform a simple calculation: remaining file size divided by current download speed. This is the locally optimal answer. But internet download speeds are erratic, and so this local optimum is rarely accurate. We have all observed the estimated remaining time jump from 5 minutes to 5 hours, just because the internet speed has momentarily plunged.

The naive calculation is good if it is realistic to assume that the current download speed will hold steady till the time of completion. It is also good if it happens that the current download speed is close to the average speed of the entire download. (Neither of those are good assumptions.) The naive algo is reasonable if the set of facts available to the algorithm is constrained to (a) the current download speed and (b) the remaining file size. Imagine you are not allowed to use anything other than those two values, and you might decide to just divide one by the other.

Constraining these facts, though, is a choice. For example, the algo can be given the recent history of download speeds, rather than having just the current speed to work with.

***

Mark recently resurfaced this discussion because Google engineers have developed an optimization method for reducing the amount of congestion at road intersections (link)

Of course, the first thing I look for is proof. Here's what they say in the blog post:

Today, Green Light is live in over 70 intersections, helping to save fuel and lower emissions for up to 30 million car rides monthly. Early numbers indicate the potential to reduce stops by up to 30% and reduce emissions at intersections by up to 10%

There is no link to any papers, or data. So we just have to trust their word. The program manager elaborated:

We offer each city dedicated reports with tangible impact metrics, such as how many stops drivers saved at an intersection over time.

***

Let's think about how one might come up with evidence to prove the effectiveness of such optimization. The engineers say that the savings come from optimizing the timing of traffic lights to reduce stopping at intersections because the restart leads to pollution.

(Having driven around in Europe, I think the Europeans solve the problem completely by favoring roundabouts over traffic lights but that's a different thread.)

First, we need a baseline value for comparison. For any given intersection (one of those 70), the current traffic light timing is optimized, thus, we'd need to know the counterfactual, i.e. what would have been the amount of stoppage, had this intersection not implemented Google's optimization. To figure that out, we'd need to build a model that predicts the number of stops given a variety of factors, and we'd hope that this model has high accuracy.

Second, let's extend our focus from a single intersection to a stretch of highway involving a series of intersections. Is Google's algo implemented at every intersection, some selection of intersections, or just one intersection? For the sake of argument, let's say it's a single intersection, and we measure the change in stops at this intersection. It's highly likely that we may have pushed the congestion to another intersection, and what is gained locally is lost elsewhere.

***

Chapter 1 of Numbers Rule Your World (link) explores the complex world of traffic engineering through the tactic of ramp metering, i.e. controlling the flow of traffic onto highways. A key insight from that field is that congestion should be considered an unavoidable consequence when demand exceeds supply. Many tactics to reduce congestion merely re-distributes it.

The ramp-metering program is also notable for a valiant effort to evaluate real-life performance by running a principled experiment.