Big Data, Plainly Spoken (aka Numbers Rule Your World)

Pining for less precise forecasting

Mark also wonders about weather forecasts on his blog. He says:

If we have a prediction that is inaccurate, displays sudden swings, and makes forecasts wildly divergent from its competitors, that raises some questions.

As many of us have discovered, the websites that carry weather forecasts seem to be wrong most of the time. But "wrong" is hard to define. In fact, even divining the meaning of the probability of rainfall is hard, as I discussed before.

My other friend, with whom I mused about the accuracy of Waze's estimated driving time (link), is on the verge of giving up on Waze completely. Last week, he told me that Waze is not just messing up the time estimation, but it's also getting the routing wrong more and more frequently.

***

For weather, it seems that forecasts can be quite accurate when they are expressed using ranges of time. An example of such a forecast is a noreaster might hit the Northeast within the next five days, bringing temperatures down to around freezing point. In my experience, that statement has a high chance of being correct. But... if the forecast was stated as the temperature will drop to freezing point in NYC on Wednesday afternoon between 2 and 3 pm because of a noreaster, my experience is it's almost always wrong. The temperature plunge would indeed come around but the timing would be off.

The above forecaster thinks temperature will drop below freezing this coming Sunday. The timing of that forecast is probably wrong. If we look at that, and modify the forecast to Sunday plus/minus 2 days, i.e. some time between Friday and Tuesday, then we'd be more confident in its accuracy. Less precision is better!

***

Then comes Mark's issue. What if the forecaster continuously updates the forecast, so by Friday morning, the temperature drop is pushed back to Monday 4 pm, then by Monday 2 pm, it's further delayed to 6 pm, and by 5 pm, it's now slated for Tuesday 9 am, etc. etc. This is indeed what these weather websites do. Eventually, their forecast is "correct".

Because weather is a "continuous" phenomenon mostly, that is to say, the noreaster isn't going to emerge instantaneously on top of NYC, such a continuously-updated prediction methodology is guaranteed to become accurate eventually.

Wishful thinking using statistics

In this piece (link), John Burn-Murdoch at the Financial Times (he's the guy who made some nice Covid-19 graphs) advances a theory of "peak obesity" saying that the new weight-loss drugs will reverse the trend of increasing obesity around the world, pointing to the recent CDC release showing a slight dip in U.S. obesity rate as evidence.

Here is a funny quote from his article:

More likely than not, this will prove another case of “where the US leads, others will follow”. In Denmark, home of Ozempic and Wegovy creator Novo Nordisk, 3 per cent of adults were using the new drugs by the end of 2023. The decades-long climb in the obesity rate slowed to a crawl that same year, and declined among several age groups.

When only 3 percent of Nerwegians are using these drugs, the impact on the population-wide number is highly limited. I don't find this argument convincing.

***

Imagine one of these patients whose use of these drugs has directly lowered the obesity rate in Norway. The person has to be obese, and is taking the drug to treat obesity (rather than treating diabetes, its original use). The person has been taking the drug for long enough time for the effect to become detectable, probably a year, depending on how the original studies measured impact. Because the flattening of the obesity rate line happened during 2023, this person must have started the treatment before 2023, or in the early part of 2023.

The person must not have stopped taking the drug. The person must have lost enough weight so as to cross the boundary from obese to not obese (BMI = 30). In other words, the more obese the person is, the more unlikely this person's improvement can move the average obesity rate down because as far as I know, Novo Nordisk is claiming around 15% weight loss, and not claiming that an extremely obese person could become non-obese after a year of Ozempic or Wegovy. A severely obese person is defined as having BMI over 40. A weight drop of 15% reduces BMI of 40 to 34, which is still above the obesity threshold of 30. For simplicity, we can think of someone with BMI 30-35 as most likely to help lower the obesity rate if their BMI drops below 30 as a result of taking the drugs.

Further, as with all drugs, only a portion of those who take them would realize benefits because no drug is 100% effective.

What we know is 3 percent of all adults were taking the drugs at year end 2023. Is that a large enough slice of the population to affect the obesity rate? I doubt it.

***

Last but not least, there is a hidden model behind the FT claim - which explains the change in obesity rate using a single factor of weight-loss drugs. This model assumes that the people who successfully lost weight after taking these drugs did not undertake other activities (such as exercise or dieting) which also may lead to weight loss. Since this is data analysis, not a designed study, I don't think we know what other measures people have taken.

Aging well, book-wise

2024 is a year that has brought me quite a bit of satisfaction. More precisely, world events have elevated the relevance of my two books, which is energizing. When I developed the themes of these books, I consciously sought topics that have enduring value.

The second chapter of Numbers Rule Your World (link) describes one of the grand successes of modern statistical science: the investigation of foodborne disease outbreaks. Toward the end of 2024, the U.S. has seen a spate of food recalls due to outbreaks of E.coli, salmonella, listeria, etc. (cucumbers, frozen waffles, frozen chicken, frozen cookie dough, carrots, deli meat, ...), the most famous of which was the McDonald’s E.coli outbreak of November 2024. Investigators believed that the outbreak was caused by contaminated onions served at McDonald’s stores (link). How epidemiologists make such determination is the subject of (half of) Chapter 2. The spinach outbreak featured in the chapter represents a best-case scenario, in which the investigators were able to confirm their cause-effect hypothesis by finding contaminated spinach at its source that tested positive for the exact strain of E.coli that caused the outbreak. As I wrote on this post (link), the McDonald’s investigation was not as lucky. The cost-benefit of food recalls deserves more scrutiny.

During the hurricane season, I revisited a key question of Chapter 3 of Numbers Rule Your World (link): is hurricane risk insurable? The insurance market is a game of musical chairs, in which players shift the risk around, and once in a while, the music stops, and a few get scorched. Many large insurers have already exited the scene, while small startups keep popping up, take incentives from the government, get run into the ground, and then bailed out by the government. See this blog post for more.

Numbers Rule Your World (link)’s Chapter 4, half of which concerns steroids testing in professional sports, became relevant again this year, as the top tennis player of either gender has become embroiled in controversies due to positive tests (link; link). At the time of writing, my take on steroids testing, that false negative rather than false positive is the Achilles heel of the testing programs, was controversial; later, Lance Armstrong got snitched and subsequently confessed, which laid waste to hundreds of “negative” tests over his long career (hello Holger Rune). Both tennis stars got a slap on the wrist because the governing bodies accepted their alternative explanation of contamination. Accidental contamination, and a myriad of other excuses, have always followed positive tests; it’s one of those applications that reveal the limit of statistical evidence: we can prove only the likelihood of foul play; to really nail a perpetrator, we need a confession, or physical evidence (e.g. caught with blood bags). My post about the tennis controversy is here.

***

Inflation continues to run unimpeded in New York City, and probably also where you live. The place where I used to get croissants (cornetti, to be geographically correct) now charges $7; it was $6 a couple of months ago, $5 a few years ago, and $4 when they opened about 10 years ago. (Those prices don’t include tax and tip, which adds at least 25%.) If you do the math, an increase from $4 to $7 in 10 years implies an average growth rate of 6% per year; and if you look up official statistics (link), the CPI index moved from 237 in Nov 2014 to 316 in Nov 2024, suggesting a growth rate of 3% per year. Chapter 7 of Numbersense (link) explains why the official statistic deviates so dramatically from lived experience. It demonstrates a very important key to success in data analytics: the analyst must know inside-out how numbers have come into being.

Chapter 6 of Numbersense (link) addresses the other big economic statistic, unemployment, in which, again, the official statistic seems to misrepresent reality.

Obesity drugs are all the rage, being pitched as the ultimate solution to the obesity crisis in the U.S. Chapter 2 of Numbersense (link) looked at the measurement of obesity, and treatments ranging from diets to surgery. CDC recently announced that the obesity rate of Americans has apparently plateaued, with the latest number showing a (statistically insignificant) slight decline (link). People were quick to attribute this to the rise of obesity drugs; thus, they are making the case that obesity drugs are causing a fall in average BMI, which is the standard metric used to measure obesity. I wonder how many of these people complained about using BMI to measure obesity during an earlier time when most treatments appear to be ineffective. The jury is still out on whether these drugs lead to long-term weight loss, which is one of the toughest tests of obesity treatments.

***

If you're looking for a gift this holiday season, get Numbers Rule Your World (link), get Numbersense (link), or get both!