Big Data, Plainly Spoken (aka Numbers Rule Your World)

The player who scored on his or her birthday

The SuperBowl game was quite a bore but I latched on to the comment made by the TV host that one of the players who scored did so on his birthday. What is the probability of a player scoring on his birthday? Is it rare?

A key element determining this probability is the probability that one or more players' birthdays fall on game day (Feb 9). This question reminds me of the famous "birthday problem". The birthday problem is a staple of textbooks trotted in front of every student of probability: what is the probability that at least two people in a group (e.g. students in a classroom) share the same birthday?

The birthday problem is popular as a tool of instruction because the answer is surprising. One thinks that it should be quite rare to find two sharing the same birthdays in a small group, say of 30 people, since there are 365 possible birthdays. But working out the math leads to the conclusion that, in a class of 30 students, the chance that at least two share a birthday is around 70%. In a class of 60 students, the chance rises to 99%.

An NFL team roster has 53 players, and in any game, can activate 48 players. For simplicity, I'm going to use 50 players per team, or 100 players for two teams on game day. (Some of these, e.g. the backup quarterbacks, may not get to play at all but let's ignore that complication.)

What is the probability that at least one of the 100 players were born on Feb 9? The chance is about 1 in 4. Even with 250 players, the chance is still only 50 percent. With 400 players, the chance is two-thirds. Four hundred exceeds the number of days of the year, but remember that more than one players could share the same birthday. (All these calculations, including the classic birthday problem, assume births are evenly distributed throughout the year.)

***

The larger the group, the less surprising each scenario is. Why does the probability grow much more slowly in the NFL problem than in the classic birthday problem?

The key factor is the fixing of the date to the game day. In the classic birthday problem, the shared birthday could fall on any day of the year so there are many more scenarios that fit the requirement.

This is confirmed when we compare the two formulas, term by term. The below formulas each address the inverse of the original problem. That is to say, they compute the probability of no student sharing the same birthday, and of no player having a Feb 9 birthday. Each term is associated with an individual in the group. In the NFL problem, every term is the same because the only requirement is that the birthday does not fall on Feb 9. In the birthday problem, every term is different because as we account for each student, another date is excluded so that we don't have two birthdays falling on the same day.

Thus, each term in the birthday problem is smaller than the corresponding term in the NFL problem. The more individuals we include in the problem, the wider is the gap between the pair of terms. In other words, the (inverted) probability vanishes much faster in the birthday problem relative to the NFL problem. Invert that, and the required probability increases much faster in the birthday problem than the NFL problem. (Actually, the first term in the birthday problem is larger than the first term in the NFL problem because in the birthday problem, the "first" student can have any birthday while in the NFL problem, the Feb 9 date is excluded from every player. However, this effect is small and dwarfed by all the other terms.)

***

The original question concerns the probability of a player scoring on his birthday. From above, we have a 25 percent chance that at least one of the 100 players were born on Feb 9, the game day. Only these players could score on their birthdays. So we need to multiply this by the probability of any of these players scoring.

This starts to get trickier. We can't justify assuming that all players have the same chance of scoring in a game. Clearly, first-team players have a higher chance of scoring, so do offensive players relative to defensive players. We would need a model of the probability of scoring dependent on variables such as time of field, position of the player, etc. One factor we won't need to include is date of birth (I mean month and day): that's because I assume that the ability to score is unaffected by day of birth, which I think is uncontroversial, and this "independence" assumption simplifies the formula.

Now, from the first part, we generate a scenario with k number of players born on Feb 9, then from the scoring model, we compute the probability of at least one of these k players scoring. We'd also need some baseline statistics such as what proportion of the players are offensive. Then, we compute the weighted average of all these scenarios.

 

 

 

 

 

 

Aging well, book-wise

2024 is a year that has brought me quite a bit of satisfaction. More precisely, world events have elevated the relevance of my two books, which is energizing. When I developed the themes of these books, I consciously sought topics that have enduring value.

The second chapter of Numbers Rule Your World (link) describes one of the grand successes of modern statistical science: the investigation of foodborne disease outbreaks. Toward the end of 2024, the U.S. has seen a spate of food recalls due to outbreaks of E.coli, salmonella, listeria, etc. (cucumbers, frozen waffles, frozen chicken, frozen cookie dough, carrots, deli meat, ...), the most famous of which was the McDonald’s E.coli outbreak of November 2024. Investigators believed that the outbreak was caused by contaminated onions served at McDonald’s stores (link). How epidemiologists make such determination is the subject of (half of) Chapter 2. The spinach outbreak featured in the chapter represents a best-case scenario, in which the investigators were able to confirm their cause-effect hypothesis by finding contaminated spinach at its source that tested positive for the exact strain of E.coli that caused the outbreak. As I wrote on this post (link), the McDonald’s investigation was not as lucky. The cost-benefit of food recalls deserves more scrutiny.

During the hurricane season, I revisited a key question of Chapter 3 of Numbers Rule Your World (link): is hurricane risk insurable? The insurance market is a game of musical chairs, in which players shift the risk around, and once in a while, the music stops, and a few get scorched. Many large insurers have already exited the scene, while small startups keep popping up, take incentives from the government, get run into the ground, and then bailed out by the government. See this blog post for more.

Numbers Rule Your World (link)’s Chapter 4, half of which concerns steroids testing in professional sports, became relevant again this year, as the top tennis player of either gender has become embroiled in controversies due to positive tests (link; link). At the time of writing, my take on steroids testing, that false negative rather than false positive is the Achilles heel of the testing programs, was controversial; later, Lance Armstrong got snitched and subsequently confessed, which laid waste to hundreds of “negative” tests over his long career (hello Holger Rune). Both tennis stars got a slap on the wrist because the governing bodies accepted their alternative explanation of contamination. Accidental contamination, and a myriad of other excuses, have always followed positive tests; it’s one of those applications that reveal the limit of statistical evidence: we can prove only the likelihood of foul play; to really nail a perpetrator, we need a confession, or physical evidence (e.g. caught with blood bags). My post about the tennis controversy is here.

***

Inflation continues to run unimpeded in New York City, and probably also where you live. The place where I used to get croissants (cornetti, to be geographically correct) now charges $7; it was $6 a couple of months ago, $5 a few years ago, and $4 when they opened about 10 years ago. (Those prices don’t include tax and tip, which adds at least 25%.) If you do the math, an increase from $4 to $7 in 10 years implies an average growth rate of 6% per year; and if you look up official statistics (link), the CPI index moved from 237 in Nov 2014 to 316 in Nov 2024, suggesting a growth rate of 3% per year. Chapter 7 of Numbersense (link) explains why the official statistic deviates so dramatically from lived experience. It demonstrates a very important key to success in data analytics: the analyst must know inside-out how numbers have come into being.

Chapter 6 of Numbersense (link) addresses the other big economic statistic, unemployment, in which, again, the official statistic seems to misrepresent reality.

Obesity drugs are all the rage, being pitched as the ultimate solution to the obesity crisis in the U.S. Chapter 2 of Numbersense (link) looked at the measurement of obesity, and treatments ranging from diets to surgery. CDC recently announced that the obesity rate of Americans has apparently plateaued, with the latest number showing a (statistically insignificant) slight decline (link). People were quick to attribute this to the rise of obesity drugs; thus, they are making the case that obesity drugs are causing a fall in average BMI, which is the standard metric used to measure obesity. I wonder how many of these people complained about using BMI to measure obesity during an earlier time when most treatments appear to be ineffective. The jury is still out on whether these drugs lead to long-term weight loss, which is one of the toughest tests of obesity treatments.

***

If you're looking for a gift this holiday season, get Numbers Rule Your World (link), get Numbersense (link), or get both!

Doping pops up in the news

Thanks to the Olympics, doping has once again popped up in the news.

Some American athletes (and journalists) harshly criticized WADA (link), the international anti-doping authority, accusing it of covering up about 20 positive drug tests of swimmers from China who were subsequently exonerated and continue to compete; the Chinese investigators submitted evidence that these athletes unknowingly ingested the banned substances at a hotel. These accusations were flying around for months prior to the Paris Olympics, and created a situation in which any Chinese medalist in swimming would be viewed suspiciously, regardless of whether the athlete failed a drug test during these Olympics.

To someone who's not familiar with the landscape or history of doping in sports, the facts looked damning. But if you know anything about this subject, or have read Chapter 4 of Numbers Rule Your World (link), you should know better. The WADA process as described by these complainers is exactly how it has always been handled, even when the accused are U.S. athletes.

The vast majority of athletes who have tested positive for performance enhancing drugs have denied doping, and used a variety of excuses to explain away the test findings. There are even official "therapeutic use exceptions" (link) that permit athletes to use banned drugs if they suffer from some conditions. TUEs should be controversial. It has been previously noted that many swimmers have been granted exceptions due to asthma, making them much, much more likely than the general population to suffer from the condition.

Eventually, the Chinese authorities struck back, pointing to the case of an American track athlete who, you guessed it, failed a drug test, and was allowed to continue to compete after the American investigators submitted evidence that he ingested the substance unknowingly from contaminated meat (link).

This post was prompted by long-time reader Antonio R. who sent me an article about Jannik Sinner, the current world No. 1 tennis player from Italy, who has recently failed drug tests, and, you guessed it, was allowed to continue to compete after the Italian investigators accepted evidence that suggested that the banned substance could have entered his body through a spray used by his physiotherapist (link).

***

As outside observers, it's impossible for us to judge who's lying and who's not. In my book, I made some observations about the anti-doping program:

1. It is far more likely that dopers are getting away with false negatives than innocent athletes who are wrongfully flagged with false positives. (Hence, the reference to "timid testers" in the title of Chapter 4.)
   
   
   
2. To come up with your own judgment of the effectiveness of antidoping, you can start by guessing what percentage of Olympics athletes might be doping. Is it 30%, 10%, 1%? Then, all you need to do is to look up what percent of tests have come back positive. When I did this exercise, the proportion of positives is way below the proportion of athletes who could be doping, which tells you that most of the dopers have been tested and tested negative.
   
   
3. The book was published before the fall of famous American cyclist Lance Armstrong, who, prior to admitting guilt, had always insisted that the hundreds of negative test results during his career represented a mountain of evidence that he was not a doper.
   
   
4. We should distinguish between a "chemical positive" and a "real positive". The chemical positive reveals that a banned substance was found to exceed the allowable level in the test. Crucially, the test finding itself cannot prove that the accused athlete doped, even if we want to believe it. Such a conclusion would require the assumption that the only way by which such substance could have entered the athlete's sample is via intentional doping.
   
   The history of athletes submitting alternative explanations for their positive tests is really very long. It ranges from the relatively boring such as eating contaminated meats to the extremely creative such as "the vanishing twin" (link).
   
   To nail down a "real positive," investigators have to locate actual evidence of doping, such as drug supplies or paraphernalia. Needless to say, it's very hard to prove such a thing. Armstrong was ultimately brought down by one of his former lieutenants.
   
   
   
5. It is equally hard for the accused athlete to prove that s/he has eaten a contaminated piece of meat. At best, they could show that meat purchased from some vendor contains the forbidden substance. But that tested meat isn't the meat that was eaten.
   
   It appears that the anti-doping agencies are quite willing to accept these explanations, and whether it's China, US, or Italy, they have been exonerating positive results based on these theories. Note that these athletes have not proven that the chemical positive was a false positive; by their response, they concede that the test was indeed positive, they just argue that it's not a "real positive".

***
This topic fascinates me as it relates to statistical thinking. There are the measures (via tests) used as indicators of hypothesized causes. Then there is the causal gap, because the indicators themselves don't prove a cause.

For further thoughts on this topic, see Chapter 4 of Numbers Rule Your World (link).

Statistical reasoning in sports

With the Olympics well under way, I've had some fascinating conversations with friends and coworkers, in which we mulled over causal reasoning frequently heard over the airwaves.

Here are a couple of examples:

• Growth stunting: there is this belief that training in certain sports like gymnastics from an early age may "stunt" one's growth. The proof is: almost all top-tier gymnasts have short figures!

Can gymnastics training invert one's genetic disposition of height? We wonder. Another explanation  is that taller people (or people with figures not optimal for top-tier gymnastics) have been weeded out by the competitive processes. Perhaps it's not the gymnastics training that has capped the gymnasts' heights, but shorter gymnasts have an advantage when performing gymnastics routines. This advantage could even be made more trivial - hypothetically, if judges held a preference for shorter gymnasts, so that they consistently scored them better, then the taller ones would eventually get weeded out, leaving just the shorter ones in competition.

• Effect of steriods/PEDs: One friend relates a conversation with another friend, who's an ice skating coach: the coach claimed that she could tell that a certain skater was doping by noticing some unnatural movement in her body, and later, she felt vindicated when that athlete tested positive for something.

This chain of reasoning requires an unproven step - that the substance the skater tested positive for in fact has the desired effect of improving her performance. What we observed are two things: a) the athlete did something amazing, perhaps beyond human ability; b) the athlete tested positive for a substance. We actually don't know that the latter caused the former. It may be likely but it by no means is certain. Given that these outlawed substances are rarely used, and any usage will be secretive, I'm not sure how these commentators are so confident in making the connection.

 

 

 

A sports analogy for data processing rules

I'm been mulling over how to explain the potential impact of strange data processing procedures used in the observational studies of Covid-19 vaccines. These are by no means standard operating procedures.

And I think I've landed on one explanation that is tantalizing.

***

Imagine a football (soccer) match that is tied 0-0 and heading into overtime. Overtime is a 30-minute affair, played through, and not sudden death. The overtime period is divded into two 15-minute halves with a short break in the middle.

That is the standard operating procedure.

***

Now, we are going to change the rulebook.

The first additional rule is that any goal scored in the first 10 minutes of overtime doesn't count. This rule is inspired the "case counting window" used in the Covid-19 vaccine studies. We're justifying it because players need time to get back to their optimal condition after the exhausting 90-minute regulation time. Therefore, it is not fair to count goals scored at the start of overtime.

The second additional rule is that any goal scored in the first 10 minutes of overtime by the visiting team doesn't count but those scored by the home team counts. How should I justify this rule? Well, to establish the need for recovery time, we take vitals of the players. The measuring machines are only installed in the home locker rooms and not the guest locker rooms, and thus, we are unable to determine if the guest players are at their optimal condition. This rule is inspired by not applying the case counting window to non-vaccinated people since we can't determine when they took the second shot (unlike the situation in clinical trials, when non-vaccinated people are given placebo shots.)

The third additional rule divides the first 10 minutes into two parts, and stipulates that if someone scores in the latter part (i.e. minutes 6-10), then they are immediately kicked out of the match, and their goal doesn't count. This rule is inspired by the special exclusion criterion disclosed in the booster study I've been discussing (link).

This third rule is complicating as it modifies the previous rules. Taken together, we have the following scenarios: for the home team, if they score in the first 5 minutes, the goal counts; if they score in minutes 6-10, the scorer is kicked out of the match, and the goal is disallowed. For the away team, if they score in the first 10 minutes, the goal is always disallowed; in addition, if the goal is scored in minutes 6-10, the scorer is ejected.

***

One can easily get lost in the details of pharmaceutical studies and lose sight of the implications of the methodologies. Transferring the context to sports helps me grasp the effects of these data processing procedures. Hope this helps you too.

PS [8-4-2023] In my haste to put down my thoughts relating to this analogy, I skipped a rule. This is a Rule #2.5. This rule says that if a goal is scored in the first 10 minutes of overtime by the visiting team, it is counted as a goal scored by the home team. In Rule #2, the away-team goal is just disallowed; in Rule #2.5, the goal is added to the home-team tally. This rule is inspired by the booster study, in which they kept people in the no-booster group for 7 days after they took the booster.

 

 

Covid vaccines contain PEDs, according to an athlete

Another sport has the doping demons. News arrived that U.S. triathlete Collin Chartier has been banned for three years after testing positive for EPO, a favorite performance-enhancing drug for endurance sports.

Doping is featured in my book Numbers Rule Your World (link), written before Lance Armstrong finally admitted to doping after vehement denials for years. I made several surprising observations. Firstly, if you or I are running an anti-doping lab, we'd calibrate the test to reduce false positives at the expense of false negatives, since a doper who is getting away with it isn't going to expose our error but a falsely accused clean athlete will surely erode our reputation. Secondly, and consequently, we expect most positives to be true positives. Thirdly, a high proportion of dopers are getting away with it.

To Chartier's credit, he did not invent excuses but admitted to the offence. As reported in the article, other triathletes have paraded out the usual unlikely excuses - one even claimed that a "tainted burrito" and a "tainted Covid-19 vaccine" were responsible for the positive tests.

Now, if we believe Chartier (and many other athletes who got caught doping), he only took EPO once and only after the two greatest wins of his career. He also claimed that no one else but him knew about the doping, and his coach, who is related to the world champion of the sport, said he was blindsided by his own athlete.

***

I'm still waiting for the day when an athlete shakes up the establishment by admitting all the negative tests s/he got while doping. That would take courage. Wait - Armstrong frequently said he was the most tested athlete on Earth while he was getting away with it. Does that count?

Great data journalism should have destroyed Djokovic's story but it didn't

If you follow sports, you could not avoid the Novak Djokovic saga at the Australian Open, which is scheduled to start this week. In brief, Australia, having pursued a zero Covid policy for most of the pandemic, only allows vaccinated visitors to enter. Djokovic, who's the world #1 male tennis player, is also a prominent anti-vaxxer. Much earlier in the pandemic, he infamously organized a tennis tournament, which had to be aborted when several players, including himself, caught Covid-19 (link). He is still unvaccinated, and yet he was allowed into Australia to play the Open. People are upset. Some players who got themselves vaccinated in order to play in the tournament are not cool. Spectators who also must be vaccinated in order to watch the matches in person are not amused.

When the public learned that Djokovic received a special exemption, the Australian government decided to cancel his visa. Djokovic's camp, however, proceeded to fight his case in court. This then became messier and messier, as the superstar told his side of the story. His parents, his fans, and the Serbian government aggressively supported the player. [Djokovic lost in court for the second time this Sunday, and was deported and no longer could play in the tournament.]

In the midst of it all, some enterprising data journalists uncovered tantalizing clues that demonstrate that Djokovic's story used to obtain the exemption is full of holes. It's a great example of the sleuthing work that data analysts undertake to understand the data.

***

A central plank of the tennis player's story is that he tested positive for Covid-19 on December 16. This test result provided grounds for an exemption from vaccination, although the Australian government tightened entry requirements due to the Omicron surge. The timing of the test result was convenient, raising the question of whether it was faked. Intriguingly, Djokovic attended a children's event the day after he said he tested positive, and also gave an in-person interview to a French reporter two days after. His team maintained that the test was authentic, and offered evolving explanations and apologies for his not isolating after testing positive.

Digital breadcrumbs caught up with Djokovic. As everyone should know by now, every email receipt, every online transaction, every time you use a mobile app, you are leaving a long trail for investigators. It turns out that test results from Serbia include a QR code. QR code is nothing but a fancy bar code. It's not an encrypted message that can only be opened by authorized people. Since Djokovic's lawyers submitted the test result in court documents, data journalists from the German newspaper Spiegel, partnering with a consultancy Zerforschung, scanned the QR code, and landed on the Serbian government's webpage that informs citizens of their test results.

The information displayed on screen was limited and not very informative. It just showed the test result was positive (or negative), and a confirmation code. What caught the journalists' eyes was that during the investigation, they scanned the QR code multiple times, and saw Djokovic's test result flip-flop. At 1 pm, on December 10, the test was shown as negative (!) but about an hour later, it appeared as positive. That's the first red flag.

Since statistical sleuthing inevitably involves guesswork, we typically want multiple red flags before we sound the alarm.

The next item of interest is the confirmation code which consists of two numbers separated by a dash. The investigators were able to show that the first number is a serial number. This is an index number used by databases to keep track of the millions of test results. In many systems, this is just a running count. If it is a running count, data sleuths can learn some things from it. (This is why even so-called metadata can reveal more than you think. Djokovic may have become the latest victim.)

Djokovic's supposedly positive test result on December 16 has serial number 7371999. If someone else's test has a smaller number, we can surmise that the person took the test prior to Dec 16, 1 pm. Similarly, if someone took a test after Dec 16, 1 pm, it should have an serial number larger than 7371999. There's more. The gap between two serial numbers provides information about the duration between the two tests. Further, this type of index is hard to manipulate. If you want to fake a test in the past, there is no index number available for insertion if the count increments by one for each new test! (One can of course insert a fake test right now before the next real test result arrives.)

The researchers compared the gaps in these serial numbers and the official tally of tests conducted within a time window, and felt satisifed that the first part of the confirmation code is an index that effectively counts the number of tests conducted in Serbia. Why is this important?

It turns out that Djokovic's lawyers submitted another test result to prove that he has recovered. The negative test result was supposedly conducted on December 22. What's odd is that this test result has a smaller serial number than the initial positive test result, suggesting that the first (positive) test may have come after the second (negative) test. That's red flag #2!

To get to this point, the detectives performed some delicious work. The landing page from the QR code does not actually include a time stamp, which would be a huge blocker to any of the investigation. But... digital breadcrumbs.

While human beings don't need index numbers, machines almost always do. The URL of the landing page actually contains a disguised date. For the December 22 test result, the date was shown as 1640187792. Engineers will immediately recognize this as a "Unix date". A simple decoder returns a human-readable date: December 22, 16:43:12 CET 2021. So this second test was indeed performed on the day the lawyers had presented to the court.

Dates are also a type of index, which can only increment. Surprisingly, the Unix date on the earlier positive test translates to December 26, 13:21:20 CET 2021. If our interpretation of the date values is correct, then the positive test appeared 4 days after the negative test in the system. That's red flag #3.

To build confidence that they interpreted dates correctly, the investigators examined the two possible intervals: December 16 and 22 (Djokovic's lawyers), and December 22 and 26 (apparent online data). Remember the jump in serial numbers in each period should correspond to the number of tests performed during that period. It turned out that the Dec 22-26 time frame fits the data better than Dec 16-22!

***

The stuff of this project is fun - if you're into data analysis. The analysts offer quite strong evidence that there may be something smelly about the test results, and they have a working theory about how the tests were faked.

That said, statistics do not nail fraudsters. We can show plausibility or even high probability but we cannot use statistics alone to rule out any outliers. Typically, statistical evidence needs physical evidence. That's one of the key takeaways in Chapter 5 of Numbers Rule Your World (link).

***

Some of the reaction to the Spiegel article demonstrates what happens with suggestive data that nonetheless are not infallible.

Some aspects of the story were immediately confirmed by Serbians who have taken Covid-19 tests. The first part of the confirmation number appears to change with each test, and the more recent serial number is larger than the older ones. The second part of the confirmation number, we learned, is a kind of person ID, as it does not vary between successive test results.

One part of the story did not hold up. The date found on the landing page URL does not seem to be the date of the test, but the date on which someone requests a PDF download of the result. This behavior can easily be verified by anyone who has test results in the system.

Because of this one misinterpretation, the data journalists seemed to have lost a portion of readers, who now consider the entire data investigation debunked. Unfortunately, this reaction is typical. It's even natural in some circles. It's related to the use of "counterexamples" to invalidate hypotheses. Since someone found the one thing that isn't consistent with the hypothesis, the entire argument is thought to have collapsed.

However, this type of reasoning should be avoided in statistics, which is not like pure mathematics. One counterexample does not spell doom to a statistical argument. A counterexample may well be an outlier. The preponderance of evidence may still point in the same direction. Remember there were multiple red flags. Misinterpreting the dates does not invalidate the other red flags. In fact, the new interpretation of the dates cannot explain the jumbled serial numbers, which do not vary by the requested PDFs.

***

Statistical investigations can be very powerful, and have gained strength in the Big Data era due to digital breadcrumbs. Nevertheless, statistical arguments suggest plausibility or probability, never certainty. Short of a confession or whistle-blowing or leaking, those who are inclined to disbelieve can always find reasons to disbelieve. Similarly, interested investigators can easily fool themselves.

 

 

Round 2: 10 Ways to Rank the Rio Summer Olympics

In a recent post, I took data from the Tokyo Olympics to demonstrate how one can find any number of data-driven stories in a sufficiently complex dataset. I outlined 10 different ranking schemes that place a different country first.

A few readers noticed and complained that the metric used for Germany was "not positive". They didn't like the insight that Germany saw the greatest decline in medals from Rio 2016 to Tokyo 2020. While "not positive" metrics have their practical uses, I indeed was hoping to make a purely "positive" post, and ran out of time. (I said the challenge can be completed; I never said it was easy!)

Since today is a new day, and this is a new post, I will now close the gap. Germany's older athletes, those over 45 years and over, were by far the strongest. Incredibly, two out of third earned medals.

Now, surely no one can question the completeness of my answer.

***

Since a few of you were cheering for me to fail, I decided to double down, and present to you another 10 metrics, proving that if you spend enough time with the data, you can pretty much say anything you want.

Scratch that. In the course of doing this second round, I bumped into one of those awkward data science moments: you see, the exploratory analysis caused me to look up an athlete's bio, and doing so, I discovered that the spreadsheet I snatched online did not show Tokyo athletes as expected but athletes from Rio 2016. So in this second round, what I will deliver is the same challenge but using strictly the data from the Rio 2016 Summer Olympics.

As before, I start with the official top 10 medals table from Rio 2016.

Nine of the 10 countries are the same as before. The Netherlands was swapped for South Korea. (Russia competed under its proper name rather than ROC.)

The United States won on four metrics, golds, silvers, bronzes and total medals - and plenty of others, if you start digging around.

China won the second most medals at Rio 2016. But the Chinese team had the highest proportion of female athletes - by a wide margin.

Team GB came in #3. No other team, though, produced a gold winner older than Nick Skelton, who at 59 proved that age is no barrier.

Russia was 4th in the overall medals table. Its women athletes were the most efficient at earning medals.

Germany was #5. No other team achieved more success amongst older athletes - those 45 years old and over.

France came in #6. Among the top 10 countries, the French medal-winners had the highest age.

Japan was #7 overall. They were the mirror image of the French: the average Japanese medal-winner was the youngest.

Australia showed up in the 8th spot. The Aussie gold medalists were even younger than the Japanese.

The Azzurri placed 9th overall. Previously, we learned before that the Italians had the most close calls. It turned out their gold medal winners tended to be younger.

In Rio, South Korea squeaked in at the 10th position. The nation was unique in sending a team in which the women were older than the men.

I end on a strong note. This last metric was very satisfying as it single-handedly placed South Korea on on side of the ledger with the other nine countries on the other side.

 

***

There you have it. Take the Rio Olympics, and one can achieve the same result - one can find 10 metrics that rank each of the top 10 countries as #1.

This is the power of statistics. How one uses this power is a different matter. Use it to inform, not to mislead.

(For those who may be snorting at this exercise, notice that every statement I made about the Rio Olympics presented historical facts. I did not generalize these observations to the future. Some of these insights are likely to be artifacts of this historical dataset. It's when someone uses such insights to speak about the future when you should be concerned.)

Demo of how data can tell any story you want, ode to Tokyo

A reader sent me to this BBC article, which is typical of a genre that popped up once the Tokyo Olympics closed.

As sports fans know, the United States ended up topping the medals table, whichever way you look at it. But to journalists, that makes for a boring headline.

So, they invented new ways of ranking the countries. For example, using medals per capita, they elevated tiny countries like San Marino to the top. Or, using GDP data, they found China to be the winner.

***

In this light-hearted post, I gave myself a challenge: elevate each of those top 10 countries from the official medals table to the top by inventing new metrics; also stick with Olympics data and don't bring in other datasets. What I am demonstrating is that when you have enough dimensions in your dataset, you can pretty much give top rank to anything you want.

My starting point is the official medals table. The United States come up top:

China is #2. But China is top if I look at the average number of golds won per athlete.

ROC came in #3. But ROC is top if we count the average number of medals (any color) won per athlete... and the ROC women did particularly well on this count.

Team GB came in 4th. Nevertheless, I can make them #1 by looking at the breadth of sports in which the British athletes won golds.

Japan, the host nation, ended up #5 on the medals table. But Japan is #1 in terms of the proportion of medals won that were golds.

Things get tougher as we go down the list. But no worries. Data to the rescue.

Australia placed 6th overall in the medals table. When it comes to their women athletes winning golds relative to the men, they outshone the other teams by far.

Italy showed up at #7. But the Azzurri suffered the most close calls of them all, as it had the most silvers and bronzes relative to golds.

Germany was 8th. It has possibly the most disappointing campaign as its gold haul dropped by the most from Rio 2016.

The Netherlands grabbed the 9th spot. In contrast to their neighbor, the Dutch earned the greatest growth in number of medals.

France rounded off the top 10 in the official ranking. But the next host nation is #1 when I look at how much the French men outshone their women in winning medals in Tokyo.

***

So I completed my challenge. You can indeed find a way to rank any of these nations #1.

That's a fun way of saying if you have enough dimensions in your dataset, you can pretty much prove anything.

 

 

Bargaining with statisticians

If you are running A/B tests in industry, you're used to this scenario: two days (or indeed, two hours) after the test started running, the test sponsor - someone who suggested testing that variant of the marketing copy because they believe strongly that the new version is of that miracle vaccine they have always dreamt - pointed to a huge spike in performance on the real-time tracker, and urged that the test be stopped early because the new version is obviously better, and we should pocket our gains immediately rather than conducting "academic research" for the sake of "scientific purity".

For those with any statistical training, alarm bells should be ringing loudly because such decision-making is the mother of all false positives. The key to understanding why is to imagine what would have happened if the first few hours of performance was in the opposite, undesirable direction. The test sponsor would have been very quiet. (The sponsor could have knocked on your door and said we should stop the test early and declare the new version an instant dud. I have not gotten that knock on the door once after having run hundreds of tests. It's just human nature.)

Worse, what really happens is that the test sponsor will remain silent until such time as the test version displayed a "significant" positive gap relative to the control version. This fallacy is technically called "running a test to significance". This may happen in two hours, two days, five days, whenever. The only time the test would run to its designed end date is when the gap between the test and the control versions never exceeds an acceptable minimum distance.

***

A few years ago, I wrote a piece for FiveThirtyEight, mischievously suggesting that baseball fans should leave the ballpark once the score gap is larger than a certain number of runs. It appears that most baseball fans - presumably including many business managers - think such behavior is blasphemous. Well, it's also blasphemy when they demand an early stop to an A/B test because the favored version of the marketing copy has a "substantial" lead!

It's also the same behavior when a pharmaceutical executive goes to the FDA and pushes to end a clinical trial early, which is the real topic of today's post.

***

The FDA has made a bargain with the pharmaceutical companies to allow "interim analyses," the point of which is to enable early stopping of clinical trials under clearly defined circumstances. Some safeguards have been imposed on such analyses, which as recent events have proven, are easily gamed.

Safeguard #1 is a limit on the number of interim analyses one can conduct during the course of the trial. Each analysis contributes a false-positive probability, and the more times one "peeks" at the outcomes, the higher the chance of a spurious conclusion.

Safeguard #2 is to raise the required standard of evidence for earlier analyses. This is a simple application of statistics: in an interim analysis, the sample size is much smaller, so the error bar around any outcome is much wider (it's worse than linear), and thus the required gap between test and control must be larger. Instead of a 5% significance level, the required level in interim analyses could well be 0.05%. This is useful but does carry risks. One way the gap clears the higher bar is if something strange happens by the time of the first analysis (i.e. an outlier event that will not repeat). Analogously, a baseball team might score 10 runs in the first inning, clearly enough for me to leave the ballpark but should one expect this team to score 10 runs in the first inning in the next games?

Safeguard #3 is to pre-specify three possible decisions during the interim analysis: a) stop the trial for efficacy b) continue to collect more data and c) stop the trial for futility. This is a sensible requirement.

***

Sadly, recent events have shown that statisticians have made a Faustian bargain. The other side has usurped the rules, rendering these safeguards toothless.

Exhibit #1: the recent approval of "aducanumab", a "treatment" for Alzheimer's disease by Biogen. I have not followed the entire saga, but according to this article in StatNews (link), the FDA has made many concessions throughout the process, including waiving Phase 2 trials, and allowing interim analyses of Phase 3 trials. During the interim analysis, the trials were stopped for futility. In other words, while the investigators hoped that at early read, the drug would prove so spectacularly successful that they could stop the trial and declare victory - they instead discovered that aducanumab performed badly enough that the trial had to be ended prematurely.

If the story ended there, I have no complaints.

Biogen continued to mine the data after the trial was terminated, and subsequently, submitted "more data" that they claimed overturned the interim analysis results. This action violates all three safeguards listed above. Biogen is no longer adhering to a limited number of interim analyses - in fact, it appeared to be running the trial to significance. The significance requirements in Safeguard #2 are tailored to the specific analysis schedule, and so those become irrelevant when the schedule is abandoned. Finally, the FDA has also dropped Safeguard #3 because a fourth decision path has been created: stop the trial for futility tentatively and continue to mine for any exploratory analysis that can contradict the interim analysis.

As before, the harm of this action is most readily seen if we imagine what might happen if the trial had been stopped for efficacy. Will Biogen continue to mine the data looking for evidence that the drug in fact did not work?

We don't even have to speculate about the answer because of other recent actions by the FDA and pharmaceutical companies.

***

Exhibit #2: the coronavirus vaccine trials have adopted a modified version of the interim analysis plan. Because of various other shortcuts, the allowable decisions at the interim analysis were a) granting emergency use authorization on efficacy and continuing the trial to its normal end date b) not granting EUA and continuing to collect more data c) stopping the trial for futility. It's actually not clear to me whether c) was ever considered an option but let's assume it was. The vaccine developers committed to continue to run the trials to at least one or two years even if they sucessfully received EUA.

In essence, the successful vaccine trials were stopped for efficacy - although the trials were not stopped because full approval has not been granted. Did the pharmas continue to mine the data so they can volunteer negative information to invalidate the EUA? The answer is emphatically no!

On the contrary, the pharmas argued that upon granting of the EUA, it has become unethical to keep people in a placebo group (i.e. unvaccinated). Therefore, within a few months, almost everyone in the placebo groups has been given the vaccine. In other words, they can't be mining  any more data given that the placebo group has ceased to exist.

This situation is the worst nightmare for any statistician who made the Faustian bargain of allowing early stopping through interim analysis. They were hoping that the other side is making a genuine effort to balance science with pragmatism. They are sorely mistaken. When the test is stopped for efficacy, actions are immediately taken that prevent further data mining. When the test is stopped for futility, the sponsor does not really stop the test, and continues to mine the data.

These actions make a mockery of the interim analysis bargain, which means we are back to square one - clinical trials are being run to significance, which is the mother of all false-positive findings.

***

P.S. This post is particularly relevant in light of the Citizen Petition to the FDA about potentially granting full approval to the Covid-19 vaccines based on interim results.

[6/11/2021: So far, three members of the scientific advisory board has resigned in protest of the FDA decision (link). No one on the board voted to approve the drug but the FDA leaders overruled the board.]