Big Data, Plainly Spoken (aka Numbers Rule Your World)

Strange PR for statistics

The University of Waterloo put out a press release to tout an extracurricular activity of a statistics professor. He devised a way to game a free prizes giveaway by the Canadian coffee chain Tim Hortons. (More coverage: here, here)

It's a bit weird because statistics is mostly absent from the strategy.

Here's the game. You buy a coffee and you get a chance to play a lucky draw for prizes. The advertised chance of winning a prize is supposedly 10 percent. Since the pandemic, all this happens online. There is a prize given out every 0.1 second.

The article described two methods of gaming this. The first method exploits the following operational detail: if there are no players in any interval, the prize is re-allocated to a future interval picked at random. Thus, as the day wears on, there are more and more intervals in which more than one prizes would be given away. So, the professor chose a time in the wee hours to play, and said he achieved an almost perfect win rate. He played 96 times, winning 94 prizes.

For this method, there were no data collected, so there could't be any statistical analysis. The article cited a one-time result and it's unclear how repeatable this strategy is. The win rate of 98% is based on a single sample. Perhaps the only analytical question is the best time to play. I'm not sure it's possible to game this precisely.

Also, to play 96 times, the professor had to have purchased 96 coffees. So he got 94 coffees paid for. Another amusing tidbit is that as a British guy, he usually drinks tea, not coffee. So he actually purchased 96 coffees that he wouldn't have otherwise... money spent to create an example for his intro stats class. (Actually, he won 67 coffees and 27 donuts. Assuming donuts are more expensive than coffees, the situation was a bit better than I portrayed. That said, in the crazy inflationary world we live in, a cappuccino in NYC now costs over $6 while a donut typically costs less.)

***
Then apparently, Tim Hortons changed the game a bit, or other people started to play the game the same way, so the professor had to change his strategy.

The second method has marginally a little bit of statistics in it. The coffee chain started to put up real-time statistics of win rates so at least now he could collect data. He didn't want to break the no scraping rule so he had students manually collect the data. As a result, the dataset has "holes". Nevertheless, the general strategy remains the same as the first one - finding a time interval that has the highest win rate.

It sounded like this was just a simple curve fitting exercise, interpolating between the observed time points. What surprised me was the precision with which he stated his results:

He found that the statistically best time to play Roll Up To Win was 3:16 a.m. ET. When he played his rolls at 3:16 a.m, he had an 80 per cent win rate... The worst time of day? 11:46 a.m. ET

Is this supposed to mean that 3:15 a.m. is not good enough? Does the statistical lesson also include prediction error bars?

***

It's also mildly amusing for them to admit that these methods can only win people free coffees - apparently there are bigger prizes, including a free car, but since those prizes are rare, they didn't have enough data to provide any strategy.

***

While this is a fun nerdy thing to do, the people involved in the press release seem to have a warped understanding of businesses. They seemed to think Tim Hortons's marketing team would be alarmed by such gaming, and would actively want to thwart them. In reality, the budget for the game had been set, and the number of prizes given out was fixed.

The strategy of playing late in the game late at night meant that the professor would have been winning surplus prizes that no one else claimed during waking hours. Plus, he was winning only free coffees, not the free car. So, I suspect Tim Hortons's staff weren't losing sleep over this.

P.S.[4/10/23] Corrected spelling of the coffee chain's name in a few places.

The cost of bad advice

In the last week, I kept being prompted by the recommendation engines to read the article in Slate titled "Don’t Just Look at the Odds: The Math Says You Should Buy a Mega Millions Ticket." It's obviously false but eventually I succumbed and clicked on the link.

I wanted to stop reading after I encountered this sentence, about half way through the article: "One assumption in this model is that you’ll take the annual payment option and receive the full $1.6 billion."

The pertinent assumption that they made isn't whether the winner should take the lump sum or annual payment options. The blockbuster is to assume that there would only be a single winner.

Earlier in the article, they stated a true but irrelevant point: that the odds of a winning ticket and the price of each ticket are still th same no matter how many tickets are sold.

I persisted in reading the entire article for the sake of writing this post. I wanted to know whether they will acknowledge the biggest factor here, which is that the more tickets are purchased, the more likely there will be more than one winners. Even having two winners can be devastating. The $1.6 billion suddenly turned into $0.8 billion. With three winners, it's $0.53 billion. If the ticket is purchased as part of an office pool with, say 10, participants, then $1.6 billion becomes $160 million. Not a bad payday but much, much worse than projected.

 

PS. They updated the post because too many people are pointing out the obvious about multiple winners. They continue to deny its importance.

Their math is wrong. Here is a good webpage that explains the math from a previous situation 10 years ago. The number of winners follows what is called a Poisson distribution. In the example given, there is a 60% chance of a single winner, but 30% chance of double winners and 10% chance of more than two winners. How can this be called "incredibly low odds"?

Another issue is that there are certain sets of numbers that attract much more tickets than usual. If one of those sets is the winner, you're going to have lots of winners.

They continue to completely miss the point.

 

 

 

 

Some statistics about nutrition statistics

I only read nutrition studies in the service of this blog but otherwise, I don't trust them or care. Nevertheless, the health beat of most media outlets is obsessed with printing the latest research on coffee or eggs or fats or alcohol or what have you.

Now, the estimable John Ioannidis has published an editorial in BMJ titled "Implausible Results in Human Nutrition Research". John previously told us about the crisis of false positives in medical research.

Oops, here are some statistics on nuitrition "science":

• In 52 attempts at using randomized experiments to validate findings from observational studies, the number of times the findings were replicated: 0
• In the NHANES questionnaire (the basis of all those findings), two-thirds of the participants provided answers that imply an energy intake that is "incompatible with life". I haven't read this paper; seems like worthwhile reading.
• There are at least 34,000 papers on PubMed with keywords "coffee OR caffeine" which means this one nutrient has been associated with almost any interesting outcome.
• Almost every single nutrient imaginable has peer reviewed publications associating it with almost any outcome. A statistician should never give the advice "If at first you don't succeed,..."
• Many findings are entirely implausible (and still get published in top journals)... for example, the idea that a couple of servings a day of a single nutrient will halve the burden of cancer is clearly "too good to be true," even more so for anyone who is familiar with this literature
• "Big datasets just confer spurious precision status to noise"
• Randomized experiments offer hope but are woefully undersized (like requiring 10 times the current sample).

Just to nail home the point, John concludes: "Definitive solutions will not come from another million observational papers or a few small randomized trials."

 

The tricky business of thinking about uncertainty, and why banks can never repay us enough

Andrew Gelman alerted us to a classical probability error that showed up on the Freakonomics blog recently. (by way of Xian).

Probabilistic thinking is not very intuitive. One wishes the Freakonomics team would attack conventional wisdom, instead of embracing it. Here, they trumpeted the "very long odds in the Israeli Lottery": the fact that the same six numbers won the same lottery three weeks apart. This feels like an extremely rare event but it is a case where we tend to underestimate severely its rarity.

My readers may want a refresher of what the mistake is.

***

Let's start with the much simpler "Birthday Problem". Say you're in the jury selection room with a hundred people. What's the chance that you can find two people born on the same day? The chance is much higher than you would imagine! In fact, the math shows that it is more or less 100%! Even if you have 60 people in the room, it is almost certain. With only 30 people in the room, there's about a 65% chance of finding at least two with the same birthdays.

The key to understanding this puzzle is to realize that we did not fix the day of birth; we are not looking for two people born on January 1st. The two people born on the same day could have been born on any one of 365 days (excluding Feb 29 for convenience). The chance that we can find two people born on exactly Jan 1 is much less likely than the chance of finding two people born on the same day.

Similarly, there are 4,950 distinct pairs of people in the room, any one of which can be the pair of people with the same birthday. So, there are many ways in which we can find two with the same birthday and that's why it is much more likely than we think it is.

The point being: If you take enough drives in the safari, you will find a leopard.

***

Turning to the Israeli lottery. The "rare" event in question can also happen in many ways: the numbers could reappear one, two, three, ..., ten weeks apart (assuming it's a weekly lottery); and the reappearance could happen in any one of I'd think thousands of lotteries all over the world. In fact, every single time a lottery is drawn anywhere in the world, there is a chance that the outcome will prompt some journalist to write up the same story: in X lottery, the same Y numbers won Z weeks apart, what long odds!

The point being: The more drives you take in the safari, the more likely you will eventually find a leopard.

***

Perhaps the difficult part of taking this argument is: why should we be thinking about these imaginary scenarios if we already know that it happened in the Israeli lottery and it happened three weeks apart? Doesn't the reality make the imaginary scenarios irrelevant?

Here is how I'd think about this: if you already know that the Israeli lottery drew the same numbers six weeks apart, then the question of what are the odds is silly because the odds are 100% since we know for sure that it happened! 

The question of "what are the odds?" makes sense only at the moment when we don't know if something would happen or not. That means one of many possible scenarios could happen.

This need to place ourselves in a state of uncertainty is what I think trips up a lot of people. We do not like uncertainty.

***

Because of the same reason, some people (most mainstream journalists, it seems) buy the government line that we "made money" or "did not lose money" from the TARP rescue of failed too-big-to-fail banks.

This analysis is done from a state of certainty, from hindsight, taking into account the banks thus far survived the danger. But evaluating TARP from this view is like celebrating a lottery win after you found out you are the winner. The fact that you won the lottery does not change the fact that economically, it was silly to play the lottery in the first place.

When a decision is made in a state of uncertainty, it has to be evaluated in that state of uncertainty. For it is the uncertainty that imposes the greatest cost. In the case of TARP, the key question is to determine whether the government negotiated the right terms on behalf of the people at the time of the decision, when the banks faced enormous existential uncertainty. There is little question that the government did not get a good deal. Just take a look at the kinds of deals that the banks themselves get when they save failing companies from bankruptcy! (This often involves the banks owning a vast majority of the shares of said companies in addition to multifarious onerous conditions.) Or, how banks act when we don't pay our credit card or loan bills: is it okay for us to stop paying for a year, repay the amount in full, and declare no harm done?

Of course, there were a zillion other policies not called TARP that directly benefited banks at a cost to the people, and thus a variety of arguments why the bank rescue was not costless. My favorite is the suspension of mark-to-market accounting so banks can list their assets at fictitious prices (some of these assets are cash flows from "liar loans" and so on). I will outsource the big picture to economist Dean Baker who does a great job "beating the press" every day (here, and here).