Big Data, Plainly Spoken (aka Numbers Rule Your World)

Were the U.S. polls accurate?

Andrew chimes in to defend the polling profession (link). He's responding to a piece in the London Review of Books that claims that the polls for the U.S. 2022 mid-term elections predicted a red wave that did not materialize.

He pointed to the Economist model that is an aggregation of polls that did not insert punditry type assumptions. This model is different from, say, Nate Silver's models for FiveThirtyEight, which incorporates some subjective judgement.

The poll aggregation model did remarkably well in predicting the overall outcomes in both the House and the Senate. In the House, the model predicted 225 Republican seats (208-244) compared to the eventual outcome of 222. In the Senate, the model predicted 51 Republican seats (46-55) compared to the eventual outcome of 49.

The poll aggregation model did not predict a red wave, as the interval estimates of the number of seats for both parties substantially overlap. In other words, it predicted a close election, which it turned out to be.

***
Andrew speculated that the LRB writer thought the polls predicted a red wave because he was "sticking to an existing narrative." The journalist regurgitates the "conventional wisdom" which happens to be wrong.

Andrew distinguishes between poll aggregation and poll forecasting, the latter combining poll data with punditry. He also focuses on the aggregate outcome (number of races predicted correctly), not the outcomes of individual races (e.g., did the polls get in right in state X?).

***

Andrew's post triggered a few thoughts of mine.

I dislike the conventional metrics of accuracy that treat every race as equally hard to predict. Given that most races in all U.S. elections are uncompetitive - as in, even a chimpanzee can predict the winner - a serious metric of accuracy should only consider competitive races. This issue is similar to that of grade inflation in education: the nominal scale (A-F) is misleading when all grades are compressed into the A-B range.

Worse than sticking to an existing narrative, the journalist is reinforcing a false narrative. I see the world as comprising two types of people in their relationship to data: data-first versus story-first (link). People who are story-first starts with the story and finds data that support the story. In the case of the 2022 elections, the Republican mouthpieces were everywhere pushing the narrative of the red wave. The polling data were an inconvenience that didn't matter.

Democrats can also be story-first. In fact, some Democrats might have allowed the false narrative to persist in the hope that fearing the red wave would drive their supporters to the polls.

When I came up with the data-first/story-first framework, I wanted something that is non-judgmental. Story-first is not always wrong; indeed, the data can be consistent with the story. In my experience, story-first is the default human instinct, even though being a data person, I wish the opposite. I'm definitely data-first but data-first isn't right for every decision either, especially when the data are inadequate, contradictory, or inappropriate.

The recent U.S. elections have been so close that it's virtually impossible to predict the result accurately. Statistical models are calibrated to say exactly that. The interval estimates shown above overlap substantially. This is a situation in which the statistician would be unwilling to put a strong bet on either side. For example, in a Bayesian sense, the Republicans were found to win control of the Senate in about 60% of simulations. This means the Democrats won control in the other 40% of simulations. Real life is just one simulation!

The model hitting the right ranges of results was a matter of skill in collecting and analyzing data. The model coming quite close to real life was a matter of luck. People find it frustrating when statisticians don't give a clear-cut answer. The real issue is not the statistics, it's the difference between story-first and data-first thinking. Story-first thinking is typically black and white; data-first thinking often involves shades of gray - as Andrew likes to say, we "embrace uncertainty".

 

P.S. [4/28/2023] Reader Mark P. has some further thoughts on this topic. His point about the inutility of election forecasting reminds me of this past post.

Long live anonymity

Anonymity is in my thoughts at the moment.

First, I learned that the U.S. Census Bureau has moved from paper surveys to online surveys. What does this mean for its central promise of confidentiality? Here is NPR's description of this promise:

Under current law, the federal government is not allowed to release personally identifiable information from the census until 72 years after it's gathered for the constitutionally mandated tally. The bureau has relied on that promise of confidentiality to get many of the country's residents to volunteer their information once a decade, especially among people of color, immigrants and other historically undercounted groups who may be unsure about how their responses could be used against them.

Responding to the Census is mandatory, and yet the Census Bureau does not offer anonymity - instead, it claims to protect our sensitive data. This is exactly what the tech industry says, and yet we all know our data aren't really protected: the truth is that sooner or later, our data will get leaked.

Anything online is much more easily leaked than anything on paper. If the information is on paper, the thief has to be present physically, and s/he would need a truck to steal them - after which it would take a lot of time to scan or copy the data. The efficiency and compression gains plus worldwide access from powerful technologies make large-scale theft much quicker to execute.

***

Second, at a recent conference, I heard about a scheme designed to protect "sensitive" variables from online/mobile advertisers. They notably missed the simplest solution, which is to not collect, store, or use such sensitive data.

By "they", think any of the big tech players, like Google, Microsoft, Meta, etc. In their world, they want to use sensitive data to target advertising and extract revenues from advertisers, while they "protect" their users from the prying eyes of these advertisers. So, the advertisers are cast as the bad players while they themselves are good.

The speaker did not define what he meant by sensitive data. I imagine things like someone being a political activist (especially if they live under a repressive regime), sexual orientation, divorce status, disability status, etc.

The main idea of the talk appears to be using clustering to hide sensitive information about individuals. Each user has an array of sensitive and non-sensitive variables. A cluster analysis forms groups of users that are similar across the entire array of variables, and the advertisers will be provided cluster identifiers for users, rather than the variables.

There seems to be a flaw in this logic. For example, if the sensitive variable we are trying to protect is pro-choice activism, and the clustering analysis is sharp, then we may find a cluster in which all or almost all individuals are pro-choice activists, i.e. the cluster is dominated by a single sensitive variable. In this case, not only is the sensitive information exposed but the clustering strategy has backfired - as an adversary is now able to learn the sensitive variable for a whole group via the cluster identifier.

The definition of sensitive information is itself a complex problem. For example, if I am short, and our society discriminates against short people, I might want to hide my height, which makes it a sensitive variable for me - although the tall people would see it as an advantage, and not think of height as sensitive.

 

 

 

Strange PR for statistics

The University of Waterloo put out a press release to tout an extracurricular activity of a statistics professor. He devised a way to game a free prizes giveaway by the Canadian coffee chain Tim Hortons. (More coverage: here, here)

It's a bit weird because statistics is mostly absent from the strategy.

Here's the game. You buy a coffee and you get a chance to play a lucky draw for prizes. The advertised chance of winning a prize is supposedly 10 percent. Since the pandemic, all this happens online. There is a prize given out every 0.1 second.

The article described two methods of gaming this. The first method exploits the following operational detail: if there are no players in any interval, the prize is re-allocated to a future interval picked at random. Thus, as the day wears on, there are more and more intervals in which more than one prizes would be given away. So, the professor chose a time in the wee hours to play, and said he achieved an almost perfect win rate. He played 96 times, winning 94 prizes.

For this method, there were no data collected, so there could't be any statistical analysis. The article cited a one-time result and it's unclear how repeatable this strategy is. The win rate of 98% is based on a single sample. Perhaps the only analytical question is the best time to play. I'm not sure it's possible to game this precisely.

Also, to play 96 times, the professor had to have purchased 96 coffees. So he got 94 coffees paid for. Another amusing tidbit is that as a British guy, he usually drinks tea, not coffee. So he actually purchased 96 coffees that he wouldn't have otherwise... money spent to create an example for his intro stats class. (Actually, he won 67 coffees and 27 donuts. Assuming donuts are more expensive than coffees, the situation was a bit better than I portrayed. That said, in the crazy inflationary world we live in, a cappuccino in NYC now costs over $6 while a donut typically costs less.)

***
Then apparently, Tim Hortons changed the game a bit, or other people started to play the game the same way, so the professor had to change his strategy.

The second method has marginally a little bit of statistics in it. The coffee chain started to put up real-time statistics of win rates so at least now he could collect data. He didn't want to break the no scraping rule so he had students manually collect the data. As a result, the dataset has "holes". Nevertheless, the general strategy remains the same as the first one - finding a time interval that has the highest win rate.

It sounded like this was just a simple curve fitting exercise, interpolating between the observed time points. What surprised me was the precision with which he stated his results:

He found that the statistically best time to play Roll Up To Win was 3:16 a.m. ET. When he played his rolls at 3:16 a.m, he had an 80 per cent win rate... The worst time of day? 11:46 a.m. ET

Is this supposed to mean that 3:15 a.m. is not good enough? Does the statistical lesson also include prediction error bars?

***

It's also mildly amusing for them to admit that these methods can only win people free coffees - apparently there are bigger prizes, including a free car, but since those prizes are rare, they didn't have enough data to provide any strategy.

***

While this is a fun nerdy thing to do, the people involved in the press release seem to have a warped understanding of businesses. They seemed to think Tim Hortons's marketing team would be alarmed by such gaming, and would actively want to thwart them. In reality, the budget for the game had been set, and the number of prizes given out was fixed.

The strategy of playing late in the game late at night meant that the professor would have been winning surplus prizes that no one else claimed during waking hours. Plus, he was winning only free coffees, not the free car. So, I suspect Tim Hortons's staff weren't losing sleep over this.

P.S.[4/10/23] Corrected spelling of the coffee chain's name in a few places.