Big Data, Plainly Spoken (aka Numbers Rule Your World)

Hasty election observations that won't stand the test of time

The cable hosts have a lot of time to fill during election week so it's not surprising that they let slip a bunch of observations that won't stand the test of time. Many of these sound plausible but will fall apart on further examination.

More Republicans voted for Trump than in the last election.

I saw this data nugget used by progressive Democrats to argue that the Lincoln Project (a group of prominent Republicans who oppose Trump's reelection) had little to show for the money and effort they put in (link). According to some reports (also see section below), 93 percent of Republicans voted for Trump in 2020, up 3 percent from 90 percent in 2016.

Not so fast. The base of those two percentages are not the same. Between 2016 and 2020, some Republicans digusted with Trump have left the party so the remaining Republicans are even more likely to be Trump supporters. The statistic may be fine but it doesn't prove that the Lincoln Project failed to make a difference.

A supplementary analysis should start with all 2016 registered Republican voters and see what they did in 2020.

The only demographic group in which Trump lost support was white men.

How do we know such a thing so soon after the election (link)? At this stage, all such statements come from results from exit polls. In fact, the data nugget about Republicans voting for Trump also emerged from exit polls.

The problem: exit polls are unreliable in a normal year, and basically worthless in 2020.

Before the pandemic, exit polls are given the veneer of credibility because they are conducted at selected polling sites on election day, and because they feed news anchors with lots of data to support stories about the election results.

The credibility of exit polls depends on (a) a sound sampling design, not easy given the highly skewed distribution of population (b) identifying people who actually voted and (c) getting truthful answers from such people.

In a normal year, (b) is not a problem. The pollsters intercept people as they leave the polling sites so no one doubts that people who answer questions voted. However, (a) and (c) can both be problems.

The rise of mail-in voting during the pandemic has eliminated the exit polls' only advantage. This year, a huge proportion of voters sent in their ballots. The pollsters can only reach these voters through phone, web or mail. But it's a fishing expedition because they don't know who voted. The response rate of phone/web/mail is also much much lower than interception at the polling site. Finally, the responders are self-reporting that they voted, potentially days or weeks after they did. There is no visual confirmation of gender, ethnicity and age over the phone or internet.

The exit polls this year have a mixture of polling site interviews and phone/web/mail. The quality of the data is much better in the former than the latter. But the former subset skews heavily Republican. So it's a grand mess! I wouldn't trust any of the information from exit polls.

 

Progressives i.e. "socialists" are the black sheep of the Democratic party.

It didn't take long for Democratic centrists to take a dump on the progessive wing of their party. The Washington Post has a leaked recording in which a Democratic representative blamed her progressive colleagues for their disappointing results in Congressional elections. You may also recall, according to Hillary Clinton, what happened in 2016 was odious progressives.

As far as I can tell, the evidence for this accusation is Trump's claim that Biden is secretly being controlled by Bernie Sanders.

Since the total of Trump and Biden vote shares exceeds 98 percent this year, and the participation rate exploded, I think it's safe to say most progressives voted for Biden. If that is confirmed later, then centrists in the party should be thanking progressives for gifting them this victory. If these "black sheep" Democrats were indeed the marginal voters who cost Clinton the Presidency in 2016, then they would be the marginal voters who pushed Biden to victory in 2020. They are two sides of the same coin. The centrists can't have it both ways.

 

Trump hurt himself by fomenting fear of mail-in ballots.

This conclusion represents a common fallacy in reasoning with data, a confusion between selection and causation. What is undisputed is that mail-in votes skew Democrats and in-person votes skew Republicans. What is the implication of this statistic?

Each voter has a choice of how to vote. In this election, Biden voters disproportionately select to vote by mail while Trump voters disproportionately select to vote in person. The argument seems to say that if more people have voted in person, there would have been more Trump votes.

Perhaps it is not obvious but this argument has an implicit assumption - that voting in person causes someone to vote for Trump. If a mail-in voter had decided to vote in person, then s/he would have voted for Trump instead of for Biden. If true, Trump would have hurt himself by fomenting fear of mail-in ballots.

But that is absurd. The same argument from the other direction is also absurd. If Trump had not sowed doubts about mail-in voting, more Republicans might have voted by mail but voting by mail would not cause them to suddenly elect Biden.

No, changing how someone votes is unlikely to affect who they vote for.

 

We can't declare a winner in PA, AZ, NV, GA, etc. until all votes are counted, till the recount threshold has been breached, etc.

Those were made-up rules that changed by the day. If they were consistent from the start, then no states would have been called with 5 percent of the votes in, or 50%, etc. These networks were claiming that they couldn't call the last few states because the other candidate could "mathematically" still erase the gap. If this rule was applied to all states, then they sholudn't have called New York or Alabama either since a black swan is mathematically possible until one candidate won at least half of all outstanding votes. 

Which is more likely: that the provisional ballots would be so lop-sided and different from the already-counted 99+% of the ballots, or that the news media are stringing out the horse race for ratings?

 

 

 

The formula you need to figure out if the lead will flip in a toss-up state

For anyone following the nailbiter U.S. Presidential election, we keep wanting to answer the question - how does the next batch of votes affect the margin between Trump and Biden?

Let me give you a quick formula to use. You can do this faster than the TV hosts!

 

By how much will the next batch of votes close the gap?

 

Start with what we expect the Biden and Trump vote shares to be. Let's say 60% to 40%.

Take the difference of the vote shares, which is the voting margin. So 60% minus 40% is 20%.

How much the gap closes is equal to this voting margin multiplied by size of the batch. If 10,000 new votes are released, with Biden winning 60% of them, the gap will close by 20% * 10,000, which is 2,000.

 

For example, in Georgia, as of this writing, Trump leads Biden by 1,800 votes.

99% of votes have come in. So 1% are outstanding. The already-counted votes total about 5 million. So uncounted votes are approximately 50,000.

If all 50,000 get released, and they favor Biden 55% to Trump 45%, the margin is 10%. Multiply that by 50,000 means the gap closes by 5,000 votes. This would make Biden the winner.

[On the other hand, if the vote shares are reversed, of course, Trump will win Georgia. We can still do the same calculation. Instead of closing the gap by 5,000 votes, the gap widens by 5,000 votes.]

***

We can also turn the question around. What is the voting margin required for Biden to overcome the 1,800 margin in Georgia? Or, what is the voting margin required for Trump to overcome the 45,000 margin in Arizona?

What voting shares will be required in the remaining votes to flip the current result?

 

First, divide the current gap by the remaining votes. In Arizona, this is 45,000 divided by 340K, which is 13%.

Now, divide that margin by two and add it to 50%. This gives us 56.5%. So if Trump wins those remaining votes by at least 56.5% to 43.5%, he would overcome the gap of 45,000 votes.

***

A couple of issues to keep in mind when interpreting these numbers.

Whether the required level of performance is possible depends on which parts of the state the remaining votes come from, whether they are mail-in ballots, etc.

If the remaining votes are released in many batches, then remember that any batch that underperforms must be compensated by a future batch that overperforms, and vice versa.

Let's continue the above example and say Arizona releases 50,000 votes in which Trump wins 55% to 45%, which is slightly lower than required. So he would need to do better in the next batches but by how much?

The new remaining vote count is 340K - 50K = 290K. The gap widens by the voting margin x the size of the batch, i.e. 10%*50,000 = 5,000. So, the new gap is 50,000 votes. Divide 50,000 by 290K gives us 17%. Divide by 2 and add to 50% gives us 58.5%. So In the remaining votes, Trump needs to win at least 58.5% to get even.

 

 

How to act like a data scientist 13: no love for independence

[I know, some of you are hungry for coronavirus content. Don't worry, keep reading. I'll get there in this post :) ]

As the U.S. election workers continue to count votes, the scenario predicted by election watchers has emerged: Election Day votes favor the President while mail-in ballots lean Biden. Mail-in votes arrive mostly before Election Day but the Election Day votes were counted first.

Despite predicting this scenario, most pundits have not prepared the dataviz that is necessary to follow the situation. We need the following line chart, which I found at New York Times. (Maps do not have a time dimension!)

(P.S. It's better to cut out the left side so we can see the right side more clearly.)

The order of vote counting should not matter in the final result. The winner of an election is determined by the total vote shares, regardless of the mode of voting or its timing. So once we reach the 100% point in the above chart, the path getting there has no relevance.

However, the order of vote counting matters a great deal in partial results. If mail-in votes lean Democrats while in-person votes favor Republicans, and if in-person votes are counted first (such as in Michigan, shown above), then partial results will show a big lead followed by a tightening. Depending on who the ultimate winner is, there may be a flip to the other party.

Data scientists in particular should know this and never draw conclusions based on partial results.

Nevertheless, in real life, there is always pressure to do a sneak peek. People want us to do interim analyses, to see which way the wind is blowing. They assure us they will remain patient. When we deliver the interim analyses, people want us to project the full result. Such projection is dangerous because it usually requires an assumption of serial independence.

We are implicitly assuming that the first m units tabulated are a random subsample of the entire sample of size N. Applied to vote counting, we are assuming (wrongly as the media keep reminding us) that the sequence of votes is random.

This election has just proved why we should not assume random sequencing within a sample. This is not how it works in real life when a sample accumulates over time.

***

Anyone running online AB tests has encountered the problem of serial dependence. The sample is assembled over time as visitors show up at the website or the app. If, during part of the test, the company drives traffic through advertising, then in that window of time, there is an influx of visitors, and if the advertising campaign is targeted (as is the norm), the incremental visitors are not like the average visitor. Any interim results for that period can be generalized to the types of visitors in the subsample but not to the entire universe of visitors.

The same problem is plaguing the coronavirus vaccine trials. Recall that the vaccine developers announced midway through the trials that they stepped up recruitment of minorities and high-risk people. So the interim analyses will be based on subsamples that skew whites and lower-risk people. This is potentially dangerous because once the first vaccine is approved, there will be pressure to convert placebo to treatment. We might never get a read on those minorities and high-risk people! The partial result is not useless - it just does not generalize to the entire population.

[I asked about this at a recent webinar by an NIH statistician working on these trials, and at first, he said the non-random sequence of enrollment should not matter but then he realized it does matter in any interim analysis.]

***

Thankfully, serial dependence is easy to diagnose once the full sample is available. We compare the characteristics of the subsample with the full sample. In the vote count scenario, we will find that the proportion of mail-in votes shifts over time, which also means there is a difference in party affiliation and various demographics.

What's the lesson for data scientists? Always look out for implicit assumptions in our analysis methods. In thinking about analysis of partial samples, we tend to focus energy on how statistical significane is affected by the sample size while assuming implicitly that the subsample is representative of the full sample in its contents. That is frequently correct in textbook examples but usually wrong in real-life situations.

***

One last thing: dependence comes in many stripes. This post addresses serial dependence. The post from last week concerns dependence between variables.

 

Story-first thinking from the business press to Election Night

I'm been toying with this idea. I want to move away from the tired data vs intuition dichotomy. I don't think it accurately describes what is going on. The people who are labeled as intuitionists often include data in their arguments. The people who are experts at harvesting insights from data readily admit using intuition. So that distinction is hard to nail down.

Instead, I think a more useful dichotomy is data-first versus story-first. Some people have a story, and they are good at building support for the story by assembling pieces of data. Other people like to shake the story out of the data. You know who you are.

***

Last week, I came across a good example of story-first thinking in the business press. An article in Business Insider has the headline "Bed Bath and Beyond is slashing the coupons that have long been synonymous with the brand..." The article talks about how the struggling retailer may have found a lifeline in the pandemic; good for them!

Then came this juicy bit:

The writer tells us that BB&B customers don't care about those coupons anyway. To support that, she said "50% of [...] customers said that they shopped at the store because of its coupons". I mean, if half your customers came to the store because of the coupons, I'd say coupons are really important to this business!

Don't worry though because she rescued the story by telling us that the 50% is "a lower number than in previous years."

***

What's going on here? I don't think she was blinded by intuition. She literally used data in this argument and throughout the article so she qualifies as data-driven.

You can't say she messed up the data either. Her reasoning is standard - it just doesn't work with the numbers she's got. If only 5 percent of Bed Bath and Beyond customers said they shopped there because of coupons, I wouldn't have written this blog post.

I also do not think the writer is being manipulative. I think what she did came naturally to her - the story-first mindset, by which you assemble the pieces of data that support the story you have in mind. This style of thinking is commonplace, in my experience.

***

It's the day before Election Day in the United States. We are about to drown in a flash flood of story-first thinking. Journalists have been writing articles to warn against premature conclusions on Election Night. They even paint the worst-case scenario in which Trump leads in in-person voting but ultimately loses the election when mail-in ballots are tallied. (If I am not mistaken, the New York Times was the first to sound this alarm, way back in early September.)

For my international readers, here's how messy elections are here. There is no federally managed and standardized electoral process. Every jurisdiction, sometimes at the state level, sometimes at lower levels, sets their own rules. As a result, there is a built-in unfairness because your ease of voting depends on where you live. In particular, people who live in densely populated cities frequently have to wait in line, sometimes hours, to cast a vote. Because of the raging pandemic, some people just don't find it safe to stand in lines for hours, and many jurisdictions have created alternatives such as mail-in voting and drive-through voting.

When it comes to mail-in voting, the two parties have been fighting over it. An incomplete list of the major issues are: whether one needs to apply for a mail-in ballot or if the state should just distribute everyone a mail-in ballot; how many collection boxes should there be for mail-in ballots; whether churches or other private groups can set up their own collection boxes for mail-in ballots; the deadlines for sending in mail-in ballots; whether the Post Office should prioritize or retard the delivery of ballots; what to do with "late"-arriving mail-in ballots; what constitutes a "valid" mail-in ballot; when states can start counting mail-in ballots; how much time states are given to count mail-in ballots, etc. etc.

What all sides agree on is that incomplete data nuggets will arrive throughout Election Night, and the final story is unlikely to emerge until days or weeks later - unless the voting margin is decisively large.

So I put this question up on Twitter last week: "Given the media keep telling us to be patient next Tuesday, will the media refrain from horse-race reporting until the final tally is available?"

What's going to happen is that the political pundits will assemble these data nuggets to support whatever story they have in mind. In fact, this has already started. On cable news, they have been reporting the number of mail-in ballots already received in various states, and the party registration of these voters (including, mind you, a good chunk of unaffiliated voters who don't want to register with either party).

And then, they are telling us which party is ahead - clearly based on assumptions of how much "disloyalty" there exists within each party, and the vote split among the unaffiliated.

How do these journalists or data journalists form their assumptions? Remember that historical data are not useful since there is an explosion of mail-in voting due to the pandemic. The assumptions are colored, partially or wholly, by how they think the election will turn out. In other words, they will be engaging story-first thinking.

 

 

The complexity of modeling uncertainty

Today's post is a continuation of my previous post about the two election forecasting models, by FiveThirtyEight and the Economist.

Recall Adam Pearce's visualization of the correlation matrices from both models:

The left matrix allows us to evaluate the FiveThirtyEight model on one particular metric, how it projects the correlations of vote shares between states. Intuitively, we think that certain state results should be highly correlated while other pairs might not be.

The most interesting - and controversial - aspect of that heatmap is the presence of pink, which corresponds to negative correlations. The FiveThirtyEight model suggests that in certain state-pairs, when Trump does better than expected in one state, he does worse than expected in the other state. Andrew Gelman thinks the opposite: if Trump does better than expected in a blue state, Trump does better than expected also in red states. And thus, the Economist model has a different heat map of correlations than FiveThirtyEight's - no pink in sight.

I find the heat map too rich for its own good. So, I made some boxplots. (R code at the end)

I grouped the correlations by state. Each state has 50 correlations with other states (D.C. is treated as a "state" for the purpose of election modeling).

Here is a simple boxplot representation of the correlations of Biden shares between Washington, DC and the 50 states, according to the FiveThirtyEight's model:

To map this back to Adam's heatmaps, look at the plot on the left, pick out the column labeled DC (two-thirds of the way across), take that whole column, remove the one dark green cell (which is DC correlated with itself), and make a boxplot out of those 50 correlations.

As a reminder, the box in the boxplot contains the middle 50 percent of the data, or the middle 25 correlations. The dots on either side are considered outliers - there are none. For DC, the FiveThirtyEight model assigns negative correlation with 20 other states (to the left of the brown dashed line).

***

Here then are the sets of boxplots for both models for all 51 states. The states are ordered by increasing average correlations in the Economist model so that it's easier to compare the two sides. (Notice that I placed the Economist model on the left, the reverse of Adam's order. Also, I plotted Biden vote shares instead of Trump vote shares.)

This graph reveals some major differences in how the two models were constructed.

(a) The spread of the boxes and particularly the "whiskers" are much wider in the FiveThirtyEight model than in the Economist model. The boxes (middle 25 correlations) have an average range of about 0.5 vs 0.3; that's a big difference. The FiveThirtyEight model incorporates a lot more variability of this type.

(b) The widths of the boxes are more variable in the Economist model than the FiveThirtyEight model. For the Economist, the boxes are narrow to wide; for FiveThirtyEight, it runs from wide to wider. In the Economist model, the top few boxes have very tight clustering.

(c) There are no negative correlations in the Economist model at all, nothing to the left of the brown dashed line. By contrast, the FiveThirtyEight model has negative correlations for most states.

(d) The biggest differences occur in the top six boxplots (DC, HI, MD, NM, GA, TX). In all these states, the Economist model expresses much lower uncertainty about correlations than the FiveThirtyEight model, and also the Economist model assigns these states the lowest average correlations.

(e) The midpoint of each box, i.e. the median correlation, is roughly sequenced the same although there are rather drastic differences. For instance, Texas (TX) is correlated over 0.5 with half the states in the FiveThirtyEight model while any correlation over 0.3 in the Economist model is an outlier.

All these observations relate to how the two models deal with the variability between states. As you can see, there are many moving parts.

***

Let's turn our attention back to DC (the top row). Here are the two relevant boxplots:

The Economist's model expresses the view that DC's going to vote for the Democratic candidate regardless of what happens in other states, thus low correlations tightly cluster around 0.15. One can fit almost the entire Economist's boxplot (excluding MD) into the right half of the FiveThirtyEight's box! The FiveThirtyEight model has a box that ranges from -0.1 to 0.3, and 20 states are negatively correlated with DC. That means, the more Biden wins in DC, the more he loses in those states.

Looking at the individual simulations (from Adam's tool):

Neither model calls DC for Trump under any scenario. FiveThirtyEight thinks that the more Trump overperforms in DC, the lower his vote share in Alabama (one of the states with negative correlation with DC). The Economist model basically delinks the two, the cloud of dots are almost vertical. [Note: The regression lines shown on Adam's plots may not be correct although it's hard to tell where the dots are where it is very dense.]

***

These boxplots allow us to analyze the heatmap, and describe the differences between the two models more effectively. I have no idea how much variability there should be between states

 

P.S. Here is R code that generates the side-by-side boxplots.

# starting point is bcorrlist: a list of the two correlation matrices
# stored in data frames

# stack, and remove self-correlations
bcorrstackedlist = lapply(bcorrlist, stack)
bcorrstackedlist = lapply(bcorrstackedlist, function(x) x[x$values != 1, ])
# sort order choices
stateorderlist = lapply(bcorrlist, function(x) round(sort(apply(x,2,mean), decreasing=TRUE),2))

# set parameters for boxplot
myboxplot = function(data, gorder) {
    boxplot(data[[1]]$values ~ factor(data[[1]]$ind, levels = gorder),     horizontal=TRUE,
    ylim=c(-0.5,1), col="gray80", axes=FALSE, cex.title = 0.6,
    xlab="", ylab="", main = paste(capitalize(names(data)), "Biden-Share     Correlations by State"),
    whisklty=7, whisklwd=0.25, outcex=0.5, outpch=16, boxtwd=0.1, medcex=0.6,
    whiskcol="gray80", staplecol="gray80", boxcol="gray80", outcol="gray50",     medcol="gray30")
    axis(1, at=seq(-0.5, 1, by=0.2), col="gray", col.ticks="gray")
    axis(2, at=1:51, labels=gorder, cex.axis=0.6, col="gray", col.ticks="gray")
    abline(v=0, col="brown", lty=2)
}

# make charts
par(mar=c(3,3,1,1), mgp=c(1.5,.3,0), las=1, cex.axis=0.6, tcl=-0.2)
# Economist
myboxplot(bcorrstackedlist[1], names(stateorderlist[[1]]))
# FiveThirtyEight, order by FiveThirtyEight
myboxplot(bcorrstackedlist[2], names(stateorderlist[[2]]))
# FiveThirtyEight, order by Economist
myboxplot(bcorrstackedlist[2], names(stateorderlist[[1]]))

 

Two election forecasting models, negative correlations, and model assumptions

This post by Andrew Gelman addresses an issue that has always bothered me when I look at stochastic simulations.

It's very trendy to run simulations. It's conceptually simple. Imagine forecasting the future - there are some uncertain factors, leading to numerous possible scenarios, you flip a whole bunch of coins to "simulate" the future, and then analyze the distribution of simulated outcomes.

Beneath this simplicity is a tricky issue. You put probability distributions on the variables in your model. Typically, you specify the mean and standard deviation (which tunes how variable your variable is) - although some more advanced distributions require more parameters. You do this for each variable. Then, you press run.

When you press run, you've just made a very strong assumption, that every variable is independent of other variables. If you have 10 variables in your model, there are 10*9/2 = 45 pairs of variables, all of which you've imposed independence. That is unrealistic in almost every real-world application!

The simulation setup becomes a lot more complex if you give up independence. In the example of a 10-variable model, you'd have to specify dozens of pairwise correlations. You don't typically have enough data to estimate all these numbers reliably. So you end up straddling the fence - you impose some structure on the pairwise correlations, e.g. a subset of variables are independent of each other, which reduces the number of required estimates.

***

This situation is illustrated very well by Andrew's post already mentioned. This post is the latest in a series of related posts (here, here and here) that address differences between the election forecasting models built by FiveThirtyEight (Nate Silver) and the Economist (Gelman and associates).

You might find it worthwhile to read my 2016 post about election forecasting (link) before proceeding. I agree with Andrew's final point that given the low frequency of U.S. presidential elections, it's impossible to say who has the best model... scientifically. In the court of public opinion, the FiveThirtyEight model has built a robust reputation. (Okay, so it's only his final point, ignoring all the epilogues and postscripts!)

In the mentioned post, Andrew spends time looking at the correlations of Trump vote shares between states, as projected by FiveThirtyEight, pointing out several unexpected outcomes. These cases are unexpected because the Economist model outputs very different correlations. Then, he documents his thought process of how their team derived the between-state correlations while speculating on what the FiveThirtyEight crew are doing. (The two sides have not spoken directly at the time of the post.)

***

The discussion paints a very true picture of data science modeling. Everyone makes assumptions, lots of them. Each side has justifications for their assumptions - firstly for ourselves, then for others if requested. Observers have to decide whose rationale they believe more. One of the ways to judge a simulation model is to look at the projected outcomes, and see if they make sense. That's the first motivation of Andrew's post.

Before proceeding, you may want to look at my tour of the Oxford epi model, which gives some background on modeling in general, particularly the idea that these models use data but also impose structural assumptions, as well as assumptions on parameters. Start here.

I'm skipping right to the key point of Andrew's post, concerning the correlations between Trump vote shares in Mississippi, a reliably red (Republican) state and in Washington, a reliably blue (Democratic) state. This scatter plot brings out a number of features of the FiveThirtyEight model that are absent from the Economist model:

Of greatest interest is the negative posture of the cloud of dots on the right side, each presenting one of 40,000 simulated Election Days. (In technical terms, a negative correlation between Trump's vote shares in Mississippi and Washington states). If you draw a line through the cloud, it goes from 11 o'clock to 5 o'clock. The FiveThirtyEight Model infers that the better Trump performs in blue Washington, the worse he does in red Mississippi. If Trump outperforms in Mississippi, he underperforms in Washington.

For example, a reasonable outcome on both models predicts Trump winning 60% of Mississippi and 40% of Washington. According to the FiveThirtyEight model, in the scenario where Trump wins 55% of Washington, a real upset, Trump is projected to lose Mississippi. Andrew finds that incredible. I also have difficulty understanding how this can be the case. By contrast, in the Economist model (left side), the scatter plot has a positive posture, meaning the state vote shares are positively correlated - the better Trump does in one state, the better he performs in the other state.

Some responses to Andrew say including the FiveThirtyEight's scenario is reasonable though remote as it represents a radical party realignment. The scatter plot shows many such outliers, though. In most scenarios in which Trump beats Biden in Washington, he loses Mississippi to Biden.

Negative between-state correlations are a part of the FiveThirtyEight model, and not present in the Economist model. Adam Pearce helpfully put together this dataviz that allows you to look at every pairwise correlation. The chart above came from his tool.

This gets at the heart of the tricky simulation problem. What's a reasonable assumption for an extremely remote possibility, like Trump winning Washington or Biden winning Mississippi? Even very smart people will disagree!

***

We can explore the discrepancy further.

Let's break down the projected vote share of a given state (say Washington) in a stylized way: (TVS = Trump vote share)

The most naive and not-very-accurate model estimates TVS for every state to be equal to the estimated national Trump vote share.

TVS (Washington) = a + b*TVS (national) + unexplained (hereafter, U)

To this, we can add state-specific effects. So, the national TVS is modified up or down based on historical data. This model can differentiate between red, blue and tossup states.

TVS (Washington) = a + b*TVS (national) + c*State_Effect(Washington) + U

To this, we can add two-state interaction effects:

TVS (Washington) = a + b*TVS (National) + c*State_Effect(Washington) + d(X)*State_Effect(X) + e(WX)*Paired_Effect(Washington*State_X ) + U

As mentioned at the start, to manage the complexity of this model, we'd rather not think every state effect as correlated with Washington's. We pick a subset of states (X) that have expected correlations and include only those terms. Also think of the paired effects as modifications of the individual state effects, and so I add the series of state effects for the subset of X states as well.

Andrew's argument says that if Trump does surprisingly well in Washington, he would have done well nationally. This suggests that the national effect in the Economist model is quite important.

The other argument requires a negative correlation between Washington and Mississippi, and so the action is in the paired effect and state effects.

After I clean up some code, I'll put up another post that pursues this graphically.

Dr. Backlog jumps the pond

The UK has joined California, Texas, etc. in the list of organizations that couldn't keep their testing data straight. It just admitted that over the last week or so, it under-counted Covid-19 cases by 30 percent.

(Graphic from BBC.)

The official reason is an Excel limitation, which is a brilliant stroke of PR (I have previously praised the PR firm behind the UK's coronavirus response. See here.) This message is intended for those who do not use Excel, which, I guess, is the majority of the UK population. Among those who have been exposed to Excel, there exists a set of vocal dissenters, people who hyperventilate at the mention of Microsoft's software, and they will help propagate the message.

Anyone who uses Excel a lot will not fall for this.

Unlike California and Texas, where the officials remain mum about the details of their supposed faux pas, the UK government explained to the Daily Mail what this Excel error is. We are to believe that there is a master spreadsheet recording all cases, that spreadsheet ran out of space, nobody noticed it until a backlog of 16,000 cases has stacked up, and they fixed the issue by splitting the data onto multiple spreadsheets.

One word: implausible.

***

The limit on the number of rows on a single Excel spreadsheet is above 1 million. The cumulative number of cases in the UK as of October 2 was fewer than 500,000. Assuming that they collect data for each individual case, so that there is one row of data per case, the spreadsheet should be at least 50% empty.

I'm not convinced that's the workflow. It would seem more likely that there is one line per reporting unit, not one line per individual case. If that is so, then the spreadsheet is even smaller.

Let's play along and assume somehow someone created a process filled with duplicates and errors or other horrors such that the spreadsheet manages to overrun 1 million rows.

We now have to believe that the Excel spreadsheet is maintained by a machine without a single human opening it. If a human is transferring the data onto this master spreadsheet, as I expect to be the scenario, how is it that this live person is not alert to the space outage? One simply cannot add any more rows to the bottom of the sheet!

If a machine is programmed to add rows to this spreadsheet, we now have to believe that neither Excel nor the program returns any error messages or warnings when the spreadsheet runs out of room!

Let's keep playing along. Against all odds, the spreadsheet is somehow completely full and nothing more can be added. Neither humans nor machines notice anything odd. The new data simply fell into a black hole.

Now, how do we explain that the case counts are increasing each day for a week before they noticed the missing cases?

From September 25 to October 2, the period in question, the UK reported on average 4,368 cases before admitting the mistake. The revised data bumped the daily average to 6,348 cases, which means the public was given a number 30 percent below the real tally.

According to the Daily Mail, "some 16,000 confirmed infections had to be added to the daily totals running back more than a week." If we buy the Excel story, what should have happened is that one day, the spreadsheet ran out of space, and from that point on, the case count plateaued with no more additions.

What should have happened is suddenly no more cases are recorded, and the error is immediately obvious.

P.S. Lets' keep playing. So they keep one gigantic spreadsheet not just of positive cases but of all test results from which they extract the cases. As of October 2, the U.K. said they have "processed" 22.7 million PCR tests. Remember the limit of a single spreadsheet is 1 million, which has been passed long time ago.

***

Dr. Backlog sure chose a curious time to jump the pond. The undercounting occurs right as cases were spiking in the U.K., France, Spain, and all over Europe, and governments had to make the difficult decision of whether or not to re-impose restrictions to ward off a second wave.

Yes I will be the first to admit the data world is messy, and there can be innocent explanations for data discrepancies. However, any experienced data analyst realizes one cannot lazily cling to the most convenient explanation; all possible explanations must be investigated, including politically incorrect ones.

Good news from Cornell

Source: Redbubble.com

Some colleges and universities have re-opened for in-person classes for the fall term. Previously, I examined the Cornell model (link), which outlines a very aggressive plan for preventing the virus from entering the campus, and then snipping transmission chains during the semester. Also, we learned that several schools that opened in August quickly ran into trouble as hundreds if not thousands of cases surfaced, and some decided to return to online teaching. (link)

I just attended a fantastic webinar by Peter Frazier, who's part of the Cornell team that did the statistical modeling work supporting the re-opening. The webinar was sponsored by Columbia.

It's about three weeks after Cornell reopened for in-person instruction, and the situation in Ithaca is significantly better than on most campuses. The following chart best illustrates why the team is taking a victory lap, although they absolutely realize that there are still risks ahead.

[Applause]

[Note: I expunged a number of Excel defaults from the above chart, including line colors, legend, and date labels. Also, replaced infections with cases, which is a more precise description of the metric.]

As I indicated in the previous post, if Cornell's plan proves successful, it would repudiate what the U.S. government did in the early phase of the pandemic. It shows that extremely aggressive surveillance - and in particular, broad-based testing - will dampen case rate to around zero, and keep it there. That's what other countries like China and South Korea managed to do.

At the end of the talk, Prof. Frazier offered advice to data scientists who want to make real-world impact. I agree with every bullet point and am reiterating the whole list here.

1. Focus on solving the real-world problem, rather than what can be published. He discloses that they have not written a paper about the work they did here. This is advice I give to researchers who want to transition into industry positions.

2. Create mental models of your computer models. A different way of expressing this point is that the modeler should develop some intuition behind how the models work, even if the methodologies are complex and opaque.

3. Communications. He spent a lot of time explaining the model to the public, and the team was open to feedback and continuously listening to the community. People are emotional about the subject, and some of the commentary were harsh but it's part of the job. Frazier's presentation today is impressive evidence of his communications skills.

4. Understand the political environment. Different groups have different objectives and biases.

5. He calls this "luck." The key decision-makers at Cornell are scientists by training, so they have an easier time persuading them to believe in science. That's an important point when it comes to "choosing your boss." For many data scientists, jobs have been plentiful, and job satisfaction will depend on whether you're fighting with your boss all the time. Imagine the scientists at FDA or CDC who had to deal with political appointees who are reportedly pressuring them to alter their language. Those meetings would be a completely different beast than what the Cornell team had to deal with.

Throughout the presentation, Prof. Frazier pointed to other "luck" factors that favor Cornell's response. For example, Ithaca is very isolated. He said that the nearest towns are an hour's drive away. Also, their veterinary school has abundant expertise in PCR testing, and has been extremely helpful.

He also reported that pessimists were much more vocal than optimists, and so assumptions may have been driven more conservative than warranted. Modelers may also have a conservative bias because better safe than sorry.

***

In his presentation, Prof. Frazier demonstrated the power of statistical modeling. The core simulation model is highly complex, with many moving parts and assumptions, as I discussed in the earlier post. The assumptions are necessary because most of those parameters are not directly observable, and highly uncertain. While the model was not "accurate" in the sense of forecasting precise outcomes, it accurately predicted the general trends. Making assumptions is not a bad thing!

Then, we saw Occam's razor in action. Statisticians have long preached that models should be as simple as possible, but not simpler. The core model is actually built on top of simple epidemiological models, and those models alone were able to capture the general trends.

***

Since the time I reviewed the Cornell plan, there were a couple of important additions that probably contributed to the on-the-ground situation being better than the original projection.

First, the frequency of surveillance testing was doubled for undergraduates from once a week to twice a week. The insight is to tailor the testing rate to the amount of contacts expected in different subgroups. The mathematical models help calibrate what level of testing is required to keep the infection rate below a critical threshold.

Second, to work around the limited capacity for contact tracing, and to take advantage of their expertise in PCR testing (via the vet school), they are testing the entire social circle for each positive case. They call this "adaptive testing". The goal is to reveal as many hidden cases as possible before the virus has a chance to spread further.

Testing is a huge commitment in terms of money, resources and time. Cornell is conducting over 5,000 tests per day - obviously, they are returning results very fast. It appears also that the students are cooperating, and grasping the key issues. This pandemic is a test of community spirit.

***

Perhaps lost in our national conversation is the decisiveness of early, aggressive action. Public-health experts have stressed this point from the start, and the Cornell experience (relative to other universities) shows they were right.

Take compliance to testing as an example. When there isn't much virus in circulation, we can accommodate some degree of non-compliance because the non-compliant person will rarely run into an infectious contact. When the epidemic is in full swing, the chance of interfacing with infectious people is much higher, and non-compliant members are more likely to spread the virus around.

We can't think linearly. Any community is moving toward one of two final states: almost no virus in the community, or almost everyone gets infected at which point the virus also spreads slowly because most contacts are between infected people. Without intervention, the second state is inevitable. Once an epidemic gets out of control, it's very hard to revert to the first state.

Cornell's success to this point is showing us that places like China and South Korea can indeed eliminate the virus with early aggressive actions. These countries don't have to fake the data if they did the right things.

Whether the success will last is an open question. It's only three weeks into the term. Prof. Frazier noted that they strongly discourage students from traveling outside Ithaca. I'm not sure this is possible during breaks and Thanksgiving. Similarly, countries that successfully fought off the first wave still face risks if the virus is re-imported from other countries that failed in their initial response. For Cornell, I'm guessing that they will have to roll out arrival testing after each long break.

Masks and vaccines

United Nations Covid-19 Response

Wasn't planning on blogging today but the news cycle won't leave me alone :)

CDC Director Dr. Redfield made a bit of a stir yesterday when he told Congress - and I paraphrase - that masks may offer individuals greater protection than vaccines.

This is a very thoughtful statement which rewards some further comments.

***

The way medicine is practiced, we are often misled to over-estimate the impact of drugs and vaccines. When the doctor says take pill X to treat condition Y, we think if we take pill X, condition Y will be cured. In reality, for most conditions and meds, if we take pill X, there is a chance C that pill X will cure condition Y. C is not 100 percent, and is usually way below 100 percent.

Flu vaccines are said to be 50 percent effective or less. What does that mean? Perhaps, that half the people who've gotten flu shots will be protected while the other half will still catch the flu. This isn't how experts use the word "effectiveness" but let me stay with this intuitive definition for now. (I'll turn to the technical definition in the last part of the post.)

This situation reminds me of the famous put-down of TV advertising: half the budget is wasted but we don't know which half.

The trouble of course is that if you halve the TV spending, you will still waste half the money while also losing half the benefit. Same with the vaccines. If only half the people get vaccinated, a quarter of the people will be protected.

The FDA has announced that a SARS-Cov-2 vaccine with 50% effectiveness is acceptable. Staying with the intuitive definition, this means getting the vaccine shot is like buying a lottery ticket with a 50 percent chance of winning. This is not news though. Taking any pill X or undergoing any surgery  has always been a lottery with C percent chance of winning. The value of C depends on the treatment.

One of the reasons why some vaccinated people will still catch the flu is biological variation. Different people respond to the vaccine differently.

***

What about wearing a mask? The mask works as a physical barrier and its function is not affected by anything biological. The same mask worn properly confers the same degree of protection on anyone. Any decent mask blocks some fraction of virus from entering one's respiratory system. Imagine an infectious person coughing directly at me. The mask reduces the amount of viral projectile even if it is not eradicated.

In other words, wearing a mask has a high C. Even though I am a skeptic of many things, mask wearing is not one of them.

Whether the vaccine will work for me or you depends on biology. I have no control over that. Either my body responds to the vaccine or it doesn't. The mask, on the other hand, works on everyone who wears one. That's what the CDC Director is saying, and he's right.

***

Now that I've gotten your attention, let's switch to a proper interpretation of vaccine effectiveness. We have a coronavirus vaccine that demonstrates 50% effectiveness in a clinical trial. One possible scenario is it reduces the infection rate from 1% to 0.5%. (For reference, in the Moderna trial, they assume the baseline infection rate with no vaccine is 0.75% over a six-month period.) That's a 50% drop, and statistically significant.

For every 1000 patients receiving the vaccine shot, five patients will benefit from it as they would have caught Covid-19 without the vaccine. Another five will get sick despite being vaccinated. So C is 5/1000 = 0.5 percent for the vaccine. Getting vaccinated is like buying a lottery ticket with 5 percent chance of winning.

In medical jardon, we say the number needed to treat is 1000/5 = 200. For one person to benefit from vaccination, we must administer 200 vaccine shots. Similar to TV advertising, we can't know who would benefit ahead of time. Statistics tell us that on average, one out of 200 will.

Notice that C isn't 50 percent but only 0.5 percent. It can't be 50 percent because most people won't catch Covid-19 even without the vaccine. For every 1000 people vaccinated, 995 of them won't get sick. Of those 995, 990 would have stayed healthy regardless of vaccination. The 50% effective vaccine really only benefits 0.5% of those who took the shot. (Note again that I'm using the assumption from the Moderna clinical trial design.)

But this is not the full story. The practical effectiveness of the vaccine depends also on what proportion of the population takes the shots. Let's pretend that half the population get vaccinated. Then the 0.5% benefit of vaccination is halved. The infection rate in the population is reduced from 1% to 0.75% since the infection rate for the unvaccinated will remain at 1%.

Herd immunity now complicates this picture. If enough people get the vaccine, typically 60% or higher, the amount of virus circulating in the population is suppressed enough that the infection rate for the unvaccinated will dip below 1% so collective action creates a positive externality.

 

 

 

Confusion over case trends and positivity ratios, an aftertaste of targeted testing

We are now experiencing the aftertaste of targeted (aka triage) testing, a policy that has unsavory consequences that were predictable (link). Targeted testing is the idea that only people with severe symptoms or at high risk of infection should get tested for the novel coronavirus. Compared to a broad-based testing plan (whether comprehensive or randomized), targeting economises the number of test kits. The basis of any targeting strategy is the skill to predict who is infected. The group that is targeted should have a higher infection rate than the group that is not targeted. It's important to bear in mind that targeting not only suppresses how many people are tested but also affects who gets tested.

One way in which targeted testing hurts us is that it ruins our datasets from which we obtain insights about the pandemic. This post is prompted by Ken's comment on my previous post on casual presumptions of randomness:

The number of daily new cases in some countries doubled as fast as every 2 days. This didn't mean a greater rate of infections, it was simply a result of catching up all the non-tested cases.

The connection between infections, tests, and cases is complicated, and the media shift back and forth between the rate of cases (positive test findings) and the positivity ratio (proportion of tests that came back positive). I spent some time explicating these relationships, and hope that this post will succeed in helping you gain some intuition around those concepts.

We start by decomposing the positive test findings. In the flow chart below, starting with the dark gray box on the left side, we examine a fixed population of people. At any given time, the people are divided into three "compartments," the infectious people who currently have the virus, the susceptibles who have not been infected, and the recovered (which by convention, includes those who died from Covid-19).

Also by convention, anyone can be infected only once, and we thus ignore recovered people from further consideration. From the dark gray box emerge two paths that terminate at positive tests: the upper path consists of infectious people while the lower path, susceptibles. Some portion of the population are infectious; within that, some proportion have been tested for Covid-19, and within that, some proportion have tested positive. This last tally is one part of the confirmed cases. The other part of the confirmed cases comes from susceptibles, represented by the lower path.

The number of confirmed cases is the number of positive tests. This is a key metric on all Covid-19 dashboards. The other important statistic is the positivity ratio. This is the number of positive tests (represented by the dark red box on the right) divided by the number of tests conducted (the sum of the two yellow boxes).

The next diagram describes the drivers of the two dashboard statistics:

• the weights of infectious or susceptible compartments in the population (excluding recovered)
• the probability of getting tested among the infectious or susceptible group
• the weights of infectious or susceptible people within the tested subpopulation
• the positivity ratio for the infectious or susceptible people who got tested (call these the group-level positivity ratios to distinguish from the overall positivity ratio)

The overall positivity ratio is affected by three signals. First up is the rate of infection in the population, reflected in the relative weights of the I and S compartments. Second, the testing regime imposes a selection effect on these weights, producing the second set of weights. The notorious triage or targeted testing protocol restricts testing to patients showing severe symptoms or at highest risk - these disproportionately arise from the infectious group, and so the second set of weights are more skewed toward infectious people, compared to the first set of weights. Thirdly, test accuracy affects the number of positive test results, given any tested subpopulation.

Test accuracy is typically measured by sensitivity and specificity. On the upper path, the positivity ratio of the infectious-tested group is just the sensitivity, which we assume to be 90 percent. On the lower path, the positivity ratio for the susceptibles-tested is the rate of false positives (as this group is not infected), which is 1 minus the specificity, which we assume to be 5 percent. (For the sake of this post, we assume that the performance of the diagnostic tests has not changed during the pandemic and thus the group-level positivity ratios are fixed. This simplifies the following examples.)

 

What's special about 5% positivity ratio?

The media keep reminding us that the CDC wants the positivity ratio to fall below 5%. What is the significance of the 5-percent threshold? I can explain using the flow chart.

We hope for the epidemic to dissipate. When this happens, the infectious compartment dwindles to zero while the susceptibles stabilize. This means there will be no more flow through the upper path. All of the positive test results will come from the lower path (and are false positives).

The overall positivity ratio is a weighted average of the group-level positivity levels, the weights being the proportions of those tested who are infectious or susceptible. In this extreme case, the weight is entirely skewed towards the susceptible-tested, for which the positivity ratio is 5 percent. Thus, the overall positivity ratio will be 5 percent.

In other words, when the positivity ratio dips to 5 percent or below, new infections should be dying out.

 

What's the relationship between rate of cases and rate of infections?

I will now use this setup to investigate Ken's comment. He was reacting to the naive interpretation of case growth as infection growth, popularized by the media. Such a conclusion is invalidated because of targeted testing.

Note that infection growth cannot be directly observed. By contrast, case growth is reported on every dashboard. In Ken's scenario, the number of cases (i.e. positive test results) are growing rapidly.

Start from the dark red box on the right of the flow chart, and work backwards. The jump in positive cases may come from either of the two paths.

First, consider the lower path representing susceptibles. During the course of an epidemic, more people get infected, which reduces the number of susceptibles in the population, which in turn decreases the number of susceptibles testing positive. So if the cases from this lower path have to go up, then it must be that susceptibles are suddenly more likely to get tested.

One such scenario is if politicians expand targeted testing to those with mild or no symptoms. This is Ken's hypothesis.

Such a policy shift also affects the upper path. In fact, infectious people are more likely to experience mild symptoms. Even if the rate of infection is stable, the number of positives tests arising from the upper path grows. If infections are also accelerating, both factors contribute to case growth.

Ken argues that the entire case growth can be attributed to the expanded testing capacity while the media generally interprets the case growth as infection growth, ignoring possible shifts in testing policy.

The trouble is that both factors are at play, and we don't know the relative sizes of their contributions. This confounding is brought to you by targeted testing! (I can't resist: it's free and comes with a hidden cost.) This example demonstrates the aftertaste of targeted testing. It messes up the data so we can't interpret them properly.

It doesn't appear that Ken's scenario is happening in the U.S. as the federal government has continued its misguided targeted testing policy. Some politicians even grumble that we are conducting "too many" tests. (This can only happen if we were conducting too few tests, and thus under-reporting cases early on.)

 

How does targeted testing affect the positivity ratio?

In Ken's scenario, he assumes that the increase in testing accounts for the entire jump in cases. So, let's stipulate that the rate of infections is stable, and look at how the testing policy affects the positivity ratio.

The overall positivity ratio is the weighted average of group-level positivity ratios, so the key is to understand what happens to the second set of weights, which describes the mix of infectious and susceptibles in the tested subpopulation.

Under a randomized protocol, the second set of weights just represents the underlying mix of infectious and susceptibles since both groups have equal probability of getting tested. Under a targeted protocol, the second set of weights is biased toward infectious who are more likely to experience symptoms. Thus, early on, the positivity rate is over-stated. If targeting restriction is relaxed, the bias toward infectious is reduced. Thus, the relative weight of susceptibles-tested is pushed up, and so the positivity rate is driven down.

Since the rate of infections is assumed stable, the shift in positivity rate tells us only about the testing policy. We've just discovered how politics enter the data.

***

Targeted testing introduces bias into the data, which pollutes the link between the rate of cases and the rate of infections. Remember: we use the rate of cases to infer the rate of infections because in the real world, no one knows the rate of infections.

Targeted testing is the aftertaste that you can't seem to get rid of. It polluted the data when first implemented. Then, every subsequent change in policy, and the gradual behavioral adaptation to these changes, add to the confusion. The effect on number of tests is observable but the effect on the types of people taking the test is not.

This tragedy on the national level is now being replicated on individual school campuses. Many schools are doing targeted testing only on students who self-report symptoms. Their dashboards are reporting confirmed cases, the trend of which is being interpreted as the trend of infection. What is the point of data-driven decision-making when the data are known to be bad?