Big Data, Plainly Spoken (aka Numbers Rule Your World)

All else is never equal

Andrew's short note about someone's hypothetical Trump victory scenario prompted this post.

The concocted scenario is: "Had 45,000 gone the other way in Arizona, Georgia and Wisconsin, Donald Trump would still be president." This sounds like a reasonable hypothesis. The analyst took the three states that Trump lost by the smallest margins, and flipped them.

But such an analysis is flawed. The flaw is the unspoken assumption of "all else being equal". Said differently, the assumption is that those 45,000 voters represent an independent bloc of voters - their voting propensities are perfectly aligned with each other, and yet perfectly uncorrelated with other voters.

As Andrew explained, if 45,000 more votes had gone to Trump, this would have suggested a shift in national swing towards Trump: the voting margins in many other states would also have shifted away from Biden. The state-wide voting margins are themselves correlated, so an all-else-equal condition is unrealistic.

If those 45,000 voters in Arizona, Georgia and Wisconsin made a different decision, many more voters in other states probably also would have changed their votes!

Andrew then did a first-order approximation of this correlation. His model assumes that the 45,000 extra voters in those three states embodied a purely national trend so that the same percentage swing occurred in every state. Under this condition, we'd expect about half a million more votes for Trump nationally, rather than just 45,000.

(Note: Because of the strange electoral vote system we have in the States, we'd need to see a national swing of 500,000 votes so as to find 45,000 extra Trump votes in those three states, which would be necessary to flip the electoral votes of those states, thus flipping the election.)

Of course, the first-order approximation is too simplistic. The correlation of state voting margin to the national voting margin depends on whether a state is red, blue, or purple, on turnout, and other factors. One can build more sophisticated models but the principle is still the same: different parts of the system are correlated, and so manipulating one part almost definitely causes other parts to change.

***

All-else-equal assumptions are extremely popular.

In sports commentary, for example, one often hears arguments, such as if the star 3-point shooter on the team had scored the average number of baskets, then his or her team would have wiped out the deficit and turned the loss to a win. However, if that player had played better, the whole team would likely have played better, the other team might have adjusted their tactics, etc. If more of that player's shots have gone in, s/he might have attempted more (or fewer) shots.

Businesspeople like all-else-equal assumptions too. Consider an online retailer who currently charges customers to return merchandise. What happens if the return fee is eliminated? We'd expect the frequency of returns to jump, costing the retailer a bundle.

But that's not the only behavior that may change. The retailer might enjoy an increase in sales as some customers previously didn't make purchases because they disliked the return fee. Another jump in sales could result from customers buying different sizes or styles of the same item, with the intention of retaining just one. These effects may apply across all merchandise and regions or they may be restricted to certain product categories or regions.

***

On a second look, the Trump re-election hypothesis requires some further assumptions about voter behavior. Biden's total vote margin in those three states was just under 45,000, according to Wikipedia. If voters flipped their votes from Biden to Trump, it would only require half of those voters to bridge the gap. Thus, the concocted scenario is one in which 45,000 additional Trump voters showed up to even up the margin. In addition, the scenario requires that no additional Biden voters appeared. Or, it requires that a net 45,000 additional votes for Trump materialized.

Ironically, illegal measures to rig the election break these correlations. If one were to just stuff the ballot boxes with 45,000 additional Trump votes, then it would be possible to close the gap without seeing other voting margins shift.

Do better

The Covid-19 pandemic unleashed a torrent of medical research, but the quality of the presentation is frequently unimpressive.

In this post, I cover an example typical of the problems I've encountered when reading Covid-19 vaccine studies. In the "final analysis" of the J&J vaccine (published here in a NEJM peer-reviewed paper), the authors made the following unambiguous claim:

The Kaplan-Meier cumulative incidence curves for moderate to severe-critical Covid-19 separated after 14 days (Fig. 2A)

What should readers expect to find in Figure 2A? Let's guess: evidence that the cumulative incidence curves (that is, of vaccine and placebo groups) separated at 14 days after the J&J shot was administered. The claim was precise and unambiguous: the separation occurred on the 14th day, not 13th or 10th or 15th or 21st.

Here is all of Figure 2A:

In a flash, the chart might be seen as confirming the finding. But...

... since we are sophisticated consumers of data graphics, we should notice

1. the chart is much shorter than it is wide: the cumulative incidence rate is only about 0.5% in the first 14 days, while the vertical axis extends to 12.5% (the median participant in the trial has only reached 120 days at the time of the analysis)
2. the dots have a diameter of 20 points, which is equivalent to 0.6% in the vertical scale: these dots are so big that one can fit only four such dots inside each vertical tick interval
3. it appears that 21 dots are squeezed between tick marks on the time axis, causing massive overlapping
4. the blue dots are printed on top of the red dots

As shown below, in the first 14 days, the chart crams 14 dots each of which has a diameter equivalent to 0.6%, while the entire 14 values span a total range of 0.5%. It is simply impossible to tell when the two curves "separate". It is highly unlikely that separation occurred on the 14th day as asserted.

The density of these dots robs their presence of any meaning. There is in fact a line through the centers of those dots and if the chart presents only those two lines, we'd be able to confirm whether or not separation occurred on the 14th day.

***

Found right below, Figure 2B directly contradicts the claim.

Figure 2B plots the estimate of vaccine efficacy (VE), which is a function of the relative cumulative incidence rates at any given time t. As I explained in this post, 1-VE is essentially equivalent to the height of the cumulative value of the vaccine group divided by the height of the cumulative value of the placebo group.

For example, on Day 15, VE is shown at about 50%. That means 1-VE is 50%, which means the height of the placebo group's value on Day 15 is roughly double that of the vaccine group on the same day. If on Day 14, the blue dot was at 0.5% as eyeballed above, then the red dot would have been at around 0.25%. Thus, the curves have already diverged by day 14, and did not just "separate after 14 days".

Figure 2B also shows VE around day 10 to be 30%. Thus, 1-VE = 70% meaning the vaccine group's cumulative value by Day 10 is 70% that of the placebo group on that day. So we know a gap already existed by the 10th day.

***

In order to establish Day 14 as the watershed, one would ideally look at cumulative case data on Day 14. Thankfully, raw data are provided underneath Figure 2A. But wait... the researchers chose to show raw data in 21-day increments, making it impossible to directly verify their claim. (Meanwhile, they chose to print huge dots on the chart for each day, obscuring the underlying values.)

Further, for Figure 2B, the researchers did not stick to the 21-day increments. Instead, they provide data in 30-day increments. Notice that no tick values coincide on Figure 2A and Figure 2B until day 210 (7 months after trial start date). Thus, it is challenging to correlate these two diagrams even though Figure 2B presents results from a model presumably built from the raw data that underlie Figure 2A.

Why is day 210 significant? As with other vaccine trials, the FDA allowed the pharmas to vaccinate the placebo group, thus dismantling the trial. In J&J's trial, this started happening on March 10, 2021, about 170 days after the start of the trial. Any data beyond 170 days are essentially worthless. (Ultimately, almost every placebo participant took the vaccine.)

 

 

An outlier or the norm

Junior 1: You're asking me to cherry pick data and I obviously can't do that.

Senior: Why?

Junior 2: Because we are not going to get a clear picture of whether or not the test works.

Senior: But those were OUTLIERS.

Junior 1: Yes, I am sorry but what exactly is an outlier? ... Because it seems to me that it is just a data point that isn't doing what you want it to do and --

Senior: We consider that an outlier.

Junior 1: But then, there would be outliers in every data run we've generated so far.

Senior: Right, and WE DELETE OUTLIERS.

Junior 1: OK, I'm, like, not trying to make a big thing of this. I just think ...

Senior: You sure? ... It seems like you are. [pause]

Senior: We delete outliers. That's what we do.