Big Data, Plainly Spoken (aka Numbers Rule Your World)

Pfizer modeling in Python

Here is code for those who want to play around with the Bayesian analysis used in the Pfizer vaccine trial study (Related blog post here). I got a request for Python code so here it is. The charts I published the other day were made in R.

# Python code for Pfizer vaccine trial analysis

import pandas as pd
import numpy as np
from scipy.stats import beta
import matplotlib.pyplot as plt


# functions

def ve2vsc (ve):
	if isinstance(ve, pd.Series) is False:
		ve = pd.Series(ve)
	return ((1-ve)/(2-ve)).round(3)

def vsc2ve (vsc):
        if isinstance(vsc, pd.Series) is False:
                vsc = pd.Series(vsc)
	return ((1-2*vsc)/(1-vsc)).round(3)

ve = pd.Series(np.linspace(0,1,11))
vsc = ve2vsc(ve)


# line chart showing VSC and VE relationship

plt.plot(ve, vsc, color="darkgray", lw=3)
plt.title("Relationship between Vaccine Efficacy and Vaccine's Share of Cases")
plt.xlabel("ve")
plt.ylabel("vsc")

ax = plt.gca()
ax.spines['right'].set_color("none")
ax.spines['top'].set_color("none")
ax.spines['bottom'].set_position(('data',0))
ax.spines['left'].set_position(('data',0))

xlabels = (ve*100).astype('int').astype('str')+"%"
xlabels[list(range(1,11,2))] = ""
plt.xticks(ve, xlabels) 
plt.yticks(pd.Series(np.linspace(0,0.5,6)), (pd.Series(np.linspace(0,50,6))).astype('int').astype("str")+"%")

ax.hlines(ve2vsc(0.5), 0, 0.5, color="brown", ls="dotted")
ax.vlines(0.5, 0, ve2vsc(0.5), color="brown", ls="dotted")
ax.hlines(ve2vsc(0.9), 0, 0.9, color="brown", ls="dotted")
ax.vlines(0.9, 0, ve2vsc(0.9), color="brown", ls="dotted")

plt.text(0.02,ve2vsc(0.5)[0]+0.02,(ve2vsc(0.5)[0]*100).astype("int").astype("str")+"%", color="brown", fontsize=8)
plt.text(0.02,ve2vsc(0.9)[0]+0.02,(ve2vsc(0.9)[0]*100).astype("int").astype("str")+"%", color="brown", fontsize=8)

plt.show()

# plotting the posterior

a1 = 0.700102	        # From protocol
b1 = 1			# From protocol
data_x = 8		# Number of cases in vaccine arm
data_n = 94		# Total number of cases

def a2 (a1, x):
	return a1 + x

def b2 (b1, n, x):
	return b1 + n - x

def vsc_posterior(samp, data_x):
	return pd.Series(beta.pdf(samp, a2(a1,data_x), b2(b1,data_n, data_x)))

vsc_sample = pd.Series(np.linspace(0,0.5,72))

# single posterior curve vs vsc

plt.plot(vsc_sample, vsc_posterior(vsc_sample, 8), lw=3)
plt.title("Predicted Prob. of Vaccine's Share of Cases")
plt.xlabel("VSC")
plt.ylabel("Probability")

ax = plt.gca()
ax.spines['right'].set_color("none")
ax.spines['top'].set_color("none")
ax.spines['bottom'].set_position(('data',0))
ax.spines['left'].set_position(('data',0))

plt.xticks(pd.Series(np.linspace(0,0.5,6)), (pd.Series(np.linspace(0,50,6))).astype('int').astype('str')+"%") 
plt.yticks(pd.Series(range(0,21,4)), [""]*6)

plt.show()

# posterior vs ve

plt.plot(vsc2ve(vsc_sample), vsc_posterior(vsc_sample, 8), lw=3)

xlabels = (pd.Series(range(0,110,10))).astype("str")+"%"
xlabels[list(range(1,11,2))] = ""
xlabels

plt.xticks(pd.Series(np.linspace(0,1.0,11)), xlabels) 
plt.yticks(pd.Series(range(0,21,4)), [""]*6)
plt.title("Predicted Prob. of Vaccine Efficacy")
plt.xlabel("ve")
plt.ylabel("probability")

plt.show()


# three posterior curves vs ve

plt.plot(vsc2ve(vsc_sample), vsc_posterior(vsc_sample, 3), color="blue", lw=1.5, linestyle="dashed", alpha=0.4)
plt.plot(vsc2ve(vsc_sample), vsc_posterior(vsc_sample, 47), color="purple", lw=1.5, linestyle="dashed", alpha=0.4)
plt.plot(vsc2ve(vsc_sample), vsc_posterior(vsc_sample, 8), 
	color="darkgray", lw=3)

ax = plt.gca()
ax.spines['right'].set_color("none")
ax.spines['top'].set_color("none")
ax.spines['bottom'].set_position(('data',0))
ax.spines['left'].set_position(('data',0))

xlabels = (pd.Series(np.linspace(0,100,11))).astype("int").astype("str")+"%"
xlabels[list(range(1,10,2))] = ""
xlabels

plt.xticks(pd.Series(np.linspace(0,1.0,11)), xlabels) 
plt.yticks(pd.Series(range(0,21,4)), [""]*6)
plt.title("Predicted Prob. of Vaccine Efficacy")
plt.xlabel("ve")
plt.ylabel("probability")

plt.text(0.02,vsc_posterior(vsc_sample, 47).max().round(2)+0.2,"47-47", color="purple", fontsize=8)
plt.text(0.02,vsc_posterior(vsc_sample, 47).max().round(2)+1,"No Effect", color="purple", fontsize=8)
plt.text(0.88,vsc_posterior(vsc_sample, 3).max().round(2)-0.2,"3-91", color="blue", fontsize=8)
plt.text(0.88,vsc_posterior(vsc_sample, 3).max().round(2)+0.5,"BEST", color="blue", fontsize=8)
plt.text(0.82,vsc_posterior(vsc_sample, 8).max().round(2)+0.2,"8-86", color="darkgray", fontsize=8)
plt.text(0.82,vsc_posterior(vsc_sample, 8).max().round(2)+1,"DATA", color="darkgray", fontsize=8, weight="bold")

plt.show()


# prior and posterior vs ve

vsc_prior = pd.Series(beta.pdf(vsc_sample, a1, b1))

plt.plot(vsc2ve(vsc_sample), vsc_prior, color="magenta", lw=1.5, ls="dashed", alpha=0.5)
plt.plot(vsc2ve(vsc_sample), vsc_posterior(vsc_sample, 8), 
	color="darkgray", lw=3)

ax = plt.gca()
ax.spines['right'].set_color("none")
ax.spines['top'].set_color("none")
ax.spines['bottom'].set_position(('data',0))
ax.spines['left'].set_position(('data',0))

xlabels = (pd.Series(np.linspace(0,100,11))).astype("int").astype("str")+"%"
xlabels[list(range(1,10,2))] = ""
xlabels

plt.xticks(pd.Series(np.linspace(0,1.0,11)), xlabels) 
plt.yticks(pd.Series(range(0,21,4)), [""]*6)
plt.title("Prior and Posterior Prob. of Vaccine Efficacy")
plt.xlabel("ve")
plt.ylabel("probability")

plt.text(0.15, 2,"prior", color="magenta", fontsize=8)
plt.text(0.76,12,"posterior", color="darkgray", fontsize=8)

plt.show()


# compute probability VE > 30%, 70% and 90% given 8-86 split of cases

beta.cdf(ve2vsc(0.3), a2(a1,8), b2(b1,data_n, 8))[0].round(3)
beta.cdf(ve2vsc(0.7), a2(a1,8), b2(b1,data_n, 8))[0].round(3)
beta.cdf(ve2vsc(0.9), a2(a1,8), b2(b1,data_n, 8))[0].round(3)

Good vaccine news without glorifying faux precision

The vaccine trials are slipping into parody with the way the scientific results are being dangled over a hungry press. It's clear that the news is great but let's not pump irrational exuberance over magical precision.

The most responsible thing to say at this point is that we have multiple highly promising vaccine candidates that have self-reported short-term efficiacy up to about 90 percent; if a vast majority of the world gets vaccinated, the world may successfully defeat the coronavirus.

***

When reading vaccine trial results, bear in mind the following:

• No clinical trial can yield an efficacy estimate as precise as 90% or 95% or 94.5%. When a statistician say 90%, we mean 90% around a margin of error. To have highly precise estimates, we must have tiny margins of error. (See also end note.)
  
  
• The precision of a statistical estimate depends on the sample size. The larger the sample size, the more precise is the estimate. In accelerated vaccine trials, decisions are being made based on only 100-150 cases, so the margins of error are not tiny.
  
  
• The difference between 90% and 95% is meaningless. The difference between 94.5% and 95% is especially pointless; it may win PR points but it suggests lack of seriousness to statisticians.
  
  
• Participants in any clinical trial do not constitute a random selection from the entire population - there are self-selection bias and exclusion criteria. The trial results are extrapolated to the entire population, and that's another reason the efficacy estimates are imprecise.
  
  
• Subgroup analysis is shaky at this stage. When filtered to subgroups, the sample sizes get even smaller, and the error bars are wider around the reported number. For example, Pfizer  stated efficacy for people over 65 was 94%. This number is essentially the same as the average efficacy for all age groups. But the error bar for the over-65 people is much wider than the error bar for the average participant.
  
  
• Press releases may highlight particular effectiveness for some subgroup; remember that this also means it is less effective for some other subgroup. AstraZeneca, for example, disclosed 90% efficacy for the 1.5-dose treatment. SInce the average efficacy was 70%, we know that the efficacy for the 2-dose treatment must be lower than 70%.

All three vaccines - Pfizer, Moderna, AstraZeneca - look promising from the short-term efficacy angle (even 60% isn't a bad result). It's hard to say more until the scientific reports are published. Efficacy or other health metrics are not the only consideration; cost, ease of transportation and storage, the schedules of availability, quantities, etc. - very practical matters - also must be evaluated.

 

***

The AstraZeneca result that just came out contains a mystery. In the design protocol, there isn't a 1.5-dose treatment so where did that come from? What the researchers appear to be saying is that an execution error - affecting a quarter of participants receiving the vaccine shots - resulted in a lower-than-expected first dose so the vaccine arm were split into two subgroups for analysis.

Most reporters have not noticed this abberation, or if they did, they didn't think it is material. This accident is immaterial only if the execution error affected a random subset of participants. In a clinical trial, we use randomization to ensure the two arms differ only by whether participants receive the vaccine or the placebo. If a third arm is added, it too must be randomly assigned. In the AstraZeneca/Oxford trial, we know the vaccine vs. placebo assignment is explicitly randomized but not so the split between 1.5 and 2 doses within the vaccine arm.

Execution errors sometimes have systematic causes, and may not be random. It's always risky to assume random, as I explained in this post.

What will happen is the researchers will give a story explaining the execution error in the scientific report, and argue why they can assume the errors were randomly scattered among the vaccine arm. This will be hard to prove. One common analysis is to show that the demographic and prior health characteristics of the group that received 1.5 doses are statistically identical to those of the group that received 2 doses. This is necessary but not sufficient; nevertheless, it will be considered good enough.

For those of us who run lots of testing, our worst nightmare is to have execution errors. What happens now is you have to trust the people, instead of trusting the (pre-registered) process. You have to trust our explanations about the errors, and the analyses justifying the assumption of random error.

I'm also curious how cutting the sample size by 75% has not affected the statistical significance of the result. That will come out in the scientific report.

That said, the average efficacy of about 70 percent is still a good result. I don't want you to think AstraZeneca's vaccine is bad. In fact, it has several advantages in terms of cost, storage, and so on.

 

P.S. The relationship between uncertainty and precision may not feel intuitive. The following probability curve is reproduced from my prior post. This curve shows the probability of vaccine efficacy ("posterior", in the sense of after looking at the data from the clinical trial).

The peak of this curve shows that the most likely value of vaccine efficacy is about 91% based on 8 cases in the vaccine group among a total of 94 cases.

The curve also shows we believe the vaccine efficacy to have a value between (roughly) 70% and 100%. Since the total probability of all possible values is 1, the total area under the curve between 70% and 100% should be exactly 1.

If you draw a vertical line under the peak of 91%, the area of that line is zero (height x 0 = 0). In other words, the probability of a precise estimate is zero. A statement such as efficacy is 91% has zero probability. But we can make other statements, e.g. efficacy is between 90% and 92%: the probability of this is the area under the curve between the values of 90% and 92%. This has positive probability.

The wider the margin of error, the larger the area under the curve, the higher the probability.

 

More notes on Pfizer's Bayesian model

I kept the previous post about Pfizer's Bayesian analysis of the vaccine trial data to a reasonable length. And then, I found some loose ends.

Bayesian analysis has the benefit of flexibility

Bayesians like to address the question head on. If our question is about vaccine efficacy (VE), we directly model that, resulting in a probability curve for VE. When we have such a curve (which I reprinted below), we have the ability to answer all kinds of questions.

In the last post, I told you that the Pfizer protocol disclosed that the FDA requires above 30% efficacy - not 50% as has been reported. That means we want at least 95% of the probability mass situated to the right of the VE = 30% point on the curve. With the results from that trial, this is almost certainly true.

What if the FDA has set 70% efficacy as the standard? We can use the same probability curve. Now we look for the probabilities to the right of 70%. This is still close to 100%.

 

Bayesian updating

I mentioned but never addressed one issue in the last post - why did the statisticians switch from the more intuitive vaccine efficacy (VE) to vaccine's share of cases (VSC)?

Here's one way to think about Bayesian models. We first set down our "prior" expectation, i.e. the pink line shown in the chart below (also reprinted from the last post):

This "prior" represents our subjective belief - prior to running the vaccine trial - of what the vaccine efficacy might be. The pink line is relatively flat across all values of VE with a slight lift towards 100%, indicating these researchers believe the most likely value of efficacy is close to 100% (!)

We can think of a Bayesian model as averaging a set of simulations. (This should remind you of the election forecasting models, which are also Bayesian.) For one such simulation, we select randomly a number from the pink curve (the "prior"). Given the flatness of the curve, pretty much any value of VE is equally likely, with a slight tip towards numbers close to 100%. Let's say we pick 75%.

In this particular simulation, the VE is now assumed to be 75%. The statistician now asks: if the vaccine's efficacy were 75%, and if I ran the trial to 94 cases, what would be the split of cases between vaccine and placebo?

If we flip a coin 94 times, how many heads will we get? That depends on the probability of the coin showing "heads". I didn't say the coin is fair so this probability does not have to be 50%. Now substitute "head" with "vaccine" and "tail" with "placebo", and you can see that the two questions are mathematically equivalent.

What is the probability of the coin showing "head" (i.e. "vaccine")? Is it 75%, the assumed VE for this scenario? Not really! The VE of 75% means the vaccine's case rate is a quarter of the placebo's. What we need is how many of the 94 cases turn out to come from the vaccine group. This number is the VSC - the proportion of cases coming from the vaccine group.

This is why the Bayesian model works with VSC instead of VE. As I showed in the previous post, for each value of VSC, we can derive the value of VE so the model can be interpreted in terms of VE, which is more intuitive.

Now back to the simulations. Each simulation starts with a different randomly-picked value of the VSC. This value of VSC is then used in our 94 coin tosses, after which we tally up how many "vaccine" sides turn up. So for that simulation, we have a split of cases between vaccine and placebo. We repeat these simulations, and average the results. This leads to the posterior probability curve (the black line) shown in the first chart of this post.

 

P.S. So far, I have ignored one small complication. In all these charts, I limited the VE to between 0% and 100% (from no effect to completely vanquishing the virus). However, VE can be negative! The vaccine could do more harm than good. In the VSC world, the share of cases can only go from 0% to 100%, and there are no other possible values. Note that when VSC exceeds 50%, i.e. the vaccine group accounts for more than half of the infections, VE is actually negative. The negative side of the VE scale is tricky - as in theory, VE can be negative infinity (all cases coming from the vaccine group).

[Added 11/18/2020] Here is the chart of the prior probability lifted from Sebastian Kranz's post. The right half of the probability density corresponds to theta > 0.5, meaning there are more cases in the vaccine group than the placebo group, i.e. VE is negative. Do they really believe that the vaccine has such a high chance (38%) of causing more cases than the placebo?

 

P.P.S. The analysis methodology is standard. As with most of statistics, the devil is in the definitions, the counting, and the exclusions.

 

 

 

 

How Pfizer plans to analyze the vaccine trial data

While we eagerly await the scientific report on the coronavirus vaccine trial from Pfizer/BioNTech, some of us in the blogosphere have checked out the Pfizer protocol in which they have pre-registered the method of analysis. This post is intended for those who are curious how the data from such trials are analyzed.

In a vaccine trial, the key metric is vaccine efficacy (VE). Pfizer previewed the "interim" analysis in a press release in which they claimed their vaccine has a VE of "above 90 percent". (Regardless of the optimism of this announcement, science by press release is an undesirable development. Such pre-announcements with few details prevents experts from offering any critique - the only reasonable response is to accept the headline tentatively, allowing it to take root.) 

VE of 90% means the vaccine cut infections by 90 percent. It does not mean 90 percent of those who are vaccinated will not get sick (link). That's because the coronavirus isn't spreading as quickly as you might presume. Inferring from Pfizer's press release, I think about 1 percent of the unvaccinated participants got infected within 30 days. So cutting infections by 90 percent means the case rate in the vaccinated group is 0.1 percent. Throughout this discussion, I refer to cases instead of infections because it's not clear from the protocol that Pfizer will detect every infection.

Since the participants are equally split between the vaccine and placebo groups, the average case rate is (1+0.1)%/2 = 0.55%. Out of every 10,000 participants, 55 have gotten sick. There were 10 times fewer sick people in the vaccine group than the placebo group. So 5 out of 55 are from the vaccine group, and 50 out of 55 are from the placebo group. The vaccine group accounts for 5/55 = 1/11 = 9 percent of the total cases. I'm going to call this the Vaccine Share of Cases (VSC) metric.

The Pfizer analysis does not work directly with the VE metric. It focuses on the VSC metric. (For those who've read the protocol, VSC is the entity known as theta.) Notice that the higher the VE, the more powerful is the vaccine, the smaller is the VSC. Those two metrics are inversely related. I did the calculation above to show you that any value of VE corresponds to some other value of VSC. Maximizing VE is the same as minimizing VSC. (The analytical reason for using VSC instead of VE will be explained below.)

The following chart shows the inverse relationship between VE and VSC.

The claimed VE of 90% implies that 9% of the cases were traced back to the vaccine group. The supposed minimum standard of 50% efficacy corresponds to cases split 1/3, 2/3 between the vaccine and placebo groups.

***

When the statisticians receive the data from the trial, they can estimate the VSC in the trial population. The Pfizer press release said at the time of the interim analysis, they recorded 94 cases in total. We just computed that 90% VE corresponds to 9% VSC, so 9% of 94 or roughly 8 cases came from the vaccinated group and 86 cases came from the placebo group. If the vaccine is completely useless, the cases would split evenly, 47 cases each.

That analysis is inadequate, though, because it does not take into account sampling error. Imagine re-running history with a different set of trial participants. It's quite possible that the VSC in this trial would be around 9 percent but it is highly unlikely that it would end up at exactly 9 percent.

Instead of concluding that VSC is 9%, as observed in the trial, Pfizer statisticians assign probabilities for the value of VSC in a Bayesian-style analysis. Like this:

In this chart (a probability density curve), the value of VSC runs along the horizontal axis, and the height represents the probability. The peak of this density curve gives the most likely value of VSC, which is 8.4%.

Statisticians focus on the middle 95% of the probability, i.e. the VSC range from 4% to 16%. The FDA scores the trial based on the chance that vaccine's share of cases is below 41%, which, if you glance back at the first chart of the post, corresponds to VE at least 30 percent. (This means that the FDA does not actually require at least 50 percent vaccine efficacy as frequently reported.)

Below is the same probability curve in terms of VE rather than VSC:

This chart is more intuitive. As you might suspect, the shape of the curve is influenced by the specific outcomes of the vaccine trial. The above chart assumes the cases split 8 and 86 between vaccine and placebo groups.

In the next chart, you can see how the probability curve changes if the trial had returned different results.

Some experts suggest that the vaccine cases could be as low as 3. For that trial outcome, the Pfizer analysis yields the blue dashed line shown below. The probability mass shifts to the right (higher VE) and also becomes tighter (lower uncertainty).

I include also the purple dashed line for the scenario in which the vaccine and placebo groups both produced 47 cases. As expected, the probability mass is now concentrated on the opposite end of VE. Even though the measured efficacy in the trial is 0%, the Bayesian model assigns probabilities of VE to be up to 50%.

If you're interested in even more technical details, Sebastian's post is your next stop.

***

As I explained the other day, the Bayesian analysis requires a "prior." The prior expresses the analyst's subjective expectation of the answer without the benefit of the data from the trial. Once the data become available, the prior is modified to yield the probability curves shown above.

The following chart compares the prior and "posterior" density curves.

You can see how far the prior has been pushed by the strong data observed in the Pfizer trial.

The shape of the prior (pink line) is rather curious though. The Pfizer statisticians seem to believe that the most likely VE is 100%, which is shockingly optimistic. They hedge by spreading the rest of the probability rather evenly across all possible values. The data from the vaccine trial pull the peak away from 100% (there were confirmed infections among the vaccinated), while concentrating all probabilities to between 70% and 100%.

Recall that the FDA wants to know the chance that VE is larger than 30%. The black line tells us that the strong data from the trial indicate an almost certainty. What was the researchers' prior expectation? That's asking how much of the probability mass of the pink line lies to the right of 30%. According to my software, it is 57%54%. [Oops, I forgot to switch from VE to VSC.]

In plain English: prior to running the vaccine trial, Pfizer believes their vaccine has a 65 in 10 chance of reaching at least 30% efficacy (the FDA minimum requirement), and based on the strong result in the trial, they now think it is almost certain that the vaccine has at least 30% efficacy.

 

P.S. I tied up a couple of loose ends in this next post.

 

[11/26/2020: In a new post, I put up some Python code that can be used to generate the charts shown on this post.]

 

The outsized role of statistics in the aftermath of the U.S. elections

Several friends have asked me whether statistics will be useful in proving either (a) fraud occurred or (b) fraud did not occur during the recently-concluded U.S. presidential election. So I address this issue today.

On p. 152 of Numbers Rule Your World (link), I inserted a seemingly innocent remark in parentheses. It read: "Unlike in Ontario, neither the Atlantic nor the Western Lottery Corporation has been able to catch any individual cheater".

This quiet comment rings very loudly in the aftermath of the U.S. election, as it shines a light on the outer boundary of statistical evidence.

The remark appeared within a case study about lottery fraud, which I used in the book to examine the nature of statistical evidence. In Ontario, Canada, lottery store owners were winning a disproportionate number of major prizes in the lotteries. A calculation showed the chance of Ontario store owners winning that much by luck alone was one in quindecillion (one followed by 48 zeroes!). A similar calculation for the Western province results in 1 in 23 million chance. Seeing those numbers, any competent statistician concludes that fraud is certain (with a vanishingly small margin of error).

***

Statistics is reasoning about a mass of data; it has little to say about the atoms. In fact, the commonly accepted assumption of "random" regards these atoms as interchangeable. In the Atlantic region between 2001 and 2006, about 1,300 major prizes were handed out. If store owners had equal chance of winning as non store owners, they should have won about four of those prizes. In reality, they won almost 10 times as many. This statistical analysis proves that some store owners cheated. That's as much as the statistical analysis can say.

If you ask the analysts, which owners cheated? They can't say. Name one cheater. They can't say either. Out of the 40 odd store-owner winners, exactly how many cheated? They can't say. Even though the expected number of wins by store owners is four, there is a chance that in any set of 1,300 major prizes, store owners won five or six or three or even zero.

In the case study, the Ontario investigators successfully traced down a victim, and then discovered the way the specific store owner cheated. The other provinces had no such luck. So they were left with the near certainty that some stores cheated but the improbability of catching an individual cheater.

In the ongoing electoral saga, Republicans face two challenges: they have yet to find statistical evidence of fraud, and if that surfaces, the data alone will not point to any individual malfeasance -- at best, the data allow statisticians to compute a probability of fraud.

When I taught a class using Numbers Rule Your World (link), I used to set a question about Chapter 5 that asks students to think about whether the probability calculations empower the authorities to arrest cheaters. I suppose if future students discover this blog post, they deserve to get points.

 

P.S. This is a good place to say something about the Bayesian school of statistics. In the above (classical) analysis, we assume no one was cheating, and ask what is the chance that store-owners win the lop-sided number of major prizes (the observed data). If that probability is low, we conclude that there is fraud. Bayesians turn this question around. Now, we ask directly what is the chance of store-owner fraud. An equivalent question is what is the share of major prizes store-owners should be winning. Bayesians think always in terms of probability. So, the answer comes in the form: there is X percent chance that the share of major prizes won by store-owners is in the range Y1 to Y2.

To compute a Bayesian probability, we need a "prior." This is like answering that question without the benefit of new data. Let's assume store-owners purchase 2 percent of lottery tickets in Ontario. One possible assumption of the prior is that the store-owner share of major prizes is between 0 and 10 percent, with equal probability (technically, this is a uniform distribution representing no specific knowledge of which number is more likely). Another possibility is a normal distribution (bell curve) centered at 2 percent, with standard deviation 0.6 percent. Now, we're claiming knowledge that numbers close to 2 percent (i.e. no fraud) are more likely than other numbers.

If you feel uneasy about the subjectivity of the previous step, you're not alone. That's a major point of contention between the classical and Bayesian viewpoints. Nevertheless, in many problems, the Bayesian solutions are not sensitive to the assumption of the prior, so we don't have to worry.

Now that we have specified the prior probabilities, we do a Bayesian update, which means modifying our prior beliefs using the observed new data. Think of this as a series of simulations. In a specific simulation, we draw a number from the assumed prior distribution; this might mean the store-owner share of major prizes is 5 percent, then with this prior probability, we give out 1,300 prizes, and count the number of prizes that go to store owners. In each simulation, a new prior probability is drawn, resulting in a different number of prizes going to store owners. Finally, we average all these simulations. (If this reminds you of election forecasting models, you're getting it. The models by the Economist and FiveThirtyEight are both Bayesian.) In short, we run a lot of simulations, each result is a split of major prizes between store owners and other players, and we weight these results using the prior probabilities.

I will have more to say about Bayesian models soon because the Pfizer vaccine trial will be analyzed using a Bayesian model.

One last thing: the Bayesian analysis does not resolve the key issue raised in this blog post. We can say there is X chance the the store-owners are winning Y percent of the major prizes. If there is not enough probability around the Y = 2 percent, we suspect fraud. But once again, we don't know how many of the 40 wins were fraudulent, and we don't know who cheated.

Click here to learn more about Numbers Rule Your World, and support my blog by getting a copy.

Watching statistical gravity in action: elections and vaccines

A quick post today to let you know what's on my mind this week - I'm still chewing on these ideas, so if you have opinions, feel free to comment below. Will follow up on some of these in future posts.

Four Quick Takes on the Election Aftermath

1. In 2016, we mocked most election forecasters who said Hillary Clinton would win with 99% probability while agreeing that Nate Silver won the least bad award. Fast forward to 2020. Nate gave Biden a 89% chance of winning while the Economist said 96%. Who's got bragging rights here?
   
   
2. A former colleague pointed out to me that people mistakenly confuse a high chance of winning with a wide margin of victory. In a simulation model, a 96 percent chance of winning just means that Biden was predicted to win the most electoral votes in 96 percent of the simulated scenarios -- the 96 percent number does not contain information about the margin of victory in any scenario.
   
   
3. What Trump voters witnessed in Wisconsin, Michigan, Georgia and Pennsylvania was "statistical gravity."
   
   
4. In a Bayesian forecasting model, we get a posterior probability distribution as the output. This is typically explained as the proportion of scenarios in a simulation that end up with a given electoral-vote share. Tail events are rare and capture unusual scenarios - e.g. James Comey in the last days of 2016. Within this framework, how does one express the view that in the forthcoming election, one expects to see a tail event?

 

Four Quick Takes on the Coronavirus Vaccine

We finally got some news yesterday via Pfizer*, which sent out a press release claiming that its vaccine candidate may have "over 90 percent efficacy". They have not published a formal report, so there isn't much we can go on right now. Given the urgent need for a viable vaccine, this is one step in the right direction.

1. The claim is based on what happened seven days after the second dose so what the statement meant was that people who got the vaccine was 10 times less likely to be counted as a case within seven days of the second dose than those who got the placebo. This is an impressive early finding. Will "statistical gravity" kick in later?
   
   
2. People keep misinterpreting vaccine efficacy. The leader of the current White House coronavirus task force tweeted that the Pfizer vaccine "prevented infection in 90% of its volunteers". As I said before, this is not what it means. The case rate for vaccinated volunteers was about 0.1 percent while that for unvaccinated volunteers was 1 percent (99 percent of placebo takers did not get infected in the first 7 days). The former is 90% smaller than than latter.
   
   
3. I was scrambling to read the Pfizer protocol. I had assumed theirs and Moderna's would be similar since their vaccine candidates belong to the same family. I was surprised to learn that Pfizer was going to measure efficacy based on seven days as I already knew that Moderna doesn't count cases until 14 days after the second dose. So if we had run these two clinical trials side by side, Pfizer's analysis would have been finished before Moderna even started theirs. When both trials report results, we will have a much richer understanding of this type of vaccines.
   
   
4. People are calling for subgroup analysis. That will take time. This interim analysis is based on a total of about 90 infections. If you break that number down to subgroup level, there's not enough data yet to provide a useful anaswer. Be patient. Allow statistical gravity to work.

* Should give credit to BioNTech which is the company in Germany that develops this vaccine. Pfizer is running the vaccine trial.

Don't cancel the pollsters yet

The backlash on the election forecasting industry is coming and it will be fierce. If you read my previous post from 2016 (here), you know I have doubts about the social value of such work so this is a strange post for me to write. But I suspect this backlash will be overdone so I want to bring up an important issue for you to think about before you cancel the pollsters or modelers.

Were the models wrong? Our visceral reaction is yes, a million times yes. But don't jump to this quick conclusion. Let's first dissect the problem. I bet we can't even agree on what "wrong" means.

***

On the simplest level, wrong can mean the predicted outcome does not match the actual outcome. Think of the model as making 51 A or B choices (ignoring the two states that split their electoral votes), and from the actual outcome, we can compute an error rate. That type of analysis was Nate Silver's claim to fame. He got "all 50 states right" a few cycles ago. (Here's Mashable's article from 2012.)

That metric is silly. As we all know, only maybe 10 states have truly competitive races. In the other 40 states, everyone who knows anything about US politics can predict the outcomes perfectly. By including those non-competitive races in the error count, we allow forecasters to pad their statistics, and we effectively rendered the majority of the scoring scale meaningless.

Let me do a back-of-the-envelope analysis. Every forecaster starts with a 40/50 = 80% baseline accuarcy rate. Even though theoretically the scale runs from 0/50 to 50/50 i.e. 0 to 100%, the "effective" scale is from 80% to 100%. This situation should remind you of grade inflation at our universities. They claim a nominal scale of A, B, C, D, F but in reality, the scale is basically A and a little bit of B (and maybe a sprinkle of C, D and F). The GPA scale is nominally 0 to 4 but if almost everyone's GPA is between say 3 and 4, then the effective scale is from 3 to 4. Back to election forecasting, an accuracy rate of 80% indicates no skill whatsoever. Ninety-percent sounds great but it is actually crappy - that means getting only 5 out of the 10 toss-up states correct. In a properly calibrated scale, I'd call that 50% accuracy not 90%.

[Data scientists: this is not merely politics-talk. Think about how you measure accuracy of your predictive models. Think about comparing your model to random versus comparing it to a next-best model.]

***

That isn't the only problem with the accuracy metric. You have heard that the inputs to the model - the polling data - are flawed. Garbage in, garbage out. Maybe the pollsters are the culprits here.

Not so quick. Let me paint a different possibility. The pollsters are excellent; they acquire a representative sample of the voters, and the voters tell it like it is. Notwithstanding, it turns out that the actual result is far from the poll-predicted result.

This can happen without any incompetence or malfeasance. The voters can change thier mind. What they intend to do does not have to match what they actually do on election day.

You can ask me what I intend to do before I step into the polling booth, I can tell you what I have in mind, but I can still change my mind when I vote. In this case, the poll-predicted result ought to differ from the actual result!

No poll or model will be able to capture this flip. Even if you have implanted a chip in my brain, and managed to learn about the flip milliseconds before I do it, it would be too late for anyone to act on this piece of information.

What I'm saying is if the poll does not reflect the actual result, it doesn't always mean the pollster has gotten it wrong. Maybe you've caught what I'm about to say...

If people don't have clear intentions, there is no such thing as the correct answer. The model merely tells us what the mood is at the moment of measurement. Since the "mood" is not a real thing, it's an invention, the concept of accuracy is unattainable.

When a company like Spotify claims it can predict my mood, they face the same problem. They can guess what my mood is and act accordingly but there is no easy way to validate such a prediction. My brain does not have a built-in tracker of my "mood" that can act as the "ground truth". So, it's pointless to discuss the accuracy of a mood predictor.

[In recent years, these types of problems have invaded the data science space. They are not governed by the scientific method. This doesn't mean they are not worth solving. We just can't measure success like we do with traditional statistical or scientific models.]

***

Here's how I live with polls:

Think of polls as measuring the mood of the country (or state) at the time of data collection. Think of the mood as reflecting what the poll respondents are telling the pollsters at the time. Some people may be lying, some may be confused, some may be indecisive, it doesn't matter. Mood is like the weather.

The actual election outcome may differ from the poll predictions. The gap may just represent a shift in mood. It probably also contains some polling errors - I'm not saying polling is perfect; I'm just saying polls may not be as bad as critics allege.

 

 

Hasty election observations that won't stand the test of time

The cable hosts have a lot of time to fill during election week so it's not surprising that they let slip a bunch of observations that won't stand the test of time. Many of these sound plausible but will fall apart on further examination.

More Republicans voted for Trump than in the last election.

I saw this data nugget used by progressive Democrats to argue that the Lincoln Project (a group of prominent Republicans who oppose Trump's reelection) had little to show for the money and effort they put in (link). According to some reports (also see section below), 93 percent of Republicans voted for Trump in 2020, up 3 percent from 90 percent in 2016.

Not so fast. The base of those two percentages are not the same. Between 2016 and 2020, some Republicans digusted with Trump have left the party so the remaining Republicans are even more likely to be Trump supporters. The statistic may be fine but it doesn't prove that the Lincoln Project failed to make a difference.

A supplementary analysis should start with all 2016 registered Republican voters and see what they did in 2020.

The only demographic group in which Trump lost support was white men.

How do we know such a thing so soon after the election (link)? At this stage, all such statements come from results from exit polls. In fact, the data nugget about Republicans voting for Trump also emerged from exit polls.

The problem: exit polls are unreliable in a normal year, and basically worthless in 2020.

Before the pandemic, exit polls are given the veneer of credibility because they are conducted at selected polling sites on election day, and because they feed news anchors with lots of data to support stories about the election results.

The credibility of exit polls depends on (a) a sound sampling design, not easy given the highly skewed distribution of population (b) identifying people who actually voted and (c) getting truthful answers from such people.

In a normal year, (b) is not a problem. The pollsters intercept people as they leave the polling sites so no one doubts that people who answer questions voted. However, (a) and (c) can both be problems.

The rise of mail-in voting during the pandemic has eliminated the exit polls' only advantage. This year, a huge proportion of voters sent in their ballots. The pollsters can only reach these voters through phone, web or mail. But it's a fishing expedition because they don't know who voted. The response rate of phone/web/mail is also much much lower than interception at the polling site. Finally, the responders are self-reporting that they voted, potentially days or weeks after they did. There is no visual confirmation of gender, ethnicity and age over the phone or internet.

The exit polls this year have a mixture of polling site interviews and phone/web/mail. The quality of the data is much better in the former than the latter. But the former subset skews heavily Republican. So it's a grand mess! I wouldn't trust any of the information from exit polls.

 

Progressives i.e. "socialists" are the black sheep of the Democratic party.

It didn't take long for Democratic centrists to take a dump on the progessive wing of their party. The Washington Post has a leaked recording in which a Democratic representative blamed her progressive colleagues for their disappointing results in Congressional elections. You may also recall, according to Hillary Clinton, what happened in 2016 was odious progressives.

As far as I can tell, the evidence for this accusation is Trump's claim that Biden is secretly being controlled by Bernie Sanders.

Since the total of Trump and Biden vote shares exceeds 98 percent this year, and the participation rate exploded, I think it's safe to say most progressives voted for Biden. If that is confirmed later, then centrists in the party should be thanking progressives for gifting them this victory. If these "black sheep" Democrats were indeed the marginal voters who cost Clinton the Presidency in 2016, then they would be the marginal voters who pushed Biden to victory in 2020. They are two sides of the same coin. The centrists can't have it both ways.

 

Trump hurt himself by fomenting fear of mail-in ballots.

This conclusion represents a common fallacy in reasoning with data, a confusion between selection and causation. What is undisputed is that mail-in votes skew Democrats and in-person votes skew Republicans. What is the implication of this statistic?

Each voter has a choice of how to vote. In this election, Biden voters disproportionately select to vote by mail while Trump voters disproportionately select to vote in person. The argument seems to say that if more people have voted in person, there would have been more Trump votes.

Perhaps it is not obvious but this argument has an implicit assumption - that voting in person causes someone to vote for Trump. If a mail-in voter had decided to vote in person, then s/he would have voted for Trump instead of for Biden. If true, Trump would have hurt himself by fomenting fear of mail-in ballots.

But that is absurd. The same argument from the other direction is also absurd. If Trump had not sowed doubts about mail-in voting, more Republicans might have voted by mail but voting by mail would not cause them to suddenly elect Biden.

No, changing how someone votes is unlikely to affect who they vote for.

 

We can't declare a winner in PA, AZ, NV, GA, etc. until all votes are counted, till the recount threshold has been breached, etc.

Those were made-up rules that changed by the day. If they were consistent from the start, then no states would have been called with 5 percent of the votes in, or 50%, etc. These networks were claiming that they couldn't call the last few states because the other candidate could "mathematically" still erase the gap. If this rule was applied to all states, then they sholudn't have called New York or Alabama either since a black swan is mathematically possible until one candidate won at least half of all outstanding votes. 

Which is more likely: that the provisional ballots would be so lop-sided and different from the already-counted 99+% of the ballots, or that the news media are stringing out the horse race for ratings?

 

 

 

The formula you need to figure out if the lead will flip in a toss-up state

For anyone following the nailbiter U.S. Presidential election, we keep wanting to answer the question - how does the next batch of votes affect the margin between Trump and Biden?

Let me give you a quick formula to use. You can do this faster than the TV hosts!

 

By how much will the next batch of votes close the gap?

 

Start with what we expect the Biden and Trump vote shares to be. Let's say 60% to 40%.

Take the difference of the vote shares, which is the voting margin. So 60% minus 40% is 20%.

How much the gap closes is equal to this voting margin multiplied by size of the batch. If 10,000 new votes are released, with Biden winning 60% of them, the gap will close by 20% * 10,000, which is 2,000.

 

For example, in Georgia, as of this writing, Trump leads Biden by 1,800 votes.

99% of votes have come in. So 1% are outstanding. The already-counted votes total about 5 million. So uncounted votes are approximately 50,000.

If all 50,000 get released, and they favor Biden 55% to Trump 45%, the margin is 10%. Multiply that by 50,000 means the gap closes by 5,000 votes. This would make Biden the winner.

[On the other hand, if the vote shares are reversed, of course, Trump will win Georgia. We can still do the same calculation. Instead of closing the gap by 5,000 votes, the gap widens by 5,000 votes.]

***

We can also turn the question around. What is the voting margin required for Biden to overcome the 1,800 margin in Georgia? Or, what is the voting margin required for Trump to overcome the 45,000 margin in Arizona?

What voting shares will be required in the remaining votes to flip the current result?

 

First, divide the current gap by the remaining votes. In Arizona, this is 45,000 divided by 340K, which is 13%.

Now, divide that margin by two and add it to 50%. This gives us 56.5%. So if Trump wins those remaining votes by at least 56.5% to 43.5%, he would overcome the gap of 45,000 votes.

***

A couple of issues to keep in mind when interpreting these numbers.

Whether the required level of performance is possible depends on which parts of the state the remaining votes come from, whether they are mail-in ballots, etc.

If the remaining votes are released in many batches, then remember that any batch that underperforms must be compensated by a future batch that overperforms, and vice versa.

Let's continue the above example and say Arizona releases 50,000 votes in which Trump wins 55% to 45%, which is slightly lower than required. So he would need to do better in the next batches but by how much?

The new remaining vote count is 340K - 50K = 290K. The gap widens by the voting margin x the size of the batch, i.e. 10%*50,000 = 5,000. So, the new gap is 50,000 votes. Divide 50,000 by 290K gives us 17%. Divide by 2 and add to 50% gives us 58.5%. So In the remaining votes, Trump needs to win at least 58.5% to get even.

 

 

How to act like a data scientist 13: no love for independence

[I know, some of you are hungry for coronavirus content. Don't worry, keep reading. I'll get there in this post :) ]

As the U.S. election workers continue to count votes, the scenario predicted by election watchers has emerged: Election Day votes favor the President while mail-in ballots lean Biden. Mail-in votes arrive mostly before Election Day but the Election Day votes were counted first.

Despite predicting this scenario, most pundits have not prepared the dataviz that is necessary to follow the situation. We need the following line chart, which I found at New York Times. (Maps do not have a time dimension!)

(P.S. It's better to cut out the left side so we can see the right side more clearly.)

The order of vote counting should not matter in the final result. The winner of an election is determined by the total vote shares, regardless of the mode of voting or its timing. So once we reach the 100% point in the above chart, the path getting there has no relevance.

However, the order of vote counting matters a great deal in partial results. If mail-in votes lean Democrats while in-person votes favor Republicans, and if in-person votes are counted first (such as in Michigan, shown above), then partial results will show a big lead followed by a tightening. Depending on who the ultimate winner is, there may be a flip to the other party.

Data scientists in particular should know this and never draw conclusions based on partial results.

Nevertheless, in real life, there is always pressure to do a sneak peek. People want us to do interim analyses, to see which way the wind is blowing. They assure us they will remain patient. When we deliver the interim analyses, people want us to project the full result. Such projection is dangerous because it usually requires an assumption of serial independence.

We are implicitly assuming that the first m units tabulated are a random subsample of the entire sample of size N. Applied to vote counting, we are assuming (wrongly as the media keep reminding us) that the sequence of votes is random.

This election has just proved why we should not assume random sequencing within a sample. This is not how it works in real life when a sample accumulates over time.

***

Anyone running online AB tests has encountered the problem of serial dependence. The sample is assembled over time as visitors show up at the website or the app. If, during part of the test, the company drives traffic through advertising, then in that window of time, there is an influx of visitors, and if the advertising campaign is targeted (as is the norm), the incremental visitors are not like the average visitor. Any interim results for that period can be generalized to the types of visitors in the subsample but not to the entire universe of visitors.

The same problem is plaguing the coronavirus vaccine trials. Recall that the vaccine developers announced midway through the trials that they stepped up recruitment of minorities and high-risk people. So the interim analyses will be based on subsamples that skew whites and lower-risk people. This is potentially dangerous because once the first vaccine is approved, there will be pressure to convert placebo to treatment. We might never get a read on those minorities and high-risk people! The partial result is not useless - it just does not generalize to the entire population.

[I asked about this at a recent webinar by an NIH statistician working on these trials, and at first, he said the non-random sequence of enrollment should not matter but then he realized it does matter in any interim analysis.]

***

Thankfully, serial dependence is easy to diagnose once the full sample is available. We compare the characteristics of the subsample with the full sample. In the vote count scenario, we will find that the proportion of mail-in votes shifts over time, which also means there is a difference in party affiliation and various demographics.

What's the lesson for data scientists? Always look out for implicit assumptions in our analysis methods. In thinking about analysis of partial samples, we tend to focus energy on how statistical significane is affected by the sample size while assuming implicitly that the subsample is representative of the full sample in its contents. That is frequently correct in textbook examples but usually wrong in real-life situations.

***

One last thing: dependence comes in many stripes. This post addresses serial dependence. The post from last week concerns dependence between variables.