Big Data, Plainly Spoken (aka Numbers Rule Your World)

Nothing is perfect especially in statistics

In a previous post, I explain the miracle of randomization for causal inference using a simple business example. I left out some side issues - bad things that could still ruin statistical experiments. These are real issues but edge cases.

As a recap, we pretend that Amazon is running an email test on Christmas eve, hoping that an extra email will lead to incremental holiday sales. One half of the targeted customers receive the email while the other half doesn't.

What randomization gives us is covariate balance, which supports an all-else-equal analysis. Accidents like Yahoo's spam filter unexpectedly suppressing our emails (see previous post) affect both arms of the experiment equally; in this sense, the two arms are still comparable despite the accident, and even if we are unware of its occurrence.

One issue that can arise in that scenario is if a majority of our business comes from customers with Yahoo email addresses. The spam filter did not destroy the covariate balance, and the difference between the test and holdout arms is still unbiased. However, the totality of the test and holdout arms no longer adequately represents our research population.

This situation is directly analogous to the problem I've been pointing out regarding the real-world studies of vaccine effectiveness (see this post). Recall that the matching strategy artificially creates a control group that is balanced with a test group (by construction). However, the totality of both groups no longer represents the entire population: that's because we inevitably fail to find appropriate matches for some people - if the observational dataset is sufficiently biased.

Returning to our A/B testing example, we will still have a good read on customers with non-Yahoo email addresses. We just don't know what the effect of the promotional email has on people with Yahoo email addresses. In the Clalit study, for example, since they excluded everyone living in nursing homes or confined to their homes, and they only matched 15% of the oldest age group, the published vaccine effectiveness doesn't tell us anything about the vaccine's effect on those subgroups.

***

A second problem pertains to the nature of the "accident". We are in serious trouble if the accident preferentially impacts one of the two test groups.

Let's say Amazon plans to track the performance of the promotional emails by routing customers to a special landing page, which eventually leads to the usual checkout process. Now, it turns out that the server that delivers the new landing page suffers intermittent outage during the week of the email promotion. Since this landing page is only seen by the test group (receiving the emails), this accident affects one arm but not the other in the A/B test.

Randomization cannot save us from this type of operational hiccup.

***

Bonus tip: think about how a testing analyst might discover the operational issue. It will never surface if all you're doing is looking at a dashboard that shows you the horserace between the two arms.

It goes down to having the courage to anticipate even unlikely mishaps, designing an auditing process that can identify unexpected problems, and keeping the faith that this work will one day save a wayward test! Ideally, you want to identify the outage during the early phases of the test and correct it.

Curve watching during the pandemic

The pandemic has revealed the importance of a most exciting area of statistics - causal inference from observational data. We've been inundated with all kinds of reports about the effect of [vaccines, masking, social distancing, school closures, outdoor dinings...] on the pandemic. Most of these "studies" focus on either one factor or one region or one demographic segment. Then, the headlines and PR coming out of these studies apply the specific results to the general public. Vaccines work! Masks don't work! Closing schools doesn't make sense!

The quality of the causal inference is tested when we apply the findings of these studies to other settings. So in today's post, I explore the claim that vaccination campaigns are responsible for the decline in cases seen in many countries.

In order to shut down our biases, I am presenting the graphs first without country labels. In each chart, the time period is from November 2020, before any vaccinations, to April 2021. I used data from OurWorldinData.org. The red line shows the new cases per million while the blue line shows the proportion of the population who have been vaccinated (defined as those who received at least one shot if it's a two-shot vaccine).

If we only look at individual countries, here are the stories one can tell:

Panel A presents a clean picture. Cases have tumbled from a peak in mid January to a level below that of November by end of April. The decline begun soon after initiating the vaccination campaign, which has reached about half the population. So vaccines worked!

In Panel B, this country's highly successful vaccination campaign has covered over 60 percent of the population by end of April. That appeared to be a critical mass, at which point cases dropped below the level of November. One can build a story about herd immunity.

The herd immunity story is shaky though. The 60-percent threshold does not apply to Panel A. Using the same logic, one would argue that the country reached critical mass even at 30% vaccinated. But at 30% vaccinated, the country in Panel B was approaching peak cases!

In Panel C, the sharp drop in cases happened as the proportion vaccinated rose from 20 to 40 percent while cases grew during the first two months of the campaign.

In Panel D, cases plunged within weeks of the start of the vaccination campaign, and reached a low plateau with fewer than 10% of the population vaccinated.

Then, you have Panels E, F and G. If one only analyzes those countries, then one comes to the opposite conclusion. In these places, the cases apparently have not been slowed by vaccinations. Over 60% of the people in Panel E have taken a shot but no sharp decline is evident yet. For Panel F, there have been a spurt in cases in April with the vaccination proportion nears 60%.

The picture is very confusing. That's because the vaccine is not the only factor affecting the rise and fall of cases. Any model will fit some of the data some of the time; a good causal model must work across multiple countries.

Now, I reveal the countries. These are the top 11 countries ranked by the proportion vaccinated by end of April. I excluded countries with low overall Covid-19 incidence. Seven of these countries have experienced a nice drop in cases per million.

***

It's also a good idea to take a peek at the other end of the spectrum. What's going on in countries that have barely begun their vaccination campaigns.

Here are 12 countries with very low vaccination rates but with case incidence in a similar range to the 11 countries shown above.

None of these countries have seen spikes in cases, almost all of them are seeing downward trends in April, many of them exhibit up-and-down cycles over these months, and a few have marked plunges such as Georgia in December and Montenegro since March.

***

Simple one-factor models aren't going to work to explain trends across countries and time. A good causal model should include a baseline trendline, a vaccine factor, plus lockdowns and other mitigation measures.

Another challenge for data analysts: the data post-vaccination will be hard to compare to past data. There will be fewer Covid tests among the fully vaccinated. Should cause of death be assigned differently if the patient is known to have been vaccinated? Will we see more deaths from co-morbidities? These are open questions - when I learn more, I'll report back.

 

 

The miracle of randomization in AB testing and RCTs

I'm in the midst of preparing my talk this Friday at a University of Denver conference (more info here). The talk will focus on the range of research methods that have been utilized in pandemic-related scientific research, why research methods matter, and how they are related to research quality.

I have some materials that probably will get edited out of the final presentation but I don't want to just throw them away. So I am posting some of them here. Today, I like to use a practical example to illustrate the miracle of randomization, which is an important feature of controlled scientific experiments.

It's hard to imagine running an A/B test or a clinical trial without randomization but this idea is not that old - the British statistician RA Fisher is often credited as having invented it in the 1920s. Randomization means the participants are randomly assigned (via a coin flip) to A or B groups (or test or control groups).

The consequence of randomizing treatment is profound. It leads to "covariate balance". If 50% of the research population is female, we expect 50% of the test group as well as 50% of the control group to be female. Depending on the total sample size, the margins of error around those percentages vary.

The one act of randomization confers balance between the test and control groups on all covariates of interest - not just gender but also race, income groups, comorbidities, employment status, anything that is relevant to your study.

What's more, you get balance on other covariates for free! Any covariate, including things you've never dreamed about.

It is this last point that has enormous practical implications.

Let me explain using a real-life A/B test. Let say Amazon wants to test sending customers an email on Christmas eve, and measure whether this email increases spending in the next week. The data scientist draws up the list of targeted customers, then randomly select half of this list to receive the email, and the other half to be excluded from the mailing.

If everything goes as planned, at the end of the week, the data scientist can pull the data, tabulate the spending on both groups of customers, and any difference in spending can be attributed to the email. Of course, the analyst knows that household income, gender, religion, ethnicity, education, prior spending, etc are all factors that affect holiday spending. Since treatment was randomized, all of those factors are well balanced between test and control groups, and therefore cannot explain any statistically significant change in spending.

So far so good.

In a meeting with the IT team, it was disclosed that a systems error caused all Yahoo.com emails to be dropped from mailing. Let's assume that people still using Yahoo.com emails are bigger spenders on Amazon (reasonable since these people are probably older). Has this error ruined our test?

The short answer is no. That's because randomization not only balances variables we know about, but also balances unknown variables. In this case, if the analyst pulls out the proportion of Yahoo.com email addresses from the test and control groups, it's highly likely that the difference is immaterial. Therefore, we can still interpret any difference in spending as caused by email.

It gets even better!

You might think the above scenario is concocted and unrealistic (though I can assure you it's not.) What makes it a tad unrealistic is the eventual discovery of the error. Most errors are committed and never revealed.

For example, people with Yahoo.com emails may not be dropped by our IT team; they may simply be victims of spam filters. Most email providers out there use one of several spam filters, which are black-box algorithms that change constantly. For any number of reasons, it's possible that Yahoo's spam filter algorithm dumped Amazon emails into the spam folder during the week of our email test, and so people with Yahoo.com emails mostly didn't see the emails.

This situation cannot be predicted by the data scientist, nor will it likely to discovered after the fact. However, our test analysis can proceed as usual because again, the test group will have about the same proportion of Yahoo.com emails as the control group. To the extent that a difference in spending is observed, it isn't due to the spam filter.

We can call this the unintended consequences of randomization. This is really really cool stuff.

***

Not long ago, I wrote about real-world studies of vaccine effectiveness that mimic randomized controlled trials by synthesizing test and control groups. The matching process involves explicitly balancing the two groups on a list of matching variables (age, gender, etc.).

This process does not have the magic of a randomized experiment. In practice, balance is enforced on those matching variables but there is no reason to believe the two groups are also balanced on other known covariates not used for matching. Further, it is even less likely that balance is achieved on unknown covariates.

This is one reason why experiments are considered the gold standard for causal studies.

 

 

The statistics behind the J&J blood-clot decision

Last Friday, the U.S. ended the pause of the Johnson & Johnson vaccine, which may be linked to rare blood clots in women under 50. As with the EMA's stance on the Astrazeneca vaccine, they recommend continuing to administer these vaccines based on a cost-benefit analysis.

In today's post, I summarize the key information in the documents released during Friday's meeting of experts about those blood clots. The media have failed to cover the statistical reasoning behind the decision.

In its presentation, J&J highlighted the number that is most beneficial to its commercial interest - the risk of rare blood clots in the general population regardless of gender and age. The media is also fixated on this statistic, which is the least useful of all the numbers in these documents. Out of 8 million people who have been given the J&J shot since March (remember that this vaccine only got approved at the end of February), they have reported 15 cases of rare blood clots coupled with low platelet counts. This means the risk is 2 cases per million.

Measuring the risk of this side effect is a great example of why no statistic is purely objective. The subjectivity lurks behind how the risk is defined. Risk is a ratio of two numbers: the number of cases, and the relevant population at risk. The 2 cases per million number bakes in two choices: on the denominator, J&J selected the entire population regardless of age and gender; on the numerator, J&J selected the co-occurrence of rare blood clots and low platelet counts. Those are not the choices I expected. 

The denominator should be women only, or younger women only, given that all 15 cases occurred in women under 60 years old. The choice of the denominator is not immaterial! The CDC ultimately used women aged 18-49 as the relevant population, and this change of denominator bumped the risk from 2 up to 7 per million. (Including men roughly doubles the denominator while adding zero cases to the numerator.)

The cutoff age of 50 is also a choice. Two cases were found in women between 50 and 60 years old. By setting the at-risk population as women under 50, those two cases are removed from the numerator. Each additional case increases the risk by roughly 0.5 times if the denominator is not changed appreciably.

The CDC also looked at smaller subgroups. Women in their thirties accounted for the most cases, and the risk in that age group was 12 per million (7 cases among 600K vaccinations). Meanwhile, 1.4 million women aged 50-64 took the J&J vaccine, with two reported cases of blood clots and low platelets, leading to a risk of < 2 per million. Scientists can choose the most restrictive analysis by focusing only on women in their thirties, or the most expansive analysis, on all women. The CDC's choice of 50 is reasonable while not the only option.

The numerator is also unusual, and different from what the EMA used when examining the Astrazeneca vaccine. J&J and the CDC decided to define the side effect as rare blood clots coupled with low platelet count. This is a more restrictive case definition, which reduces the number of cases under study. Interestingly, this definition neatly draws a line between the adenovirus platform and the mRNA platform (Pfizer & Moderna), as so far, the mRNA vaccines have seen some cases of blood clots but zero cases of blood clots plus low platelets.

***

The next issue is how bad is that risk. For this, we have to compare to other risks. 

One obvious comparison is the baseline rate of blood clots and low platelet count. This is where the choice of numerator presents a difficulty. The baseline risk is essentially zero: that combination of conditions almost never happens. One of the CDC documents compared the observed risk to the risk of blood clots in women 20-50 years, which is invalid. The risk of blood clots plus low platelets should be quite a bit lower than the risk of blood clots (with or without low platelets). In a different CDC document, the risk of blood clots and low platelets is estimated to be under 1 case per million. In the J&J document, they claim the baseline risk is 0.1 per million. Neither of these numbers can be directly compared to the 7 per million observed risk because the baseline risk is per million Americans, not per million American females aged 18-50.

Nevertheless, it is abundantly clear that the risk of getting blood clots and low platelet count among women under 50 who took the J&J shot is much higher than the baseline risk. The risk of dying from this side effect is also much higher than the baseline mortality risk. (About 25-30% of the known cases have already died.) On an absolute scale, the condition is very rare. On a relative scale, it is concerning.

All 15 cases were reported between 6 and 15 days after the vaccination. Notably, this is the period during which scientists insist the vaccine has zero benefits. 

***

Another relevant comparison is the risk of death from Covid-19. This is a key to the cost-benefit analysis. How should we compute this risk? We should focus on the period from March and April, the two months in which the J&J vaccine was administered. 

This CDC slide indicates that among females aged 18-50, the Covid-19 death rate was around 0.3 per 100,000, or 3 per million. This means the chance of dying from Covid-19 is about the same as from dying from blood clots+low platelets after taking the J&J vaccine.

Yes, that's right. It's also confirmed in the CDC simulation model. 

So, what does the CDC mean when it says the benefits outweigh the costs? The simulation model finds that slowing down the vaccinations causes a lot of excess Covid-19 deaths among men, and among females over 50, and therefore, across all age groups and genders, stopping J&J causes more harm than good.

***
Two other small details caught my attention. There is another fatal case of blood clots and low platelets that was excluded - seemingly because the situation is too complex. There are also other cases being investigated including cases affecting men. 

When the investigation first started, there were 6 reported cases. About 10 days later, the number of cases more than doubled to 15. The vaccine didn't become more risky suddenly. This phenomenon is caused by self-reporting of cases. Such self-reporting has always been a weakness of how we monitor side effects. The risk may be over-estimated because of under-reporting of mild cases.