Big Data, Plainly Spoken (aka Numbers Rule Your World)

The miracle of randomization in AB testing and RCTs

I'm in the midst of preparing my talk this Friday at a University of Denver conference (more info here). The talk will focus on the range of research methods that have been utilized in pandemic-related scientific research, why research methods matter, and how they are related to research quality.

I have some materials that probably will get edited out of the final presentation but I don't want to just throw them away. So I am posting some of them here. Today, I like to use a practical example to illustrate the miracle of randomization, which is an important feature of controlled scientific experiments.

It's hard to imagine running an A/B test or a clinical trial without randomization but this idea is not that old - the British statistician RA Fisher is often credited as having invented it in the 1920s. Randomization means the participants are randomly assigned (via a coin flip) to A or B groups (or test or control groups).

The consequence of randomizing treatment is profound. It leads to "covariate balance". If 50% of the research population is female, we expect 50% of the test group as well as 50% of the control group to be female. Depending on the total sample size, the margins of error around those percentages vary.

The one act of randomization confers balance between the test and control groups on all covariates of interest - not just gender but also race, income groups, comorbidities, employment status, anything that is relevant to your study.

What's more, you get balance on other covariates for free! Any covariate, including things you've never dreamed about.

It is this last point that has enormous practical implications.

Let me explain using a real-life A/B test. Let say Amazon wants to test sending customers an email on Christmas eve, and measure whether this email increases spending in the next week. The data scientist draws up the list of targeted customers, then randomly select half of this list to receive the email, and the other half to be excluded from the mailing.

If everything goes as planned, at the end of the week, the data scientist can pull the data, tabulate the spending on both groups of customers, and any difference in spending can be attributed to the email. Of course, the analyst knows that household income, gender, religion, ethnicity, education, prior spending, etc are all factors that affect holiday spending. Since treatment was randomized, all of those factors are well balanced between test and control groups, and therefore cannot explain any statistically significant change in spending.

So far so good.

In a meeting with the IT team, it was disclosed that a systems error caused all Yahoo.com emails to be dropped from mailing. Let's assume that people still using Yahoo.com emails are bigger spenders on Amazon (reasonable since these people are probably older). Has this error ruined our test?

The short answer is no. That's because randomization not only balances variables we know about, but also balances unknown variables. In this case, if the analyst pulls out the proportion of Yahoo.com email addresses from the test and control groups, it's highly likely that the difference is immaterial. Therefore, we can still interpret any difference in spending as caused by email.

It gets even better!

You might think the above scenario is concocted and unrealistic (though I can assure you it's not.) What makes it a tad unrealistic is the eventual discovery of the error. Most errors are committed and never revealed.

For example, people with Yahoo.com emails may not be dropped by our IT team; they may simply be victims of spam filters. Most email providers out there use one of several spam filters, which are black-box algorithms that change constantly. For any number of reasons, it's possible that Yahoo's spam filter algorithm dumped Amazon emails into the spam folder during the week of our email test, and so people with Yahoo.com emails mostly didn't see the emails.

This situation cannot be predicted by the data scientist, nor will it likely to discovered after the fact. However, our test analysis can proceed as usual because again, the test group will have about the same proportion of Yahoo.com emails as the control group. To the extent that a difference in spending is observed, it isn't due to the spam filter.

We can call this the unintended consequences of randomization. This is really really cool stuff.

***

Not long ago, I wrote about real-world studies of vaccine effectiveness that mimic randomized controlled trials by synthesizing test and control groups. The matching process involves explicitly balancing the two groups on a list of matching variables (age, gender, etc.).

This process does not have the magic of a randomized experiment. In practice, balance is enforced on those matching variables but there is no reason to believe the two groups are also balanced on other known covariates not used for matching. Further, it is even less likely that balance is achieved on unknown covariates.

This is one reason why experiments are considered the gold standard for causal studies.

 

 

The statistics behind the J&J blood-clot decision

Last Friday, the U.S. ended the pause of the Johnson & Johnson vaccine, which may be linked to rare blood clots in women under 50. As with the EMA's stance on the Astrazeneca vaccine, they recommend continuing to administer these vaccines based on a cost-benefit analysis.

In today's post, I summarize the key information in the documents released during Friday's meeting of experts about those blood clots. The media have failed to cover the statistical reasoning behind the decision.

In its presentation, J&J highlighted the number that is most beneficial to its commercial interest - the risk of rare blood clots in the general population regardless of gender and age. The media is also fixated on this statistic, which is the least useful of all the numbers in these documents. Out of 8 million people who have been given the J&J shot since March (remember that this vaccine only got approved at the end of February), they have reported 15 cases of rare blood clots coupled with low platelet counts. This means the risk is 2 cases per million.

Measuring the risk of this side effect is a great example of why no statistic is purely objective. The subjectivity lurks behind how the risk is defined. Risk is a ratio of two numbers: the number of cases, and the relevant population at risk. The 2 cases per million number bakes in two choices: on the denominator, J&J selected the entire population regardless of age and gender; on the numerator, J&J selected the co-occurrence of rare blood clots and low platelet counts. Those are not the choices I expected. 

The denominator should be women only, or younger women only, given that all 15 cases occurred in women under 60 years old. The choice of the denominator is not immaterial! The CDC ultimately used women aged 18-49 as the relevant population, and this change of denominator bumped the risk from 2 up to 7 per million. (Including men roughly doubles the denominator while adding zero cases to the numerator.)

The cutoff age of 50 is also a choice. Two cases were found in women between 50 and 60 years old. By setting the at-risk population as women under 50, those two cases are removed from the numerator. Each additional case increases the risk by roughly 0.5 times if the denominator is not changed appreciably.

The CDC also looked at smaller subgroups. Women in their thirties accounted for the most cases, and the risk in that age group was 12 per million (7 cases among 600K vaccinations). Meanwhile, 1.4 million women aged 50-64 took the J&J vaccine, with two reported cases of blood clots and low platelets, leading to a risk of < 2 per million. Scientists can choose the most restrictive analysis by focusing only on women in their thirties, or the most expansive analysis, on all women. The CDC's choice of 50 is reasonable while not the only option.

The numerator is also unusual, and different from what the EMA used when examining the Astrazeneca vaccine. J&J and the CDC decided to define the side effect as rare blood clots coupled with low platelet count. This is a more restrictive case definition, which reduces the number of cases under study. Interestingly, this definition neatly draws a line between the adenovirus platform and the mRNA platform (Pfizer & Moderna), as so far, the mRNA vaccines have seen some cases of blood clots but zero cases of blood clots plus low platelets.

***

The next issue is how bad is that risk. For this, we have to compare to other risks. 

One obvious comparison is the baseline rate of blood clots and low platelet count. This is where the choice of numerator presents a difficulty. The baseline risk is essentially zero: that combination of conditions almost never happens. One of the CDC documents compared the observed risk to the risk of blood clots in women 20-50 years, which is invalid. The risk of blood clots plus low platelets should be quite a bit lower than the risk of blood clots (with or without low platelets). In a different CDC document, the risk of blood clots and low platelets is estimated to be under 1 case per million. In the J&J document, they claim the baseline risk is 0.1 per million. Neither of these numbers can be directly compared to the 7 per million observed risk because the baseline risk is per million Americans, not per million American females aged 18-50.

Nevertheless, it is abundantly clear that the risk of getting blood clots and low platelet count among women under 50 who took the J&J shot is much higher than the baseline risk. The risk of dying from this side effect is also much higher than the baseline mortality risk. (About 25-30% of the known cases have already died.) On an absolute scale, the condition is very rare. On a relative scale, it is concerning.

All 15 cases were reported between 6 and 15 days after the vaccination. Notably, this is the period during which scientists insist the vaccine has zero benefits. 

***

Another relevant comparison is the risk of death from Covid-19. This is a key to the cost-benefit analysis. How should we compute this risk? We should focus on the period from March and April, the two months in which the J&J vaccine was administered. 

This CDC slide indicates that among females aged 18-50, the Covid-19 death rate was around 0.3 per 100,000, or 3 per million. This means the chance of dying from Covid-19 is about the same as from dying from blood clots+low platelets after taking the J&J vaccine.

Yes, that's right. It's also confirmed in the CDC simulation model. 

So, what does the CDC mean when it says the benefits outweigh the costs? The simulation model finds that slowing down the vaccinations causes a lot of excess Covid-19 deaths among men, and among females over 50, and therefore, across all age groups and genders, stopping J&J causes more harm than good.

***
Two other small details caught my attention. There is another fatal case of blood clots and low platelets that was excluded - seemingly because the situation is too complex. There are also other cases being investigated including cases affecting men. 

When the investigation first started, there were 6 reported cases. About 10 days later, the number of cases more than doubled to 15. The vaccine didn't become more risky suddenly. This phenomenon is caused by self-reporting of cases. Such self-reporting has always been a weakness of how we monitor side effects. The risk may be over-estimated because of under-reporting of mild cases.

 

 

 

 

 

 

 

 

U.S. regulators asleep at the wheel

My friend Mark Palko has been on Tesla's case for a while. He just commented on the latest fatal accident involving a Tesla vehicle. Amusingly, at this point in time, we are supposed to believe that a ghost crashed the car killing two passengers - as we're being told neither a human driver nor Tesla's software was directing the car.

Mark rightfully laid the blame on regulators. It's pretty clear that U.S. regulators put business profits above citizens' interests. We learned how Boeing was allowed to inspect themselves. The Europeans, not U.S. regulators, are reigning in the likes of Google and Facebook. Tesla's case is particularly telling.

Observation #1

U.S. regulators are allowing Tesla to lie about its "self-driving" technology. The "Auto-pilot" software does not auto-pilot - in fact, drivers are told to have their hands on the wheel at all times. The "full self-driving" mode is not what researchers in the field mean by full self-driving. Tesla customers are used as test subjects for an immature technology. The regulators should have stepped in to stop the false advertising but they have been absent.

Observation #2

After an accident happens, it appears that regulators rely on Tesla to report on itself. It seems that only Tesla has data on whether Auto-pilot or other features were turned on or off, whether the driver's hands were on the wheels, etc. Imagine the Chauvin trial, and the only video for the incident came from the police's bodycams. How do we know the footage hasn't been edited, and that the best perspectives were submitted?

Observation #3

The entire conceit of requiring the drivers to have their hands on the wheel at all times while using "Auto-pilot" not only makes a mockery of the term "auto-pilot" but also creates a liability-free zone for Tesla. If the driver's hands were determined not to be on wheel (and the only source of this information is Tesla), any accident is blamed on the driver. In past accidents, they even faulted drivers for momentarily not touching the wheel. If the driver had hands on the wheel, the accident is also not caused by Tesla because this accident could not have been prevented.

Observation #4

Some Tesla fans argue that everyone should know "Auto-pilot" is false advertising, and so any dead or injured Tesla customers have themselves to blame since they chose to use the feature. Using this logic, anyone who died from opioids deserves to die since they chose to ingest.

 

 

 

 

The cheerleaders of herd immunity

In epidemiology, herd immunity is the end of a pandemic. But epidemiological models do not provide a magic number of vaccinations beyond which the pandemic is certain to cease.

The messaging around herd immunity has gotten very sloppy lately.

On the one side, we have the CDC and epidemiologists screaming about vaccine skepticism, exhorting Americans to get vaccinated in search of herd immunity. On the other side, we have pharmas (Moderna and Pfizer) asserting that every vaccinated person will need booster shots, perhaps every year. If that is true, Covid-19 has become an endemic disease that does not go away: there is no herd immunity.

You either believe Covid-19 is not endemic, and will end through herd immunity, or you believe it is endemic, requiring seasonal vaccination. Not both.

***

In the anti-lockdown communities, a calculation has been circulating around that claims to track our progress toward herd immunity. This calculation takes the number of reported cases and the number of vaccinations, add them together and call that progress toward herd immunity.

Such an analysis has serious holes, and you should know about them.

First, it assumes that the second shot is useless because anyone who has at least one shot is counted toward herd immunity. If the second shot is required, plus the infamous 14-day waiting period after the second shot, then the calculation is much too optimistic.

Second, it assumes the level of vaccination required to achieve herd immunity does not depend on the efficacy of the vaccine. On the contrary, the lower the vaccine's efficacy, the more people must take it for the country to attain herd immunity.

Third, it assumes we know what the target level of immunization is. Herd immunity is an abstract construct dropping out of a mathematical model with lots of assumptions (e.g. R0 which is not directly measurable). You don't stop vaccinating people when some level is reached; you keep vaccinating more people until the cases disappear completely. At that time, the remaining people may not need to be immunized because the virus would not find enough hosts to keep spreading.

Fourth, it assumes that only uninfected people are getting vaccinated. The spreadsheet just adds up the two columns and double counts anyone who was infected in the past and received a shot.

Fifth, it assumes every reported case has ever-lasting immunity without getting vaccinated. Most infected people have some degree of immunity but we do not know how long natural immunity lasts.

Sixth, it assumes every positive test result is a true positive. People with false-positive results were not infected, and do not have immunity.

Seventh, it assumes people who got sick but recovered without getting tested (e.g. mild or asymptomatic cases) cannot have antibodies. Some of these people probably do have immunity. As pointed out by a reader, the calculation does deal with unreported cases - by inflating the case counts by a factor of 4. This is an assumption that only 25% of the infections are counted as cases. I have no idea how this factor can be estimated since the U.S. does not have a random testing program in place. Such an assumption makees assumption #4 above much worse. Many people are counted once as unreported cases and then again as vaccinated persons.

***

No matter how we slice it, at this time, vaccinations are a fraction of the number of confirmed cases in the U.S. So, much of the progress toward herd immunity has been achieved through natural infections. As I pointed out a year ago, the path to herd immunity is paved with body bags. If we believe the calculation, then the U.S. has sacrificed over half a million of our fellow citizens in the name of herd immunity. Is this the lesson of the Covid-19 pandemic? In the next pandemic, will we sacrifice another half million, one million, or more of our fellow citizens?

***

Here are a few other aspects of herd immunity that you're not hearing from the media.

Herd immunity is not an inevitable outcome of the mathematical model. It is a possible outcome if certain conditions are met. The vaccines must be effective enough, the proportion of people taking the vaccine must be high enough while the virus must not be too infectious.

Herd immunity if achieved at a level of vaccination close to 100% is practically meaningless.

It's also time to direct your attention back to the old post, in which I argue that vaccination is not about protecting others (i.e. herd immunity) -- it's about protecting yourself. Assume 80% must take the shot to get herd immunity. Those 80% protect themselves because their cells are producing antibodies against the coronavirus. The other 20% are not infected because the mathematical model predicts that they will not be exposed to the virus, but if they do get in contact with the virus, they will get infected because they don't have antibodies.

The messaging on vaccinations is confusing and misguided. Vaccinations first and foremost give one protection. Herd immunity is a by-product, far less important.

 

What to think about vaccines today, including why vaccines protect you against blood clots

Georgette's comment to my other post prompted me to write a new post on current Covid-19 topics.

Vaccines protect you against blood clots

They didn't say it out loud but they imply it. Keep reading...

The FDA's pause on the Johnson & Johnson vaccine is a significant event. The situation parallels the controversy over many European countries halting the distribution of the Astrazeneca vaccine. My take is the same as before. The only unknown is whether the FDA will issue an additional warning about rare blood clots that affect certain subgroups of people. They are likely to keep recommending the vaccine to most of us.

The circumstantial evidence related to blood clots is accumulating. Both J&J and Astrazeneca vaccines use the adenovirus technology. These blood clots are happening mostly to young women, and are paired with other conditions (low platelets), which makes natural cause being rather unlikely. The diagnoses and demographics of the affected people are similar across both vaccines, and, though rare, the same condition has popped up in numerous countries.

The cost-benefit analysis for the young female demographic is completely different from the average person. The benefit of the vaccine is much lower, given the tiny mortality rate from Covid-19 (not counting the benefit accrued to others). Meanwhile, the cost of serious illness and death from rare blood clots is relatively higher because the baseline risk of this condition is minuscule for that age group. A preliminary estimate by the European Medicines Agency (EMA) found that for the younger population, the reported cases of those rare conditions were 5 - 12 times higher than expected.

Doctors are justifiably alarmed, given the "do no harm" principle. Side effects have always been the challenge of developing vaccines. Injecting otherwise healthy people with a substance and causing harm is not accepted, except in a pandemic. The risk to most people is still low but it doesn't then follow that the risk is low to all.

The statistic of the moment says the overall incidence of blood clots is far, far lower than expected among those who were vaccinated. Therefore, they say, the vaccine does not cause blood clots. The gaping hole in this argument is that the data present strong evidence that these vaccines protect people against blood clots! If the vaccines have nothing to do with blood clots, then we expect similar incidence among the vaccinated as among the unvaccinated, and both rates being similar to the expected value. Instead, they tell us blood clots are far less common than expected among those who got the Astrazeneca shots.

I support the FDA's pause decision. It builds confidence that the agency is keeping an eye on safety as there are plenty of examples of medicines that have side effects that are not found during a clinical trial. I hope they look beyond case counts, and review other information as well. Recall my post on the law of small numbers. 

Fully vaccinated people does not have a 5% chance of catching Covid19

Your chance of catching Covid-19 after getting fully vaccinated is not 5%. It cannot be that high. If you review the data from all the vaccine trials, the typical case rate on the placebo arms is 2-3 percent. That means, of those participants who were given placebo shots during the trials, 97-98 out of 100 did not get symptomatic, confirmed Covid-19. If the chance of getting infected after 2 vaccine shots was 5%, then the vaccine would have negative efficacy (worse than placebo)!

The source of this confusion reflects a misunderstanding of a vaccine trial. Vaccine trials are different from the usual clinical trials of drug treatments. In a cancer drug trial, every participant has cancer. In a vaccine trial, we don't know who has been exposed to the coronavirus. In a short trial, with only half the participants reaching 8 weeks of observation, most of the uninfected probably have not been exposed yet.

Giving a probability of getting infected without stating a time window is also useless. If the 5% number came from the trial results, then the right way to describe it is 5% within two months (which, if you think about it, is disturbingly high).

Vaccines and the cargo cult

There is a cargo cult developing around vaccine science. Vaccines have become the single and only explanation for positive trends while not enough vaccines is the single and only explanation for negative trends.

For example, the U.K. and Israeli governments credit their fast vaccine rollouts as the reason for plunging cases. To do proper science, we must take into account all other factors, and in particular, the impact of national lockdowns - which coincided with the vaccination campaign - and the reversal of the holiday spike.

The vaccines-only narrative lends support to lockdown skepticism, mask skepticism, and social-distancing skepticism. Those brands of skepticism also weaken our national fight against the coronavirus.

All vaccines are the same

Vaccines are not all the same. To believe that a VE 60% vaccine is the same as a VE 90% vaccine is to believe that the scale for measuring VE is severely flawed. To believe that vaccines using different technologies and formulations are all the same is to believe that formulations and technologies are window-dressing. Even if all vaccines were identical, people still have preferences: some go for a black car, others a red car - even when it's the same car.

 

 

In science, truth matters

When I turn on the media and listen to the media hyperventilating about Covid, it's no different from turning on CNBC, listening to Jim Cramer etc. lecturing us on why the stock market went up or down yesterday. If cases went down, it's because of the vaccine. If cases went up, it's the variants. We can write down all the usual suspects: vaccine, lockdowns, variants, holidays, travel, people behaving badly, masks, schools, treatments, parties, dining outdoors, dining indoors, etc. Depending on what happened yesterday, the host picks one from the box. Medical science reporting has descended to the level of stock price watching.

So, I was immediately attracted to Andrew Gelman's recent account of science as "a harbor clotted with sunken vessels," in which he lamented the current state of scientific publications that elevates headline-grabbers, failing to self-correct when such studies subsequently prove to be bogus.

The problem is much broader, and it's about to hit medical research in a huge way, once the pandemic is in the rear-view mirror. The emergency situation has given cover to an ocean of pre-prints and warp-sped refereeing, in which headline-grabbers have floated to the top, without a clear path for future researchers to navigate through this sea of debris. Many - if not most - of these studies will eventually be debunked.

***

In his post, Gelman cited the example of research on the effect of neuropeptide oxytocin (OT) on trust between humans. A seminal paper published in 2005 claimed to have found an association between the two. The original experiment failed to replicate, and by now, it should be clear that the association has not been proven. And yet, Prof. Gelman found that (a) the citations of the original paper are still multiple times larger than those of the replication studies; and (b) the original paper continues to get new citations. He just did a casual Google search so we don't know if those who cited the original research also cited the refutations - I doubt it.

If science advances by cleansing bad ideas as better ones emerge, this situation is not putting it in a good light. This situation is due to a host of reasons. One can be confirmation bias - the researchers are looking for anything that supports an assumption of theirs, so they can move on with the rest of their study. Another reason is professional misconduct, or socialized behavior of a subculture.

For me, the problem is also rooted in the practice of scientific publication. It's too damn hard to find those follow-up studies! The following is a diagram of a scientific paper which links to past papers through the device of "references".

By definition, a reference is a link to the past. So a replication study, or for that matter, any kind of follow-on work, will not be found in the references, nor in the original paper. In this system, it's a chore to discover future papers that reference a given paper. Modern search engines attempt to solve this problem by the device of "citations". From the perspective of the cited paper, a citation is a link to a future paper that lists it in the references.

Unfortunately, the citation device is extremely noisy. Most of the papers that cite the original paper do not evaluate the merits of the paper. Citations are tracked as a means of measuring the impact of a given paper - and this has led to gaming. Citations are given to appease referees, or to elevate colleagues, or any number of reasons.

Buried deep inside a long list of citations may be a few replication or follow-up studies. These are future studies that directly address the validity of the original paper.

Adding to the mess, one expects follow-up studies to be published long after the original publication. Citations that are not follow-up studies are both more numerous and also sooner to arrive than follow-up studies.

Researchers may then rely on meta-analyses or survey studies. If they exist, these studies are easy to locate. But they are published far, far, far into the future relative to the original study. They also require the existence of a prior body of follow-up or replication studies.

The end result is that follow-up studies take a lot of effort to find.

Nevertheless, if there is a will, there is a way. Follow-up studies can be marked or registered. And they could be automatically attached by search engines to the original paper, just like citations, but as a separate tag.

Why should we care? Because in science, truth matters.

***

This timing problem affects other fields as well. I published a letter to the editor once about a New York Times Magazine article. But if you read that original article, you're not going to find my letter. There are no links to the future, only to the past.