Big Data, Plainly Spoken (aka Numbers Rule Your World)

Other numbers from the colonoscopy trial

You think I'm done with the colonoscopy trial - I'm not.

Previously, I discussed the core controversy, which is when investigators learned that the trial outcomes are not what they expect - in fact, colonoscopy effectiveness was far below what the medical profession has believed - they decided to cancel the primary endpoint and tout an "adjusted per protocol" analysis, which was designed with knowledge of the disappointing primary result.

Any per-protocol analysis is a partial lie. It takes a well-designed randomized clinical trial and dissolves the bedrock of random treatment assignment, effectively turning the RCT into an observational study. I call it a lie because people who do PP analysis describe their result as coming from an RCT. The authors of the colonoscopy study are proud that this is the first ever randomized controlled trial for this treatment. This is exactly what happened with all the Covid-19 vaccine trials too: they only analyzed people who took two doses (but if someone gets sick after the first dose, they are ineligible to get the second dose). If the analysis is PP, the study should be classified as an observational study even if the original design is an RCT.

In the last post, I did a deep dive into the two analyses that were presented in the NEJM paper to show why the adjusted PP analysis allowed the research team to print an outcome of 30% improvement on diagnosing colon cancer rather than 18% in an ITT analysis. I find the underlying mechanics rather implausible but as with any observational study, it's anybody's guess as to whether the statistical adjustments helped or hurt.

***

There are a few numbers reported in the study that haven't gotten as much attention in the media.

The most important is the unproven impact on all-cause mortality. Exactly 11% of the invited-to-screen group died within 10 years, and the same proportion died in the usual-care group from any cause. In other words, even if colon cancer was the cause of death on fewer death certificates in the screening group, the same number of death certificates would have been issued in both groups. This is probably because colon cancer kills at an advanced age, and people die from other (related or unrelated) causes if they didn't die from colon cancer. Thus, colonoscopy did nothing to change the life expectancy of the participants.

The researchers did report a large impact on colon cancer specific mortality in their "adjusted PP analysis" (50% compared to 10% in ITT) but they were silent on all-cause mortality. The mortality result was merely stated, and no further details were offered on compliers vs noncompliers, age groups, countries of origin, etc., thus no one can judge whether it makes sense.

Also, note that all rates are cumulative over 10 years of follow-up. Actually, that is also a partial lie. It's a "median" of 10 years of follow-up, meaning that only half of the participants need to have reached year 10. From Figure 1, I can see that about 66% have reached year 10, 85% have reached year 8, etc. The important thing to realize is that the further out in the case curve you go, the smaller the sample size that underlies the numbers (i.e. less reliable).

Another important number is the absolute risk difference. Thirty percent sounds like a large improvement as a relative ratio but it isn't because the baseline risk is very low. Cumulatively over a median 10-year follow-up period, the baseline risk was only about 1.22% (the risk of the usual-care group). Thus, 30% improvement on that is about 0.4%. That means out of 1000 people invited to screen, we can expect about 4 fewer diagnoses of colon cancer over roughly 10 years.

Flip that ratio around and we have what is called Number Needed to Treat (NNT). To prevent 1 colon cancer case, we need to screen 250. Using the ITT result of 18%, the NNT is 455.

A glaring gap is the lack of a cost-benefit analysis. A colonoscopy apparently costs $3000 in the U.S. So, out of 1,000 invitations, perhaps 500 do the screening, which cost $1.5 million, in order to lower diagnoses of colon cancer by 4. Thus, the cost of each avoided colon cancer case (not death, just case) is $375,000.

 

With adjustments comes responsibility

Yesterday, I laid out the framework for today's post, in which I compare intent-to-treat (ITT) analysis and per-protocol (PP) analysis of the colonoscopy trial. All the data I cite below are pulled from this paper and its supplement (link).

***

The outcome metric of the clinical trial is the risk of colon cancer (diagnosis) over a 10-year follow-up window (a final analysis will be performed after 15 years). Risk is based on the relative ratio of diagnosis rates between the treatment and usual-care (control) group. In the ITT analysis, the treatment group consists of everyone who's been randomized to receive invitations to colonoscopy while in the PP analysis, the treatment group consists of the "compliers" only, those who responded to the invitation and underwent colonoscopy. Because compliers self-select to accept screening, they cannot be treated as a random subset of the invited group, and thus, the researchers spun up a regression model used to "remove biases".

In the chart below, I present the results of the ITT analysis from the paper, which is pre-specified for this trial. The trial was run in three countries but Poland (64%) and Norway (31%) contributed 95% of the participants.

 

Immediately, two features of this data come to life. Firstly, the case rate in Poland (0.88%) is much lower than that in Norway (1.35%). The relative risk ratio of 35% is higher than the relative ratio between the treatment and control groups (18%). (In other words, variability between countries is larger than variability between treatment and control.) Secondly, the observed reduction in case rate among Norwegian participants was 24% while that in Poland was 16%. Since Poland comprises almost 61% of the overall population, the reported aggregate effectiveness is just north of Poland's value.

The next chart shows what happens when the researchers switched to an "adjusted per-protocol" analysis. They broke up the invited group into compliers and non-compliers. The paper reported diagnosis metrics for Poland, Norway and the aggregate. (They did not disclose mortality metrics, only reporting the relative ratio, and thus no one can validate that part of the analysis. Nor did they break out Sweden's data.)

In Norway, the paper printed the compliers' case rate to be 0.86%, which is 30% below the average of the invited group. (It is not clear from the paper how much of this shift is due to adjustment and how much is due to filtering out noncompliers. It's not even clear whether this rate is computed from data or estimated from a model.) In any case, the treatment group's case rate was marked down from 1.2% to 0.86%, and correspondingly, the effectiveness metric in Norway leaped from 24% to 45%!

In Poland, a completely different picture emerges. The compliers are said to have a case rate of 0.85%, almost the same as the 0.83% aggregate rate of the invited group. This means that the effectiveness metric in Poland has barely shifted compared to the ITT analysis. It's still about 15%.

From this, we learn that the change in analysis strategy embeds the following key assumption:

(A) Compliers are very different from noncompliers in Norway but not in Poland

Such an assumption might be justifiable; I am in no position to judge. Hold on tight, as there are more assumptions to be unearthed.

Not explicitly stated in the paper is the fact that the case rate for non-compliers must have been altered in the other direction: since in Norway, compliers form 40% of the invited group, and noncompliers 60%, we can back out the case rate of the non-compliers - implied by the adjusted per-protocol analysis - which is 1.77%. This observation reveals two other embedded assumptions:

(B) The diagnosis rate of colon cancer for Norwegians who refused to be screened when invited  (1.77%) is higher than that for people not invited to screening and also not screened (1.57%). This requires explanation since people were randomly selected to receive invitiations

(C) The implied gap between the case rates of the compliers and noncompliers in Norway is a whopping ~1%, which is very wide

 

In Poland, two-thirds of those invited did not comply. Using this information, we can back out the case rate of the noncompliers, and it's 0.82%, actually slightly below the aggregate value of 0.83% for the entire invited group. This leads to further observations:

(D) There is almost no difference in diagnosis rates in Poland between those who complied and those who didn't comply with colonoscopy screening. This is particularly jarring considering (C) above.

(E) Colonoscopy is 3 times more effective in Norway than in Poland. In Norway, it brought the case rate down from 1.57% (usual care) to 0.86% (compliers) which is about the same level as compliers in Poland - despite the control group in Poland coming in much lower at 0.99% than the control in Norway.

***

The researchers decided to do an "adjusted per-protocol analysis" which means they did two things: they filtered noncompliers from the treatment group, and they made adjustments using a regression model that includes variables like country of origin, age, etc.

Since the paper does not disclose the model details, we have to look at the model outputs to infer what happened. What happened is listed above as (A)-(E). These are all changes to the data that ultimately caused the estimated effectiveness to shift from 18% to 31%.

***

What I want to show is that current reporting practices for observational studies in medical journals is wholly inadequate. It should be much easier for readers to comprehend what adjustment models are doing. Much more space should be devoted to justifying the effects of adjustments on the conclusions. Any of the above assumptions can make sense, but there is nothing in the paper to back them up.

The models used to adjust data should be disclosed in full. Tables showing covariate balance between groups should be standard, for every pair of subgroups that feature in any of the metrics included in the publication. For example, age is clearly an important confounder in this study, since colon cancer tends to appear in older people but there is nothing in the paper that shows age distribution between the compliers subset and the usual-care group.

Why no serious researchers conduct "per protocol" analyses

All observational data must be adjusted in order to correct biases. But our science teems with irresponsible statistical adjustments - throwing a bunch of variables into a regression model, and declaring that all known biases have been satisfactorily resolved is all too common.

In the medical literature, publishing one's adjustment model is rarely done. I don't trust any adjusted models unless the researcher discloses the equations, the coefficients and goodness of fit statistics. Even better, the researcher should compare the adjusted to unadjusted results, note how the adjustments affected the outcome, and explain why these adjustments make sense.

Handwaving about the structure of the adjustment model, referencing other papers that supposedly use similar methods, etc. suggest that the researchers lack confidence in their methods.

In this blog post, I demonstrate how to evaluate statistical adjustments, using the recently released, highly controversial results from the colonoscopy clinical trial. (See my related writeup here.) It took a lot of work to put this together, because the disclosure of details in the paper is scarce (this is the norm rather than the exception, unfortunately.)

***

The colonoscopy trial was the first randomized clinical trial (RCT) conducted to measure the effect of colonoscopy screening on risks of diagnoses or deaths from colon cancer. The trial enrolled two groups of participants: those invited to screening, and the "usual care" group which means no screening (as is the practice in Poland, Norway and Sweden, where the trial took place).

For this post, I focus only on risk of colon cancer diagnoses (i.e. cases). That's because the disclosure on mortality outcomes is even worse, making it impossible to judge the adjustment procedure. The trial specifies an intent-to-screen analysis (more generally known as intent-to-treat, ITT), The ITT analysis shows that 0.93% of the invited group were diagnosed during the 10-year follow-up window, compared to 1.13% of the usual-care group. This result is described as a 18% reduction in risk of colon cancer, even though the difference in diagnosis rate is only 0.2%, that is to say, out of 1,000 people invited to colonoscopy, only two fewer cases will be detected relative to usual care.

As discussed in the previous blog, the medical profession was none too happy with this outcome because doctors, esp. American ones, have convinced themselves that colonoscopy could reduce both cases and deaths by 70% or more. Neither did the authors of the paper disclosing the trial results - in the section of the paper in which they discussed the findings, they did not mention the primary endpoint, but only talked about an "adjusted per-protocol" analysis, which moved the reported result from 18% reduction in risk to 31%. Still far from the 70% previously believed, but deemed more palatable.

The media then mis-reported this analysis as "per-protocol" analysis, dropping the all-important word "adjusted". By the end of this post, you'll understand why I say this.

***

Intent-to-treat vs per-protocol is a long-running debate but any serious researcher goes with the former, which is exactly what the designers of the colonoscopy trial did. Let's look at this debate using a marketing example.

Let's say Target is running a Black Friday promotion in which online customers get an extra 30% discount if they type the code SAVE30 when checking out. In order to measure the effectiveness of this promotion, Target data scientists design an A/B test in which shoppers are randomly divided into the exposed group and the non-exposed group upon visiting Target's website. They intend to analyze the results of this experiment by looking at metrics such as proportion of visitors who made a purchase, and average purchase amount, comparing the two groups. This analysis is known as an intent-to-treat (ITT) analysis.

Ouch! The ITT analysis showed that the additional sales generated by exposure to the SAVE30 promotion failed to cover the cost of giving out these extra discounts. Inevitably, there will be at least one business manager who complains that the wrong analysis was done. They argue that we should not include people who did not notice the promotion. Being exposed to the promotion just means Target shows notifications while the customers browse around but some fraction of customers did not pay attention. So, what do they think is the right analysis? We should only analyze those people for whom we know for sure they noticed the promotion - in other words, we only analyze those people who entered the promotion code on checkout (or other variants, such as only those who clicked on a promotion banner). This style of analysis is "per protocol".

Why would all serious researchers shy away from per-protocol analyses? In this example, everyone in the exposed group who subsequently entered the promotion code on checkout is someone who is surely, or almost surely, making a purchase. In the non-exposed group, there is no such thing as entering the promotion code, thus we effectively narrowed the exposed group to almost sure shoppers against the unexposed group that contains the average online shopper. Which group will show better metrics? Will the better results on the exposed group be due to the promotion or inherent differences in the group composition?

***

Back to the colonoscopy trial. In the per-protocol analysis, the data analyst threw out roughly half of the invited-to-screen group because they did not "comply". The reasoning is that they could not have benefited from the screening (read: SAVE30) and thus, they drag down the efficacy. But the "compliers" typically have different characteristics than the "non-compliers", and so per-protocol destroys the randomization aspect of the "randomized" controlled trial. In other words, the complier subset is different from the usual-care group not just in being screened but in unknowable dimensions. We've turned the RCT into an observational study.

People who advocate PP analysis say not to worry - they will throw a bunch of demographic variables into a regression model, and all biases will disappear. I find this quite fascinating. If such adjustment strategies indeed can cure all biases, then why would the community embrace RCTs as the gold standard? It would appear that observational data blended in a mixer known as regression modeling is equally as good as a randomized experiment!

In fact, the colonoscopy controversy clearly shows the opposite. Prior observational studies have consistently shown extremely positive results of 70% or so, and the first time a research team conducts an RCT, the outcome is below 20%. If anything, this proves that carefully analyzed observational studies cannot replace the gold-standard RCT.

***

Even if all biases are cured, and we are able to believe the PP analysis, the real-world results are guaranteed to disappoint.

Let's go back to the Target example. Let's say the PP analysis showed that customers who used the SAVE30 promotion were 80% more likely to complete the transaction than the customers who were not shown the promotion. So now, the head of marketing tells everyone if all shoppers are shown the promotion, then the experiment proves that they should see an 80% jump in number of transactions.

Of course, reality bites. The problem is that the 80% incremental sales apply only to the subset who used the promotion code. If only 10% of those exposed used the code during the A/B test, then only about 10% would contribute an 80% jump in sales. The other 90% are exposed but would not be using the code.

If half the trial population elect not to take the colonoscopy screening during the trial, it is likely the case that half the population would make this decision if colonoscopy was rolled out to all. The 30% reduction in colon cancer risk would still only apply to the subset choosing to be screened, and would not make a dent on those who chose to forego it.

***

It looks like I have to defer showing you my analysis till the next post. The plan is to compare the ITT analysis and the "adjusted PP analysis", pointing out how the adjustments affected the outcomes. Come back tomorrow.

(By the way, all Covid-19 vaccine trials were per-protocol analyses as well, even though such analyses are problematic.)

 

 

 

All evidence are not created equal and some evidence are more equal than others

A casualty of the Covid-19 emergency is scientific rigor, and in particular, certain long-established statistical principles in evaluating evidence, and specifically, that the weight of evidence depends on the method of research, and how data are collected and analysed.

The media, and the scientists they employ as talking heads, rarely identify the nature of the evidence behind health policies. On air, they make little or no distinction between results obtained from randomized clinical trials (RCTs), observational studies, surveys, lab experiments, animal experiments, etc. Like a drumbeat, they promote the impression that all evidence are created equal, despite their disparate origins.

***

While such an attitude towards scientific evidence is not new, it became normalized during the pandemic. A great illustration is found in this recent article reporting the results of a clinical trial on colonoscopy (screening) for colon cancer (link).

In the U.S., the standard of care calls for all men above 45 years old to get a colonoscopy every 10 years. Colonoscopies are both expensive and invasive. In some other countries, less invasive alternatives like stool testing are preferred.

American doctors have pointed to strong evidence that colonoscopies work. According to the article:

Past research always showed that colonoscopy could put a huge dent, on the order of 70%, in the incidence and mortality from colon cancer.

In other words, we can reduce colon cancer deaths by 70% - purely by screening, and presumably catching cancer in its early stage. A researcher was quoted saying the profession has expected colon cancer to become "extinct" if every man were to take colonoscopy.

***

Chances are men were not informed that all past evidence about this practice has come from observational studies only.

Observational studies involve comparing the outcomes of people who have taken colonoscopies and those who haven't. However, because people self-select into these procedures, a direct comparison of the two groups is biased. The people who choose to do it are likely to be different from those who don't, beyond the decision to be screened. Analyzing observational data involves finding ways to remove biases.

Take a current example. It is obvious that when comparing the Covid-19 outcomes of vaccinated and unvaccinated people, these two groups are not identical because self-selection introduces biases. There's age bias, political bias, etc. etc. It's challenging enough to enumerate all sources of biases. The hard nut to crack is to learn the magnitude of such bias. Even if we know older people are more likely to be vaccinated, and more likely to suffer more severe consequences from Covid-19, how do we know the degree of bias? If we don't know the size of the bias, how can we tell if we have adjusted too much or too little?

It's like asking the tailor to adjust the length of a pair of pants without telling her the height of the person wearing those pants. It's just too long, the tailor is told.

Back to the observational studies of colonoscopies. Doctors have relied on a series of peer-reviwed observational studies that converge on the conclusion that colonoscopies are extremely effective. What can go wrong?

***

The profession ignored the nature of the evidence, the deficiencies of observational studies.

Finally, some researchers in Norway had the good sense to check these results by running the gold standard of medical evidence, a randomized clinical trial. In these trials, patients are randomly selected to get invitations to take colonoscopies. The random assignment of treatment is an automatic adjustment for known and unknown biases. (Read this previous blog for a primer.)

And... after 10 years of observation, the RCT showed that colonoscopies has essentially zero effect on mortality from colon cancer, and only about 20% reduction in incidence, both metrics coming in far below 70%, the efficacy found in the observational studies. I'll dig deeper into the results in a future post. For this post, I take these findings at face value.

***

What gave me feelings when reading the Stat article is the response from various experts who effectively advocate trashing the RCT result in favor of the observational studies. This reaction is shocking given the RCT's status as the gold standard of medical evidence.

This RCT is not some summer project by a student intern. It includes 85,000 people, with data collected over 10 years. And yet, the article ends with a quote that says "colonoscopy is still king".

The negative reaction reflects the belief that all evidence are created equal, and illustrates the skepticism toward counter-intuitive scientific findings. It's not just in medicine. Anyone who have run experiments in industry has encountered this attitude.

The arguments for rejecting the inconvenient results can be classified into the following types:

(a) Some other related evidence exists that contradicts the current study - e.g. a randomized trial of a related procedure has previously shown better results

The problem with this type of argument is that it says nothing about the current study. A Chinese term for this strategy is "raising East to beat West". This strategy works under the doctrine that all evidence are created equal. The other study is on a different procedure and it is brought into this conversation purely because it obtained the more "acceptable" result. Imagine this: if that trial has produced worse outcomes than the current study, would the expert have mentioned it to the reporter?

If we allow prior experiments to overrule new experiments if the latter's conclusion differs from the former, then what's the point of running the new experiments?

(b) The treatment worked better for a subset of the test subjects - e.g. if we restrict the analysis to the older population, the positive effect of the treatment is higher

Whenever the overall results of an experiment disappoint, the sponsor of the experiment almost always asks for "deep dive" analyses, which typically means, hunting for a subset of the population for which the treatment made a difference. What's wrong with this? Imagine, what if the researcher discovered a subset of patients for which the effect of colonoscopy on cancer diagnosis was not 20% reduction but 5% reduction? This finding will be dismissed and ignored because it adds "nothing new" to the overall conclusion. The only additional findings that can get the attention of the sponsor are those that validate the desired outcome.

Besides, any such deep dive analysis runs into statistical problems. The sample size of the subset is usually too small, harming the reliability of the result. The filtering effect of statistical significance means any such finding is likely to be an outlier for the experiment.

Worse of all, the sponsor may take the positive outcome for the subset, and create the impression that it works for everyone. This actually happens a lot more frequently than one thinks. Take a Covid-19 vaccine study that uses matching to remove biases. While the matching procedure increases our confidence that the two groups being compared are similar, many records are dropped from the analysis because they are unmatchable. For example, because most older people got vaccinated, and many younger people did not, the matched subpopulation has a different age profile than the full population. While the comparison of the matched groups is not biased, the extrapolation from matched groups to the general population is biased. Nevertheless, experts used the matching study to proclaim that the vaccine is effective for all, rather than that the vaccine is effective for the matchable subpopulation.

(c) Our hypotheses are better than your data - e.g. because treatment methods are improving over time, we believe the benefit will increase over time, therefore in five years' time, the outcome will be better

Everyone is entitled to their hypotheses but only those running the experiment has submitted their hypothesis to testing with real-world data. The experts offer no data to support the alternative hypotheses. The experts are also ignoring hypotheses that reinforce the study's findings. Imagine this: what about the possibility that someone would develop a new screening method that might dominate colonoscopies in the next five years?

(d) The data should be analyzed in some other way - e.g. we should use a per-protocol analysis

This reaction makes a mockery of pre-registering the study methods. We face the same problem  of ex-post hunting for a specific outcome. If the per-protocol analysis had yielded similar or "worse" results as the intention-to-treat analysis, the same expert would not have raised this critique.

***

Imagine this: the RCT for colonoscopy has confirmed the profession's longstanding view that the procedure cuts cancer deaths by 70% or more. What would the article look like? Would there be a line-up of experts raising concerns that the RCT outcome might be wrong?

All four types of counter-arguments could still have been raised. There is likely going to be at least one randomized trial of a related procedure that showed much lower efficacy. If the analysts did a deep-dive analysis, at least one subset of the patients probably experienced much lower benefits than the average. Various hypotheses could be proposed that suggest the RCT over-estimated the efficacy. Some other way of analysing the data would have yielded lower efficacy.

Nevertheless, we wouldn't be hearing these arguments.

***

Lost in this debate is the real question raised by the RCT; in fact, it hasn't even been asked. So I'm asking it here: What went wrong in those prior observational studies to have over-estimated colonoscopy's effectiveness to such a large extent? How did the machinery that "adjusts" for biases fail?

 

 

 

 

What about safety

Readers may have wondered why I have said little about vaccine safety in my reviews of vaccine trials. Today's post, which fills that gap, also serves to explain why.

As is well known, the FDA proclaimed that the mRNA vaccines (marketed by Pfizer and Moderna) are both efficacious and safe. Each trial enrolled about forty thousand participants, split randomly into vaccine and placebo groups, with the latter receiving a saline solution instead of the vaccine.

For efficacy, the primary outcome of interest was infections (measured by its proxy, reported symptomatic cases). As observational data drifted away from this narrative, the outcome shifted to hospitalization and death attributed to Covid-19. Whether the endpoint is cases, hospitalization or death, the clinical trial design ensures that an analyst can look at the difference (ratio) of the event rates to assess whether the vaccine performed better than the placebo. [That is, until after the emergency use approval was granted, because the FDA then allowed the pharmas to dismantle the placebo groups, destroying the trial design.]

Measuring safety - on the surface - may proceed in a similar way. However, safety outcomes present two enormous challenges that make safety analyses much harder - and thus less convincing - than efficacy analyses for the same cohort of trial participants.

First, it's difficult to know which safety outcomes are of greatest concern. It's a bit of a fishing expedition. Lots of outcomes are measured to screen for signals. Such data mining always raises the probability of false signals. Thus, all else equal, one would require a larger sample size to analyze safety outcomes. Since the sample size is fixed for both safety and efficacy outcomes, this implies that the confidence we have for safety is necessarily lower.

Second, the truly horrible safety outcomes - known as "severe adverse events" (SAEs), which include hospitalization and death - are expected to be rare events. Since a vaccine is indiscriminately injected into the population, most of whom are healthy people, even a small rate of occurrence can be alarming. Let's assume a hypothetical vaccine causes death with probability 1 in 500,000 shots (this is a thought experiment for illustration purpose only). Bloomberg reported that over 600 million shots have been administered in the U.S. alone. On average, 1,200 people would have died from this vaccination.

I chose 1 in 500,000 as the hypothetical rate of vaccine-induced death for a reason - the vaccine trials as designed would not have been able to detect this signal, let alone death rates below that level. For an event that happens at a rate of 1 in 500,000 attempts, the chance of observing zero occurrences in 25,000 people is 95%, and the chance of observing a maximum of two occurrences is 99.99999%. In other words, any SAE with a rate of occurrence of 1 in 500,000 would likely produce no events during the vaccine trials, and given the amount of shots that went into arms, that level of harm would have killed 1,200 people (while the analyst legitimately touts a perfectly safe vaccine).

***
In their respective trials, at the interim analysis that happened when half the participants have reached two months of observation beyond the second shot, Pfizer and Moderna observed few SAEs, and concluded that these event rates were "balanced" between the vaccine and placebo arms. This was the basis of the safety claim.

A group of researchers have recently revisited the analysis, and drawn a different conclusion in a peer-reviewed article published in the journal Vaccine. (link) They argued that the mRNA vaccines caused an excess of 10-15 extra SAEs of special interest relative to placebo. That's a difference. As a ratio, the mRNA vaccines caused a 43% increase in SAE risk. Most of the excess risk in the category known as "coagulation disorders".

It's instructive to learn why the same data led to different conclusions.

The first difference is what is counted as an SAE. It turns out that the FDA and U.S. pharmas created their own definition of what counts as SAEs. Meanwhile, the international community, something called the "Brighton Collaboration", agreed on a priority list. The new analysis reclassifies the events reported by Pfizer and Moderna in an attempt to perform an analysis that follows the international standard. Even here, the researchers conducted two analyses, based on the "original" list and an "expanded" list.

[For those wanting to peep into the sausage factory, the reclassification involves three experts readjudicating the events based on subjective evaluation of case history and other information. The original classification during the trials also involved such subjective judgment.]

The second difference is the proper handling of differential observation time among participants. Remember that the interim analysis were conducted while people were still in trial. I noted when I read the efficacy analyses the fact that the earlier participants were observed for longer periods of time, so that, even though the "median" participant reached the two-month point post second dose, the interim analysis involved many participants who had not yet reached the start of the case-counting window, which means that for the purpose of the interim analysis, these participants were "immortal"!

It turned out that the FDA-endorsed analysis included participants who have not yet received the second shot, since they counted any SAE at data cutoff, i.e. the same cutoff date applied to all participants regardless of enrollment date. The new analysis removed participants who had not yet taken the second shots.

***

A few other noteworthy nuggets can be found in the article. (link)

First, the authors noted that the Pfizer (Moderna) vaccine reduced risk of hospitalization by Covid-19 infection of 6 (2) per 100,000. So, for every 100,000 vaccine recipients, around 2-6 are expected to have avoided hospitalization relative to not getting vaccinated, but 10-15 are expected to experience serious adverse events relative to not getting vaccinated. As the authors noted, such a benefit-harm analysis should be conducted by age group but one of the enduring mysteries of Covid-19 vaccine science is the lack of available data of sufficient granularity for third-party researchers to conduct independent studies - all the more baffling given the claim of massive benefit and zero harm, which presumably should withstand even moderate- to large-sized error bars!

Just for illustration. The 2-6 hospitalizations are likely concentrated in older people and if the 10-15 excess SAEs occur mostly in younger people, then the risk in the young is higher than average while the benefit is lower than average.

Second, Pfizer stopped reporting safety outcomes beyond one month after the second shot.

Third, Moderna counted Covid-induced SAEs in the SAE analysis, which the authors pointed out is not the "traditional" analysis. (Since supposedly the vaccine's side effects are similar to the effect of Covid-19 infection, differentiating between them requires experts to determine the cause of the observed SAEs. That's definitely not something you or I am qualified to do!)

 

 

 

What is AB testing: a 2020s view

I don't mean what is the concept of AB testing. If you've been reading this blog, you already know what it is (for those trained in design of experiments, it's the very first lecture.). Here are some prior posts about AB testing (1, 2, 3, 4). I even made a video about AB testing from a business operations perspective here.

All big websites have people whose job is to run AB tests. These tests are supposed to help figure out the optimal page designs.

The devil is in the details. Every AB test has one or more objectives it's trying to optimize. Most AB testers believe that the overarching objective is to make users "happy".

It takes very little time to debunk that myth.

I'll drop an example in this post.

***
In recent times, Twitter has been running an AB test. I know because I have been on both arms of this test. They are testing a pop-up that shows up after a small amount of scrolling. (I am not logged on.)

Here is Version A.

Here is Version B.

That's my reconstruction of Version B as I think they have locked me into a specific test group, or they have already discarded Version B from the test at the time of writing.

Notice that Version A completely blocks access to the feed unless the user logs into or sets up an account while Version B has an X in the corner that the user can use to remove the pop-up and continue to read the feed without logging in or signing up.

***

What are some obvious metrics the AB tester would use to evaluate this test?

What is the goal of the test? What is the tester's hypothesis prior to running the test?

If your goal is to make users happy, is this the test you'd design?

***

Bonus question: AB testing, or randomized experiments in general, has been heralded as the gold standard for causal inference. It's the best way to learn what is the "cause" of observed effects. In all likelihood, version A outperforms version B on the objective of generating more sign-ups or log-ins. What is the cause of this effect?

Is it the presence/absence of the "X" in the corner? Is it the decision to block/allow users to work around the pop-up? Is it blocking/allowing users to access the site contents? 

Let's say the test shows no effect. What did we learn about causality (or not)?

[Metrics: number of new sign-ups, number of log-ins, time on site, bounce rate are the most obvious.]

 

 

Book review: Don't Trust Your Gut

Seth Stephens-Davidowitz has a new book out early this year, "Don't Trust Your Gut", which he kindly sent me for review. The book is Malcolm Gladwell meets Tim Ferriss - part counter intuition, part self help. Seth tackles big questions: how to find love? how to raise kids? how to get rich? how to be happier? He invariantly believes that big data reveal universal truths on such matters.

In 2013, I speculated about the challenges faced by big data analysts in this piece for Significance. Big Data has become a real thing in the decade since its publication. Seth's book interests me as a progress report on the state of "big data analytics".

If there is a common core of this style of analytics, as demonstrated by Seth's examples, I'd include the following elements:

a) Big Data is less defined by the amount of data, but by several characteristics I previously labeled OCCAM (see this post).

The data are typically collected by passive observation (e.g. tax records, dating app usage, artist exhibit schedules). Meaningful controls are absent (e.g. no non-app users, no failed artists). The dataset is believed to be complete. The data aren't specifically collected for the analysis (an important exception is the happiness data collected from apps for that specific purpose). Several datasets are merged to investigate correlations.

b) Much - though not all - of the analyses use the most rudimentary statistics, such as statistical averages.

This can be appropriate, if one insists one has all the data, or "essentially" all. An unstated axiom is that the sheer quantity of data crowds out any bias. This is not a new belief: as long as Google has existed, marketing analysts have always claimed that Google search data are fully representative of all searches since Google dominates the market. (Seth's previous book covers his analyses of Google search results.)

c) If the analyst incorporates model adjustments, these adjusted models are treated as full cures of all statistical concerns.

The last few chapters on activities that cause happiness or unhappiness report numerous results from adjusted models of underlying data collected from 60,000 users of specially designed mobile apps. The researchers broke down 3 million logged events by 40 activity types, hour of day, day of week, season of year, location, among other factors. For argument's sake, let's say the users came from 100 places, ignore demographic segmentation, and apply zero exclusions. Then, the 3 million points fell into 40*24*7*4*100 = 2.7 million cells... unevenly but if evenly, each cell has an average of 1.1 events. That means many cells contain zero events. By the magic of model adjustments, all cells have non-zero estimated activities and associated happiness quotients. The estimates in many cells reflect an underlying model that hasn't been confirmed with data - and the credibility of these estimates rests with the reader's trust in the model structure.

I observed a similar phenonmenon when reading the well-known observational studies of Covid-19 vaccine effectiveness. Many of these studies adjust for age, an obvious confounder. Having included the age term, which quite a few studies proclaimed to be non-significant, the researchers spoke as if their models are free of any age bias. Taking this line of logic further, such a belief implies that no one should bother with running expensive randomized clinical trials because a self-selected trial coupled with post-hoc regression adjustments is a perfect substitute!

d) A blurred line barely delineates using data as explanation and as prescription.

Take, for example, the revelation that people who own real estate businesses have the highest chance of being a top 0.1% earner in the U.S., relative to other industries. This descriptive statistic is turned into a life hack, that people who want to get rich should start real-estate businesses. Nevertheless, being able to explain past data is different from being able to predict the future.

A kind of feedback loop is also at play. The average odds of getting rich surely depends on how many people are entering the industry. So if people read Seth's book, and rush into real estate, the data may no longer rank this industry as best.

e) Most of the featured big-data research aim to discover universal truths that apply to everyone.

For example, an eye-opening chart in the book shows that women who were rated bottom of the barrel in looks have half the chance of getting a response in a dating app when they messaged men in the most attractive bucket... but the absolute response was still about 30%. This produces the advice to send more messages to presumably "unattainable" prospects.

Such a conclusion assumes that the least attractive women are identical to the average women on factors other than attractiveness. It's possible that such women who approach the most attractive-looking men have other desirable assets that the average woman does not possess.

It's an irony because with "big data", it should be possible to slice and dice the data into many more segments, moving away from the world of "universal truths," which are statistical averages, said differently. In the last chapter of the book, Seth describes one study that tackles this problem.

***

I wish Seth had devoted more space to explaining model adjustments and statistical issues, and discussing whether the remedies are sufficient. His target audience is clearly people with minimal mathematical training, and I understand his decision not to open such cans of worms. The references at the end should satisfy those interested in exploring further.

As with Gladwell, I recommend reading this genre with a critical eye. Think of these books as offering fodder to exercise your critical thinking. Don't Trust Your Gut is a light read, with some intriguing results of which I was not previously aware. I enjoyed the book, and have kept pages of notes about the materials. The above comments should give you a guide should you want to go deeper into the analytical issues.

I think there is a lot more that can be done with big data, we are just seeing the tip of the iceberg. So I agree with Seth that the potential is there. Seth is more optimistic about the current state than I am.

The role of prior infection in the vaccine trials

In the final analysis of the J&J vaccine trial (link), the study authors suggested that their vaccine benefits those who have previously been infected with Covid-19:

We observed that participants with previous asymptomatic infection (defined by serologic positivity for SARS-CoV-2 N protein and an absence of history of symptomatic Covid-19) can benefit from immunization with a Covid-19 vaccine. In a post hoc analysis, previous infection alone provided 90.4% protection against symptomatic infection, and after administration of Ad26.COV2.S in seropositive participants, 97.7% protection was observed in a comparison with seronegative placebo recipients.

Attentive readers should be raising their eyebrows as they digest this paragraph. The headline vaccine efficacy (VE) reported in this study was 56%. Against this backdrop, the above claim of 98% efficacy for people who have already caught Covid-19 before (aka seropositive) is quite simply astounding.

[The authors, nevertheless, chose their words carefully, as the statement does not actually make a causal claim, but merely promotes such an interpretation.]

***

Let's unpack the underlying data.

The randomized trial design included the usual pair of treatment groups (vaccine and placebo). However, FDA allowed excluding many subgroups post-randomization in the statistical analysis. One such exclusion are people who were seropositive at baseline (actually, it appeared that they may have excluded also anyone who got infected before 14 days after the shot, an additional complication I will ignore in what follows).

We can subdivide the trial participants into four subgroups, as shown in the following table:

Roughly 10% of the vaccine and placebo groups were deemed seropositive (previously infected) at baseline and removed from the headline analysis. The "post-hoc" analysis brings the seropositive subgroups back into consideration.

The headline analysis ignores the first row of the table, and reports on the difference between the case rates between the placebo and vaccine groups in the second row (expressed as a relative ratio). As mentioned already, the VE is 56%, which means that the case rate of the vaccinated was roughly 44% that of the placebo group. Since we are allowed a causal interpretation in the context of randomized trials, the vaccine roughly cuts the case rate by half, from 5.5% to 2.5%.

If the VE were to be 98% instead of 56%, the case rate of 5.5% would have to drop to 0.1%. This is the first place where the paper's aforementioned claim can easily be misinterpreted. Because the VE is expressed as a relative ratio, a 98% drop is not the same as a 98% drop - because the base rates are not equal.

In this example, the base rates cannot be more different. The average case rate for sero-positive (i.e. with prior infection) is only one-tenth of that for sero-negative people. As shown below, on the placebo arm, those with prior infections had a case rate of 0.6% (despite not receiving the vaccine) compared to 5.5% if they did not have a prior infection.

As the study authors acknowledged, natural infection acts like a vaccine with 90% efficacy. Well, they used more scholarly words:

Previous infection alone, in an analysis involving seropositive and seronegative placebo recipients, was found to provide 90.4% (95% CI, 83.2 to 95.1) protection against moderate to severe–critical Covid-19.

Now, where did the 98% claim come from? It turns out the authors were making a comparison between two unlikely subgroups: (seropositive + vaccine) vs (seronegative + placebo).

This comparison is odd in a number of ways. Two conditions are varied at the same time. The 98% VE represents the aggregate effects of the vaccine plus prior infection. The study makes no attempt to assign relative importance between these two changes. In theory, the effect of the vaccine can range from all of the 98% to none of it.

From a practical view, each person either has caught Covid-19 before or hasn't. Unlike the vaccine decision, serological status is typically not a manipulable treatment (unless someone deliberately attends a "Covid party".)

If you're unvaccinated with a prior infection, then your decision is whether or not to get vaccinated. The appropriate comparison is (seropositive + placebo) vs (seropositive + vaccine). This is the first row of the table.

If you're unvaccinated without prior Covid-19, then your decision is - also - whether or not to get the vaccine. The appropriate comparison is (seronegative + placebo) vs (seronegative + vaccine). This question is the primary endpoint in the trial.

The vaccine efficacy for seropositive people is 76% (first row). This is quite a bit higher than the 56% for seronegative people (second row). Remember though that 76% represents a reduction of case rate by 76% - off a base of 0.6% while the 56% represents a reduction of case rate by 56% - off a base of 5.5% so they are not directly comparable values. Besides, at the time of the trial, seronegatives outnumber seropositives 9 to 1.

In glorifying numbers > 90%, the study authors picked the larger case rate from the placebo column and the smaller case rate from the vaccine column to produce a meaningless 98% number. A fuller analysis shows that the vaccine is decently effective for those who have been infected but the improvement is on top of a tiny case rate.

***

If we are serious about learning the effect of vaccination on those who have already been infected, what experiment should we run?

It's straightforward. Recruit people who had previous infections. Randomly divide them into two groups, give one group the vaccine and the other group a placebo. In such a trial, you are effectively comparing (seropositive + vaccine) and (seropositive + placebo). This is precisely the first row of the table shown above.

The only difference is that the sample size of seropositive people was too small during the original vaccine trial. There were simply not enough people with Covid-19 when the trial was initiated.

***

At this juncture of the pandemic, experts have acknowledged that these Covid-19 vaccines do not stop infection or transmission - they say the purpose of these vaccines is preventing hospitalization and deaths.

In the vaccine trials, basically no one with prior infection subsequently was hospitalized for or died from Covid-19. In the J&J trial, only 15 participants out of over 42,000 were both seropositive at baseline, and subsequently counted as cases. Therefore, we have no data on how the vaccine affected the chance of hospitalization or death among the seropositive subgroup (but we know it's not a big number).

The 98% VE number mentioned in the opening quote of this blog is a metric about preventing infection, not severity of disease. Thus, we have been served incongruent messages that (a) previously uninfected people should not worry about getting infected (and only worry about getting severe disease) while (b) previously infected people should worry about getting infected, even though their case rate is 10 times lower than the previously uninfected.

 

More case-counting puzzles

The Johnson & Johnson vaccine is being treated as a third-class citizen due to some safety concerns (link). Interestingly, such concerns did not surface in the "final" analysis of the single-shot Covid-19 vaccine, published in February, 2022 in New England Journal of Medicine (link). In this report, the scientists concluded "Ad26.COV2.S was associated with mainly mild-to-moderate adverse events, and no new safety concerns were identified". Since in the interim report, the FDA said "the analysis supported a favorable safety profile with no specific safety concerns," we were supposed to believe that the J&J vaccine is completely safe.

On the efficacy side, the final report headlined 56%. This number is meaningless unless we know how VE is defined. The 56% number comes from a counting window that starts 14 days after the shot (1D+14). During the interim report, the headline number was 67% (link).

However, these numbers are not comparable for reasons I'll get into in this post. The FDA has allowed pharmas to keep shifting the goal posts so nothing can really be compared like for like. The situation illustrates clearly why statisticians won't give you an answer unless we know all the details about how things have been measured!

***

First, the interim report was conducted when only about half of the participants have reached 2 months of follow-up - note that the largest number on the time axis was 126 days.

In the final analysis, about half the participants have reached 6 months of follow-up - the largest number on the time axis is now about 280.

So, the comparable value from the final report is not the headline VE which concerns up to 280 days of follow-up but the VE at day 126... and that number is 55% (1 - (436-100)/(938-185)). According to Table 2B (final report), the error bar around this number is 20% to 70% so the trial did not enroll enough participants to give a precise estimate of VE.

Harmonizing the follow-up period is not sufficient. Between the initial and final analyses, J&J ditched the requirement that all cases be "centrally confirmed". In the interim report, cases were counted only if a central laboratory confirmed the local PCR test positives. In the final report, cases accrued based on local test results.

***

Another major development is letting the placebo group get vaccinated. After J&J's vaccine was granted EUA at the end of February 2021, all participants reaching 6 months of follow-up were effectively unblinded so that those in the placebo group could get the vaccine if they so chose. Almost everyone did.

Since enrollment ended October 2020, and the data cutoff for the final analysis occurred in July 2021, everyone could have been followed for at least 6 months, if the above plan were implemented to the letter.

But that was not what happened. Some participants were unblinded prior to reaching the 6-month point because they requested getting one of the by-then authorized Covid-19 vaccines.

As a result, only 23% of the study participants had an observation window of at least 6 months. (By design, 100% should.) In fact, half of the participants in this trial had a follow-up period of less than four months. Because the follow-up period was truncated by the placebo participants receiving the vaccination, effectively destroying the placebo group, there will be no further useful data.

Refer back to the case curve shown above, notice that the at-risk population has dropped by more than half by day 126. The further out one went in time, the fewer participants contributed to the incremental case numbers, the wider the error bar. Such dropoff does not happen in a normal clinical trial. 

Those who reached the longest follow-up period are also the earliest enrollees in the trial. In most vaccine trials, the pharmas first enrolled the lowest risk people - in the case of J&J, the earliest batch of enrollees were young people with no comorbidities. So the right hand side of the case curve lacks both validity and reliability.

***

When I wrote about the J&J trial, I pointed out that it was one of the few double-blind trials - the Pfizer and Moderna trials were single-blind. Blinding removes a source of potential bias - a kind of self-fulfilling prophecy in which the investigators or the participants behave in a way that advantages the treatment against the placebo.

We now have solid evidence that blinding did not work well, and confirmation bias was a real concern. To begin with, given that about 30% of Americans have obstinately refused to get the vaccine, we know that this subset is not represented in these vaccine trials. When the placebo groups in these trials were offered the vaccine, almost all of the participants took advantage, implying they were vaccine enthusiasts, not skeptics.

Blinding is supposed to neutralize partisan feelings about vaccines but it didn't work as planned. In Figure S1 (in the appendix), we learned that 1,222 placebo participants "received other vaccine before unblinding", compared to 670 on the vaccine arm. That's 80% more on the placebo arm than the vaccine arm. If the treatment groups were properly randomized, and if participants were properly blinded, they would not know whether they received the placebo, and so the proportion of people who got another vaccine against trial rules should be balanced.

The number of participants who withdrew during the trial was also disproportionately higher on the placebo arm (338 vs 224), about 50% more. These are huge gaps.

***

One of the big mysteries of Covid-19 vaccine trials is what counts as a case. The case-counting window is one variable I've discussed often. But there's more. Most of these early trials were designed to measure symptomatic, PCR-confirmed cases. J&J added a definition of mild, moderate and severe cases, although as I pointed out before, the definition of "mild" disease is meaningless since almost every case has been classified as moderate or severe.

In most studies, including the J&J trial, all participants are given serological tests at regular time intervals. Seropositivity indicates that someone has been infected in the past, and thus, it is presumed that said infections were asymptomatic. This device has been used by scientists to claim the vaccines also prevented asymptomatic infections.

Apparently, J&J now believes that someone without a positive result from either PCR or serological testing can also be infected. This is how I interpret one of the footnotes from Table 1 that corresponds to a concept called "any SARS-CoV-2 infection", which includes:

"undetected cases that were subsequently detected through a positive serological result (according to the clinical severity adjudication committee), which did not count as either symptomatic cases (becqause they were RT-PCR-negative) or asymptomatic cases"

I'm not sure how the adjudicators got wind of the "positive serological result" since if such a result was present, the case should have counted as "asymptomatic", which is a different line in Table 1.

(Also, the severity adjudication committee is a misnomer as the committe members concerned themselves here with asymptomatic infections that would be milder than "mild".)

What is shocking about this line of numbers is the magnitude. There were 1,038 such infections in the vaccine group and 1,699 in the placebo group. These two numbers are much higher than the counts that made it to the headline VE calculation. In particular, only 42% of the vaccine infections, and only 52% of the placebo infections, ended up as "cases". This calculation also shows that an "any" infection on the placebo group has a higher chance of being counted as a case than an "any" infection on the vaccine group.

 

 

 

 

 

 

 

 

 

The fun and frustration of seeking context

This is a third post related to an exploratory analysis of electricity consumption data from California, published at StackOverflow. The previous two posts are here and here.

***

Exploratory analysis always contains twists and turns. The linear presentation shown at StackOverflow likely reflects an ex-post simplification that dropped deadends, U-turns, etc.

There are two points in this journey at which I'd have taken a different turn.

The first moment relates to the peak consumption day in 2020:

The analyst discovered that on August 19, between 4 and 6 pm, there was a peak in usage. The analyst then looked for context (a good idea!). This leads to the following sentences:

A quick Google search for “California Aug 19th 2020” shows that the region was suffering from wildfires, so perhaps people kept windows closed and the AC on instead of opening their windows to the cooler nighttime air. September 6 also shows up among the highest values, and a search indicates a likely cause: a record-breaking heat wave in California that hit the national news while the fires continued to burn.

If one wants to make this a key conclusion of the analysis, much more work is needed to validate the hypothesis. Heatwaves and wildfires are events that affect a region for a duration of time. It would be interesting to see the trend in energy use in that whole time window. Also, a top N google search is risky when trying to explain extreme values. Extreme values are frequently caused by extreme events, and it's not clear that such an extreme event may be picked up as one of the top N by search engines.

It's the phrase "quick Google search" that doesn't sit well with me. This part of the analysis moves from descriptive insights to cause--effect, and a "quick" search is not going to do the job!

***

The second moment at which I'd have taken a different turn concerns this scatter plot:

The analyst has now switched to a Texas dataset. The picture plots the relationship between electricity consumption and temperature. Each dot in the scatter plot is an hour.

I have seen a lot of similar analyses and they all suffer from an ecological fallacy. It's the problem of over-aggregation, ironically, it's the topic of the StackOverflow blog post, but on the spatial dimension rather than the temporal dimension.

Each temperature value is the average temperature across all of Texas, and each energy consumption value is the average usage across all Texans - for a specific hour of a specific day.

If there is a pattern that affects all (or most) of Texas localities in the same direction (and similar magnitude), then this chart will faithfully show the pattern. However, most interesting patterns are likely going to affect different localities in different directions/magnitudes, and so the above chart would have "aggregated away the signal", to use their words.

Just imagine a single dot which is a single hour of a day, and focus on the temperature dimension only. At any moment in time, the dot represents the average temperature across all weather stations in Texas, but the variability of temperature would be quite large. Some localities will experience higher than normal temperatures while others will show lower than normal. The weights of these localities on average Texas energy consumption is a factor lurking behind these numbers. The point is that the above scatter plot is not useful.

The lesson of the StackOverflow blog post is that instead of this aggregated scatter plot, the analyst should disaggregate the spatial dimension, and look for interesting patterns there.