Big Data, Plainly Spoken (aka Numbers Rule Your World)

If you're not counted, you don't count

If you're not counted, you don't count.

The first law of counting is to count everything, and I do mean everything - not just the things you think you care about.

***

Imagine - as I did in Chapter 1 of Numbersense (link) - that a college advertises that 95% of its graduates found jobs within 6 months of graduation. It turns out the school collects data from a post-graduation survey - it's not disclosed anywhere in the marketing materials that it's 95% of survey respondents, not 95% of graduates. What's also not reported is the survey's response rate, which was 20%.

Nevertheless, the college administrators stand behind their statistics. They say the respondents and non-respondents did not show statistically significant differences in their gender distribution, average age, and type of degree; therefore, they assume that non-respondents also have a 95% employment rate within 6 months.

Do we think graduates who failed to land jobs have the same chance of filling out an employment survey as those who are gainfully employed after graduation? You can draw your own conclusion.

If you only have biased data on 20% of the graduates of this school, do you think you have a full understanding of the employment prospects of its graduates?

***

At the start of the Covid-19 pandemic - think March or April 2020, the U.S. suffered from a shortage of diagnostic tests. The official policy at the time was to test only people with severe symptoms.

We knew next to nothing about the novel coronavirus at the time. The data on cases became the basis of our knowledge - not only of the severity of the pandemic but also its trends, its demographics, its symptoms and injuries, and so on.

But triage testing leads to biased data. When the pandemic also grows quietly through asymptomatic transmission, our data led to a partially blind, misleading assessment of the situation. I warned about this in a Wired column - subsequently, colleges have shown that comprehensive testing is crucial to controlling the spread of Covid-19. Random testing of small samples of the population is a viable alternative we can still implement today to monitor new variants.

***

A curious thing happened during the pandemic to our unemployment statistics. Eight million Americans not only lost their jobs overnight but suddenly lost all interest in working. That's according to official statistics, which showed that the number of people in the "civilian labor force" dropped by 8 million.

People "not in the labor force" have always been a strange species. The government claims that these people either do not want work at all, or have not been looking for a job for an extended period of time. Their unemployed status is ignored in the unemployment rate. Oddly, as more people drop out of the labor force, the unemployment rate improves, all else being equal. This group is simply not counted; they are neither employed nor unemployed.

If one has the responsibility of reducing the unemployment rate, one will focus on unemployed people who are counted in the data, not the ones who are not counted. When you are not counted, you don't count.

In Chapter 6 of Numbersense (link), I explain how the government computes the unemployment rate in detail. Not counting a group of unemployed people hides some of the bad news. As I pointed out in this previous blog post, the sharp exodus out of the labor force at the start of the pandemic proves the serious shortcoming of the concept. None of the people who just lost their jobs due to Covid-19 are people who do not want work at all; their recently-ended employment proves otherwise. There also was not enough time that passed to qualify them as having given up on a job search.

***

The Bureau of Labor Statistics (BLS) does count everyone, including data on how many people have been removed from the labor force tally. These data just show up in an appendix, with much less attention and care.

We can't say as much about how the CDC counts Covid-19 cases in 2021. As this New York Times story disclosed, the CDC "stopped investigating breakthrough infections among fully vaccinated people unless they become so sick that they are hospitalized or die."

Just as BLS defines unemployed people as not really unemployed if they "don't want to work", CDC defines infected people as not really infected if they are fully vaccinated and are not hospitalized for severe Covid-19 symptoms. In this way, our data have come full circle, back to the same sad state that it was in March or April 2020. It tells us only about severe cases. To the extent that new variants spread through asymptomatic infections, we are as blind to that possibility as we were a year ago.

Furthermore, the change in counting rules upsets any trend analysis. Consider a real-world study of vaccine effectiveness. If the researchers report that the case rate of fully vaccinated people is much lower than the case rate of unvaccinated people, is it because of the vaccine or because we count mild or asymptomatic cases for unvaccinated people but not for vaccinated people?

It's not just fun and games with counting. Don't forget: infected, hospitalized, or dead people who got sick before 14 days are also treated as not really infected, hospitalized or dead. When you're not counted, you don't count. The CDC turns a blind eye to them because they happen to catch the disease prior to reaching so-called full vaccination status. Why do their lives not matter? That's something I have been wondering about throughout the pandemic, and I still don't have a good answer.

Don't make assumptions

If you're a regular reader, you should be shocked by this headline - you might think I have gone postal.

I believe all models of data incorporate unverified (even unverifiable) assumptions - whether they are implicit or explicit in the model, or whether the modeler are aware of them or not. In fact, when the modeler posits a model to be free of assumptions is when you know they know not what they speak.

A typical assertion of this type is "we did not adjust the data because doing so involves making subjective assumptions". The belief is that by "not making an assumption", the analysis is "objective".

How does this conversation even come about? Usually it's because a source of bias has been identified. (Otherwise, there would have been no reason to adjust the data.) When analysts say we've left the data intact so as not to inject subjectivity, they have just made the subjective decision that the bias is ignorable.

***

So why am I now saying don't make assumptions?

I'm now speaking from the perspective of consuming other people's analytical findings, like reading press releases or tweets about scientific research.

A primary goal of science communications is to inform readers of the researcher's conclusions, and to summarize scientific evidence enough to convince the readers of its veracity.

When we read the precis of scientific papers, we frequently make assumptions about what the researchers have done.

***

I invite you to think about your own process of comprehension as you read the following excerpt from the BBC News coverage of why the U.K. government approved Pfizer's Covid-19 vaccine for teenagers.

The third paragraph addresses the benefits of the vaccine.

After reading the above paragraph, you may have picked up the following information:
a) a clinical trial was conducted for teenagers in the U.S.
b) the trial's result is based on "more than 2,000" participants
c) the key result was 16 cases in the placebo group versus 0 in the vaccine group
d) a secondary analysis shows the vaccine works in these teenagers "as well as young adults aged 16-25".

If, after reading those sentences, you think you learned the clinical trial for teenagers replicated the prior adult trials, producing equally amazing results, then you have made assumptions and inferences about what the press release said.

If you dig into the scientific report in NEJM about the Pfizer teenager trial, you will learn the following:

The findings came from an interim analysis, i.e. an early peek of results. The primary endpoint of the trial was not 16 vs 0 but the comparison between teenagers and young adults. This comparison between two age groups was not based on adjudicated symptomatic PCR-positive cases detected from 14 days after the 2nd shot in placebo vs vaccinated (the criterion used in adult trials) but based on antibody levels in blood samples a month after the second shots. Antibodies were tested on 200 teenagers, just 20 percent of the vaccinated teenagers. The placebo group in the trial was not utilized for efficacy. Having antibodies does not automatically mean these individuals are protected from contracting or transmitting Covid-19. In fact, no one knows the minimum level of antibodies required to stop Covid-19, and that's why Phase 3 trials typically measure clinically relevant outcomes such as cases, instead of laboratory outcomes such as antibody levels.

These facts mean
a) the clinical trial setup was not utilized for evaulating benefits because the primary analysis compared vaccinated individuals in two age groups
b) the sample size of 2,000 was irrelevant to the primary analysis
c) the 16 vs 0 metric is a post-hoc exploratory result (see this previous post for why it was not pre-specified as the primary endpoint, and should not suddenly become the primary endpoint because the trial outcome pleases the investigators.)
d) When they said "vaccine works as well," they did not mean on the basis of symptomatic, PCR-positive Covid-19 cases - as one might infer from the previous sentence - but in terms of the level of antibodies induced by the vaccination.

***

Before I read the NEJM report, I also thought the clinical trial followed the same design as the adult trial. What's interesting to me is we inferred all the wrong information when we consumed the news article.

One possibility is a deference to authority: the headline says the U.K. government endorsed those findings based on the science. But notice that the report merely said benefits outweigh the risks, provided some numbers, and then we the readers inferred that the results were strong. The scientific statements did not contain any value judgement.

BBC News did not
a) say the design of the trial was the same as for adults
b) claim 2,000 is a sufficiently large sample size to draw sound conclusions
c) claim that 16 vs 0 constitutes good/strong/amazing evidence
d) define what "to work as well" between two different age groups mean.

If you have some statistical training, you may have made assumptions about what the researchers did. Specifically, you may have assumed that:
a) the trial is completed and the analysis is final
b) The sample size of 2,000 is designed to deliver valid conclusions about the vaccine's efficacy
c) The 16 vs 0 comparison meets the minimum criterion for the primary efficacy endpoint as outlined in a pre-specified protocol
d) The comparison between teenagers and young adults also concerns efficacy as measured in the adult trials.

I have pointedly called these assumptions because the paragraph did not provide any of those details. We used our professional judgment to infer what methods produced the results as given. We also extend our professional courtesy by assuming that the investigators did their jobs, e.g. in deciding that 2,000 participants would be enough to estimate vaccine efficacy to an acceptable level of precision (say within plus/minus 10% as opposed to 30%).

***

The preceding discussion is not met to disparage the Pfizer study. A less rigorous study is not worthless. Analyses that use other types of controls are acceptable.

I'm using this example to illustrate the process of comprehension when we read communications of science. We expect that if the scientists did something out of the ordinary, they will dislose it in the summary. When the details are absent, we may fill in the blanks by assuming the conventional.

What I have learned in many years of practice is to never make assumptions when reading other people's analyses. Read their detailed methodologies, get a hand on anything you can about how they measured any data, defined any metrics, what they included and excluded from the analysis, etc. etc.

Don't make assumptions. Find out what the analysts actually did.

P.S. I previously pointed out that the NEJM paper also has one key omission (link). If you really want to understand the science, you sometimes have to dig many layers deep.

What do we know about the Sinopharm vaccine? 2

The other day, I posted some notes about the JAMA paper featuring the interim analysis of the Sinopharm vaccine trial, a Covid-19 vaccine developed in China using the tried-and-true inactivated virus technology. The trials were conducted in UAE and Bahrain and the follow-up period for the median participant has reached 3 months since the second dose, which is a longer follow-up period than the earlier trials.

One takeaway from Part 1 and also my previous reviews of vaccine trial results is that vaccine efficacy (VE) numbers are not easily comparable. Trials were conducted in different countries at different times; definitions of cases were not standardized; case-counting windows varied. The duration of follow-up also matters because we expect the effect of any vaccine to wane over time.

***

In Part 2 of my review, I explore eFigure 1 in Supplement 3 of the JAMA report. This extremely important diagram traces how "cases" are qualified to be counted in the VE metric. All vaccine trials - not just Sinopharm - count only adjudicated, lab-confirmed, symptomatic Covid-19 cases.

The final VE metric in the Sinopharm trial included only 142 cases across 2 treatment arms and 1 placebo arm. e eFigure1 (partly shown on the right) revealed that there were 962 suspected Covid-19 cases (with symptoms) at the start of the process. That number was cut by 50-60% to 420 based on PCR testing negative.

The adjudicators then used their judgement to remove another 30-50% of the symptomatic, PCR-positive cases. We have 255 cases standing at this point.

Then the case counting window is applied, and another 113 cases vanished from the count. That leaves 142 cases that ultimately entered the VE calculation, resulting in the headline number of 70-80%.

For the sake of argument, if we got rid of the adjudication step, then the VE would have been reported as 37-56%. Because this calculation includes cases from first dose, compare it to 50-66% that was mentioned in the previous post. Thus, adjudication increased VE by roughly 10%.

Adjudication is not a unique feature of the Chinese trial. All vaccine trials employed adjudicators. The other pharmas have been far less forthcoming about what happened during this process. Pfizer, for example, disclosed to the FDA that they had 3,410 suspected cases (link) but I have not found any details on how that number got winnowed down to 170 cases actually counted in the headline 95% VE number.  

***

If you have the stomach for it, I am about to open another can of worms. This happens to be a very material issue common in many settings. When comparing two numbers, one can compute a ratio (sometimes called an index) or a difference. Ratios are very tricky to deal with, and easy to misinterpret. The VE is a ratio of case rates (think of it as a relative risk index).

In the Chinese trial, the headline VE was reported as 73-78%, based on 142 adjudicated cases of Covid-19 across the three arms. In one of the sidebar analyses, the scientists added "asymptomatic" cases, defined as cases found by PCR tests for which the participants did not report experiencing qualifying symptoms.

The following table shows the data disclosed in the report. You can see that the vaccines are clearly not as effective at reducing "asymptomatic" cases, as the differences between 16, 10, and 21 are not statistically significant.

In order to simplify the calculations, I will work with the following idealized example. I'm assuming 10,000 people in two arms. The vaccine arm counts 35 cases of which 20 are symptomatic. The placebo arm has 115 cases of which 100 are symptomatic. We can compute the vaccine's share of total cases to be 23% so that the corresponding VE is 70%. (In this calculation, I include asymptomatic cases.)

An alternative efficacy metric is to compute the difference in the case rates. This means 1.15% (115/10000) for placebo minus 0.35% (35/10000) for the vaccine, resulting in a difference of 0.80%. This number has a straightforward interpretation: for every 10,000 people vaccinated, 80 fewer get sick from Covid-19 relative to placebo.

What happens if we ignore asymptomatic cases from both arms as the pharmas did? The difference in case rates remain at 80 per 10,000 because we remove 15 from each side. This makes sense intuitively since the vaccine is assumed to be ineffective at reducing asymptomatic cases (15 vs 15). Therefore, whether we count symptomatic cases or all cases, the vaccine reduces the number of cases by 80 per 10,000 relative to placebo.

As a relative risk ratio, the VE actually increases from 70% to 80% after removing asymptomatic cases. Note though that the 70% reduction is off the total case rate while the 80% reduction is off the symptomatic case rate, which is a smaller number. This points to the challenge of interpreting VE as a ratio. All the different VE ratios based on different inclusion and exclusion criteria have different base rates and so they cannot be compared to each other! Eighty perent is not always better than 70 percent: 70 percent of a big number can be more impactful than 80 percent of a small number.

You can now see why the game of the vaccine trials is to restrict the VE calculation to regions where the vaccine performs well. (I'm using "regions" in an abstract sense which includes the geographical sense.) Such restrictions lead to a higher ratio reduction but the result is only relevant to those regions, usually where the base rate is lower. In other words, it's a conditional number, as I explained in the last post. It's a higher reduction off an often unintuitive, restricted base rate. The case-counting window is the same game as asymptomatics but with a different name.

***

Another frequently ignored issue in these vaccine trials is the weak blinding. The Chinese trial is actually double-blind whereas the Pfizer, Moderna, etc. are single blind (very commonly mis-reported as double-blind because medical reporters just assume all clinical trials are double-blind).

Blinding is a protective measure that prevents investigators or participants from behaving differently - conscious or not - because they know which arm individuals are in. When the placebo is saline solution, it's actually very hard to blind participants because those getting vaccines are much more likely to experience a variety of symptoms, such as numbness, headaches, mild fevers and so on.

As a result, a few designs use non-saline placebos. The Chinese trial used an adjuvant in the placebo. An adjuvant is an additive in the vaccine that helps it work better. It's unclear to me whether this placebo is preferred to saline. It depends on the chance that the adjuvant causes mild side effects similar to the vaccine (which is a good thing in terms of masking treatment status) and the chance that the adjuvant by itself could be beneficial (which makes the base rate higher and raises the bar for the vaccine). 

The only other exception I know of was one of the Astrazeneca trials, in which the placebo was a different unrelated vaccine. Participants might be less able to guess which arm they belong to in that trial. 

***

Finally, none of these trials is really serious about measuring asymptomatic cases. The only exception is again in one of the Astrazeneca trials: they actually asked participants to send in nasal swabs every week of the trial. In the typical vaccine trial, they only do PCR testing (a) if symptoms are self-reported and (b) at infrequent intervals, including before each shot, and then once a month or longer. With infrequent surveillance, only a fraction of the asymptomatic infections can be found.

 

Bargaining with statisticians

If you are running A/B tests in industry, you're used to this scenario: two days (or indeed, two hours) after the test started running, the test sponsor - someone who suggested testing that variant of the marketing copy because they believe strongly that the new version is of that miracle vaccine they have always dreamt - pointed to a huge spike in performance on the real-time tracker, and urged that the test be stopped early because the new version is obviously better, and we should pocket our gains immediately rather than conducting "academic research" for the sake of "scientific purity".

For those with any statistical training, alarm bells should be ringing loudly because such decision-making is the mother of all false positives. The key to understanding why is to imagine what would have happened if the first few hours of performance was in the opposite, undesirable direction. The test sponsor would have been very quiet. (The sponsor could have knocked on your door and said we should stop the test early and declare the new version an instant dud. I have not gotten that knock on the door once after having run hundreds of tests. It's just human nature.)

Worse, what really happens is that the test sponsor will remain silent until such time as the test version displayed a "significant" positive gap relative to the control version. This fallacy is technically called "running a test to significance". This may happen in two hours, two days, five days, whenever. The only time the test would run to its designed end date is when the gap between the test and the control versions never exceeds an acceptable minimum distance.

***

A few years ago, I wrote a piece for FiveThirtyEight, mischievously suggesting that baseball fans should leave the ballpark once the score gap is larger than a certain number of runs. It appears that most baseball fans - presumably including many business managers - think such behavior is blasphemous. Well, it's also blasphemy when they demand an early stop to an A/B test because the favored version of the marketing copy has a "substantial" lead!

It's also the same behavior when a pharmaceutical executive goes to the FDA and pushes to end a clinical trial early, which is the real topic of today's post.

***

The FDA has made a bargain with the pharmaceutical companies to allow "interim analyses," the point of which is to enable early stopping of clinical trials under clearly defined circumstances. Some safeguards have been imposed on such analyses, which as recent events have proven, are easily gamed.

Safeguard #1 is a limit on the number of interim analyses one can conduct during the course of the trial. Each analysis contributes a false-positive probability, and the more times one "peeks" at the outcomes, the higher the chance of a spurious conclusion.

Safeguard #2 is to raise the required standard of evidence for earlier analyses. This is a simple application of statistics: in an interim analysis, the sample size is much smaller, so the error bar around any outcome is much wider (it's worse than linear), and thus the required gap between test and control must be larger. Instead of a 5% significance level, the required level in interim analyses could well be 0.05%. This is useful but does carry risks. One way the gap clears the higher bar is if something strange happens by the time of the first analysis (i.e. an outlier event that will not repeat). Analogously, a baseball team might score 10 runs in the first inning, clearly enough for me to leave the ballpark but should one expect this team to score 10 runs in the first inning in the next games?

Safeguard #3 is to pre-specify three possible decisions during the interim analysis: a) stop the trial for efficacy b) continue to collect more data and c) stop the trial for futility. This is a sensible requirement.

***

Sadly, recent events have shown that statisticians have made a Faustian bargain. The other side has usurped the rules, rendering these safeguards toothless.

Exhibit #1: the recent approval of "aducanumab", a "treatment" for Alzheimer's disease by Biogen. I have not followed the entire saga, but according to this article in StatNews (link), the FDA has made many concessions throughout the process, including waiving Phase 2 trials, and allowing interim analyses of Phase 3 trials. During the interim analysis, the trials were stopped for futility. In other words, while the investigators hoped that at early read, the drug would prove so spectacularly successful that they could stop the trial and declare victory - they instead discovered that aducanumab performed badly enough that the trial had to be ended prematurely.

If the story ended there, I have no complaints.

Biogen continued to mine the data after the trial was terminated, and subsequently, submitted "more data" that they claimed overturned the interim analysis results. This action violates all three safeguards listed above. Biogen is no longer adhering to a limited number of interim analyses - in fact, it appeared to be running the trial to significance. The significance requirements in Safeguard #2 are tailored to the specific analysis schedule, and so those become irrelevant when the schedule is abandoned. Finally, the FDA has also dropped Safeguard #3 because a fourth decision path has been created: stop the trial for futility tentatively and continue to mine for any exploratory analysis that can contradict the interim analysis.

As before, the harm of this action is most readily seen if we imagine what might happen if the trial had been stopped for efficacy. Will Biogen continue to mine the data looking for evidence that the drug in fact did not work?

We don't even have to speculate about the answer because of other recent actions by the FDA and pharmaceutical companies.

***

Exhibit #2: the coronavirus vaccine trials have adopted a modified version of the interim analysis plan. Because of various other shortcuts, the allowable decisions at the interim analysis were a) granting emergency use authorization on efficacy and continuing the trial to its normal end date b) not granting EUA and continuing to collect more data c) stopping the trial for futility. It's actually not clear to me whether c) was ever considered an option but let's assume it was. The vaccine developers committed to continue to run the trials to at least one or two years even if they sucessfully received EUA.

In essence, the successful vaccine trials were stopped for efficacy - although the trials were not stopped because full approval has not been granted. Did the pharmas continue to mine the data so they can volunteer negative information to invalidate the EUA? The answer is emphatically no!

On the contrary, the pharmas argued that upon granting of the EUA, it has become unethical to keep people in a placebo group (i.e. unvaccinated). Therefore, within a few months, almost everyone in the placebo groups has been given the vaccine. In other words, they can't be mining  any more data given that the placebo group has ceased to exist.

This situation is the worst nightmare for any statistician who made the Faustian bargain of allowing early stopping through interim analysis. They were hoping that the other side is making a genuine effort to balance science with pragmatism. They are sorely mistaken. When the test is stopped for efficacy, actions are immediately taken that prevent further data mining. When the test is stopped for futility, the sponsor does not really stop the test, and continues to mine the data.

These actions make a mockery of the interim analysis bargain, which means we are back to square one - clinical trials are being run to significance, which is the mother of all false-positive findings.

***

P.S. This post is particularly relevant in light of the Citizen Petition to the FDA about potentially granting full approval to the Covid-19 vaccines based on interim results.

[6/11/2021: So far, three members of the scientific advisory board has resigned in protest of the FDA decision (link). No one on the board voted to approve the drug but the FDA leaders overruled the board.]

A Citizen Petition

A distinguished group of scientists and experts is circulating a Citizen Petition against the premature granting of full approval of the Covid-19 vaccines. You find the full text here.

In the last few weeks, we learned that both Pfizer and Moderna have already lobbied to have their vaccines fully approved. You may recall that their vaccines were granted emergency use authorization, which as the name suggests, is not full approval. A normal vaccine trial takes roughly two years because we need to assess the long-term benefits and risks of these injections. Time passing is the only way we can measure long-term effects. When the EUAs were granted, the FDA loudly proclaimed there would be no "short cuts", that the trials will continue until their designed end dates. It looks like they were just joking.

This fight is about maintaining the standards of science. It is about stopping bad practices from lingering on after this emergency is over. As you have read on this blog over and over, while the scientific results of the trials and real-world studies have been encouraging, the science has been far from perfect - and in this emergency, people have been willing to look the other way on the flaws of these studies.

The EUA status of the vaccines is not stopping anyone who wants to get vaccinated from getting the shots. A while ago, an argument was made that full approval would encourage employers to make vaccinations a requirement for work. This is a red herring that assumes no vaccine hesitancy among top decision-makers at all employers. People who are hesitant will simply find their way to employers who embrace vaccine hesitancy. We should have learned the lesson from the mask-wearing red herring. Some erroneously believed that if vaccinated people are "rewarded" with not having to wear masks, the vaccine hesitant people would suddenly decide to get innoculated. Polls show that the same people who are vaccine-hesitant are the ones who have been ignoring masking rules all along.

Another argument is that people will trust these vaccines more if they were fully approved, rather than authorized for emergency use. The elephant in this room is that a full approval being granted prematurely is not full approval. Those people who make the distinction between EUA and full approval surely can see through this ruse.

Trust is a precious commodity. Once lost, it's hard to earn back. I agree with the petition. Why is there an urgency to grant full approval?

***

To make your voice heard, click here to comment on the petition.

 

[P.S. The authors of the Citizen Petition has a piece in BMJ explaining their reasoning. See here.

Importantly, they said: "publicly raising any element of hesitation about covid-19 vaccines will be seen by some as irresponsible, stoking unfounded fears in the public’s mind and contributing to the “vaccine hesitancy” problem trumpeted every day. But the alternatives—privately raising concerns or simply remaining silent—are arguably more detrimental to public trust in the long run."

]

Adolescent trial showcases pre-registration and the Law of Small Numbers

The scientific report on Pfizer's adolescent trial did not bring closure on the two key issues I flagged in my previous post (link). Here are my notes on this trial. 

The Key Efficacy Finding Does Not Require a Randomized Controlled Trial

The researchers explained that based on the low expected Covid-19 case rates among 2,000 teenagers (split into two groups), they were not going to see enough cases to offer the same kind of analysis as in the adult trial. They have convinced the FDA (or the other way round) to accept immunogenicity findings as a substitute. That is to say, Pfizer will show only that injecting teenagers with the vaccine generates antibodies intended to fight the SARS-CoV-2 virus. 

The sample size issue is real. The infection rate in teenagers is well known to be much lower than in adults. It's hard to find national numbers split by age group, though. I found out that in California, roughly 4% of children below 17 have tested positive for Covid-19 during the whole pandemic while the number was close to 10% for people above 18. These numbers are cumulative for the 12-15 months of the pandemic whereas a typical vaccine trial spans 3-4 months at the time of interim analysis. Detecting a difference in tiny signals (rare events) is challenging and requires a large sample size. Two thousand is like 20 times fewer participants than the adult trial. 

Typically, immunogenicity results advance a treatment to Phase 3. These lab tests prove that the vaccine produces a good amount of antibodies as intended. However, having antibodies does not guarantee the vaccinated people will be protected from infection. Nor do lab results suffice to decide what level of antibodies is sufficient to reduce infection (and transmission). Further, Phase 2 trials involve few people. Thus, a Phase 3 trial is required for FDA approval. Phase 3 trials involve thousands of people, and measure clinically relevant outcomes, which for a vaccine usually means reduction of infection.

With immunogencity as the key efficacy metric, the adolescent trial is in spirit an expanded Phase 2 trial. You don't even need to have a placebo group, nor randomize treatment. Injecting someone with saline will not produce antibodies against the coronavirus. They also measured antibodies in only 200 kids, a 20%-ish subset of those enrolled in the adolescent trial. They could have obtained the same outcome by recruiting 200 kids, giving them two shots, and measuring their antibody levels afterwards. No RCT is necessary.

The average antibody level of the 12-to-15-year-olds is compared to a number they previously obtained from the adult trial, involving those in the 16-to-25-year-old age group. If similar levels of antibodies are found, they presume that the vaccine will work in teenagers.

There are several assumptions required to justify such a claim. First, they assume the vaccine works with the same efficacy for 16-to-25-year-olds in the adult trial as the overall VE. This is not a given since the trial did not contain enough people in that (or any other) age group to produce a reliable estimate of subgroup efficacy. Second, they assume there are no material differences between the two age groups that affect immune response. Third, they assume that a direct link between antibody levels and reduction in case rate has been established in the 16-to-25-year-old age group. If that is true, I haven't seen the evidence.

In the protocol, the scientists were supposed to provide an "immunobridging" analysis, which sounds like a mathematical model that justifies the third assumption. They ultimately did not come through with that model. 

 

The Trial Provides Minimal Data on Safety

While the placebo arm is redundant for the pre-specified efficacy analysis, the placebo participants are useful to establish a background level of side effects, to help the vaccine developer make the case that the vaccine does not cause more adverse incidents than expected.

Nevertheless, the conclusion one can draw from a study of 1,000 vaccinated participants is highly limited. If a deadly side effect has a 1 in 5,000 chance of happening, there is over 80% chance that there will be zero cases of this side effect during this trial. With probably ~15 million Americans aged 12-15, such a side effect would affect 3,000 kids. Fewer than 200 American teens have died from Covid-19.

The trial did not identify any major concerns but what we don't know, we don't know.

 

What about that 100% VE talking point?

On Twitter and cable TV, the talking heads have mostly not presented the trial results as I have above. Below is a typical example:

Notice that they ignored the immunogenicity results and trumpeted a vaccine efficacy (VE) number. In the study, Pfizer reported a "surprise," saying that they found 16 cases of Covid-19 among the placebo group, and zero cases in the vaccine arm. 

In the NEJM paper, on the other hand, the immunogenicity outcome was unmistakably featured as the primary endpoint. This raises a critical scientific issue that is perhaps hard for an outsider to grasp.

All clinical trials are required to submit protocols and analysis plans well ahead of enrollment. This process is called pre-registration (investigators are not supposed to make edits, especially major edits, while the trials are running although this practice has become a casualty of the pandemic). Pre-specifying your analysis prevents data mining for results after the data have materialized. Such data mining frequently results in spurious findings - mistaking one-off phenomenon as generalizable outcome. Spurious findings will not replicate if the trials were to be repeated.

In the adolescent trial, pre-registration made Pfizer think about how to demonstrate efficacy. They feared that using the usual measurement based on symptomatic, PCR-positive Covid-19 cases is likely to saddle the study without a conclusion because they were only willing to run a trial with 2,000 participants. Therefore, they chose the immunogenicity outcome as the primary endpoint.

Now, after the trial finished, the Pfizer press release - and its readers - trumpeted the 100% VE metric as if that was the pre-specified analysis. This behavior usurps the principle of pre-registration. 

So let me now explain why there is a good chance the observed VE is an outlier. If the whole trial were repeated, I'd be surprised to see the same result repeated.

According to the FDA's press release on approving Pfizer's vaccine for teenagers, the case rate in the placebo arm was 16/978 = 1.6%. In Pfizer's adult trial, the corresponding rate was 162/18325 = 0.9%. So, the teenagers were getting sick at a rate 80 percent higher than that of adults. If I assume that the kids were equally likely to get infected as adults at 0.9%, the chance that we would find 16 or more cases in 978 kids is less than 2 percent. If we were to run this trial 100 times, 98 or more of them would observe fewer than 16 cases in the placebo group.

But teenagers have less than half the case rate of adults as mentioned above. If I assume that the case rate for kids is at 0.45%, the chance that we would find 16 or more cases in 978 kids is 0.001%. Don't forget, also, that the case rate only counts adjudicated, lab-confirmed, symptomatic Covid-19 cases occurring at least 7 days after the 2nd dose so the real-life infection rate is even higher than the 1.6%.

This is very strong evidence that what was observed during this trial was an outlier event. 

Nevertheless, this outlier event has been painted as a "pleasant surprise". We didn't think we'd have enough cases to show anything but how could we refuse the unexpected gift of 100% efficacy? The real surprise was not the VE but the inexplicable, highly unlikely incidence of Covid-19 among teenagers enrolled in this specific trial.  

In fact, this is exactly the reason why pre-registration is good practice. If we allowed this type of post-hoc data mining, spurious findings will bloom. 

***

See also my previous rant about the way the paper was written up as if the adolescent and 16-to-25-year-old age group were originally planned as two arms of a trial.

See also my post on the Law of Small Numbers, one of the seminal contributions by Tversky and Kahneman. The Law of Small Numbers is the false belief that the Law of Large Numbers applies to one's small sample.

 

 

 

 

What do we know about the Sinopharm vaccine? 1

WHO has approved the Chinese "Sinopharm" vaccine for emergency use (link), soon after an interim analysis of its Phase 3 trial was published in Journal of American Medical Association (report). We have heard nothing from the FDA about Sinopharm, nor has there been any news on the Astrazeneca/Oxford vaccine, so the only vaccines that have been approved in the U.S. remain three homegrown vaccines.

The Sinopharm vaccine is notable for using the long-established science of inactivated viruses. The vaccine does not require special cold storage, and its long-term safety record is supported by successes in fully approved flu and polio vaccines, an advantage that the newer technologies do not enjoy. The primary unknown to be resolved in the Phase 3 vaccine trial was the efficacy against SARS-CoV-2, as the flu shot based on inactivated viruses have "low" but still accepted efficacy in the 30-50% range (link).

The Phase 3 trial brings the welcome news that the Sinopharm vaccine also has efficacy in the 70-80% range against placebo, on par with Johnson & Johnson and Astrazeneca, both of which come from the adenovirus family. The new result clears well above the minimum acceptable standard of 50% required by regulators.

VE of 50% means vaccination cuts the chance of developing symptomatic Covid-19 by half. The background case rate - as measured by the placebo group - is in the low single digits. In the Chinese clinical trial, the reported placebo case rate was around 1.2% in the four-month study period. Cutting that number by 70% means the chance of getting sick with Covid-19 for those vaccinated is just under 0.4%, or 4 in 1000.

Recall that when generalizing the VE from a single clinical trial to the general population, we should look at the lower end of the 95% confidence interval for VE, in which case, we should say the VE of Sinopharm when applied to the general population is expected to be 60-65%. This proviso applies to all vaccine trials, not just the Chinese one.

The design of this Phase 3 trial is essentially identical to those of the U.S. trials that I have discussed before. If anything, the JAMA report and appendix provide more details than I have seen in the other scientific reports.

Similarities

Let's first describe the commonalities. The Sinopharm vaccine also requires two doses, taken 21 days apart. The control group is given two doses of a placebo. The trial ran from July to Nov 2020 in UAE and Bahrain.

The trial only counts adjudicated, lab-confirmed, symptomatic cases of Covid-19. The trial investigator - that is, the vaccine developer - only investigates a case if the participant experiences qualifying symptoms from a pre-published list. The infection must be confirmed by a PCR test. Finally, a committee of "experts" determined if the PCR-positive, symptomatic case should be counted or not. In broad terms, this procedure is how all the other pharmas have run their vaccine trials.

The headline VE number of 70-80% counts only cases that are detected from 14 days after the second dose. This also is in line with current practice. I am not a fan of this maneuvre. You are forced to interpret the efficacy as a "conditional" metric. Infection rate is cut by 70% - but not for everyone, only if one does not fall ill between the first dose and 14 days after the 2nd dose.

This maneuvre is a spiritual cousin of how the Bureau of Labor Statistics computes unemployment rate. When the unemployment rate is 4 percent, one cannot infer that 96 percent of working-age people have jobs. It actually means 96 percent of the "labor force" have jobs. The labor force excludes people for many different reasons, such as anyone who has given up on the job market. So a proper way to interpret a 4 percent unemployment rate is as a conditional - 96% of people are employed only if they are classified as part of the work force. Conditional metrics can be highly misleading as it shoves a chunk of the population under the rug. (See this post on unemployment rate - or read Chapter 6 of Numbersense (link) for a deeper study).  

Similarly, a college might report that the average salary of its graduates is $80,000. That usually means it's an average of $80,000 only for those who have found jobs (and self-reported their salaries to the careers office). If you're a parent, and you want to know if your kid might earn $80,000 after graduating from this college, it depends on whether the kid finds a job. The $80,000 number provides zero information about the chance of graduates landing jobs. If the kid fails to find a job, the relevant salary number is 0.

Sinopharm also computed a variety of VE metrics, just like everyone else. If they had counted cases from first dose, the VE drops to 50-66%. If they had included asymptomatic cases (but counting only from 2D+14), the VE drops to 64-74%. None of this is surprising as we have seen this movie before in all the other trials.

The Chinese vaccine also claims 100% efficacy against severe Covid-19. As with the other trials, I find this claim absurd. None of these trials have enough people in them to measure the rare event of severe Covid-19. This Sinopharm claim is based on 2 cases in placebo versus 0 cases in the vaccine arms. Also available to the Chinese team is the other dubious statement that "no one who's been vaccinated died in the trial". I'm sure you've heard that one too about the U.S. trials. (No one died in the trial.)

Also as with other trials, subgroup analyses were computed robotically but nothing of value could have come out of them as sample sizes were too small.

In a nutshell, using the rules of the game that have been set by the regulatory agencies, the Sinopharm vaccine demonstrated similar "spectacular" levels of efficacy - even though it is based on old technology. If, as some people have argued, J&J's 67% efficacy should be considered equal to Pfizer and Moderna's, then this vaccine should be considered in the same class. (Same with Astrazeneca).

Differences

There is no obvious deviation from the protocols of the U.S. trials.

One minor difference is the Chinese trial tested two vaccines (developed from different strains of the virus) against placebo. When I stated the results in ranges, that's because the two vaccines showed slightly different numbers. Thus, there were three arms to the trial, each with equal number of participants (roughly 14,000 per arm).  

In one respect, the trial result is stronger in my view - the follow-up period is longer. The follow-up time for the median participant in the Chinese trial was described as 77 days - wait a minute, they defined that as 77 days from the start of the case-counting window, thus this is 91 days (3 months) from the second dose. Recall that the FDA set a rule that they will not approve any vaccines for which the data do not have a median follow-up period of two months - and that clock starts ticking from second dose. Eventually, the FDA approved two vaccines (Pfizer, Moderna) when the published data did not even meet the two-month rule; and for Johnson & Johnson, they condoned a flexible interpretation of two months as 8 weeks (56 days).  

I was very excited to find eFigure 2 in Supplement 3 of the JAMA report (which erroneously pointed readers to Supplement 2). This set of charts lays out exactly why the VE number computed as an interim analysis must be interpreted carefully.

Look at the enrollment timeline in Bahrain, from Aug to Oct 2020. In particular, consider someone who enrolled after late September, which was easily over a quarter of the participants. They took the second shot in the second half of October but the study does not count cases until 14 days later (Nov). The cutoff date of the analysis is end of December so this set of participants have at most 1.5 months of follow-up beyond the 2D+14 time point.

Now look at the cumulative case curve. This subset (marked by the brown box) can only influence the curve between 0 and 45 days. Meanwhile, those who enrolled at the start of the trial (blue box) took their second shot by mid September so their follow-up period lasted from late September to late December (up to 3 months).

It would have been fine if the timing of enrollment was randomized - that every weekly cohort of participants have the same characteristics. But this was not true of any of the trials. Pfizer added more minorities (higher risk) later in the trial. Astrazeneca started with younger and lower risk participants. The Chinese trial enrolled older people (60 years plus) later. None of these reports discussed possible biases related to staged enrollment, and none of them corrected for such biases.

You might be wondering how the scientists found data to draw out the case curve beyond 60 days, given that only a fraction of the Bahrain participants have reached that amount of follow-up. The answer is data from the UAE sites. In other words, the right side of the case curve contains predominantly data from UAE while the left side contains data from both UAE and Bahrain.

That is no small matter either. In the same eFigure 2, the researchers reveal that the case rates in UAE and Bahrain looked very different.

UAE has twice the number of cases (but 5 times the population) of Bahrain. Also cases were increasing during the study period in UAE while they were decreasing in Bahrain.

When I reviewed the previous vaccine trials, I was only able to speculate about these potential biases because those reports omitted this type of data. With this JAMA report, it's clear the biases are present, and I urge the other pharmas to publish similar data. Even Sinopharm can publish more. It's not enough to know the timing of enrollments; it's important to know the potential covariate imbalance between weekly cohorts.

***

I have split my notes into two parts. Part 2 of my review of the Sinopharm report will be posted this week.

Sizing up the pandemic

From the beginning, the U.S. has never gotten religion about accurately measuring the pandemic. In March 2020, I wrote a piece for Wired about the need for broad-based testing because selectively testing only severe cases is bound to give us a biased picture of the crisis. It was known even then that a significant portion of infections is asymptomatic - which constitutes a silent path by which the virus passes from person to person. The situation has not improved much since. Recent counting rule changes imposed selectively on vaccinated people have made it far worse but that's the subject of a different post.

I'm delighted to report on a valiant effort to correct this oversight - in Slovenia, a country of about 2 million people. The relevant papers have just been published here and here.

***

Because we want to minimize bias, randomization is our friend. I have previously explained how randomly assignment treatment in a clinical trial creates covariate balance (here) which allows researchers to assume "all else equal" except for the treatment.

In this case, we want to test a random sample of people selected from the population to avoid the self-selection bias that permeates the usual counts of "cases". The demographic and medical profiles of the sample should mimic that of the entire population. In the Slovenian study, they apply standard statistical sampling methodology, as follows: the country is divided into 300 census districts; from each district, they select 10 persons at random; a total of 3,000 Slovenians were sent communications inviting them to participate in this study; just under half agreed, leading to a sample size of 1,368.

Because of this random (i.e. broad-based) testing, the researchers uncovered not only self-reported symptomatic cases but also asymptomatic cases. All participants were given PCR tests at the start of the study in April, which was towards the end of the first wave in Slovenia. The estimated prevalence was 0.15%, or 150 per 100,000.

It's important to understand prevalence as a measure of infections at a point in time (late April-early May 2020 in this case). The PCR test reveals whether someone is currently or has recently been infected with the novel coronavirus. This metric does not answer the important epidemiological question: what proportion of the population has been infected since the start of the pandemic?

***

The research team's second paper deals with this question - the canonical approach is through serological testing i.e. testing for antibodies. Most people who have been sick with Covid-19 have developed antibodies. So we expect the proportion of population that have had Covid-19 to be larger than the proportion who are currently/recently infected. Technically, we say the seroprevalence is greater than the prevalence.

I have discussed serological testing before - at the start of the pandemic. Two research teams - one at Oxford and one at Stanford - jumped the gun when they published serological studies during the first wave of Covid-19. The Oxford team (in)famously declared that up to 40% of the U.K. population have already been infected with SAR-CoV-2 by April 2020 - a result that has proven extremely embarrassing. To be precise, they used a mathematical model to come up with the alarming headline, and called for serological studies. The Stanford team argued that total infections in California were 50 to 85 times the reported number of cases - back in April 2020. This group actually performed tests to derive their estimate. Their analysis was roundly criticized. Recent handwaving analyses tend to multiply cases by 4 times, which gives you an idea of the magnitude of the error in the Stanford study.

Meanwhile, I'm still expecting that one day, the New York State Department of Health will release the data behind the almost daily pronouncements by Governor Cuomo - again in spring and summer of 2020 - that at least 15% of New Yorkers have been infected. Those numbers were supposed to have come from antibody testing throughout the state of New York but no data or science has seen the light of day. While not as far-fetched as the two other studies, that estimate of seroprevalence is also clearly repudiated by subsequent waves of Covid-19.

In the Slovenian study, the unadjusted estimate of seroprevalence around mid-November 2020 (in the middle of a devastating second wave) was only 4 percent. At the time, the cumulative reported cases was around 27,000, or 1.3% of the population. This estimate is much more solid than the previous studies because these researchers used a random sample. The design of research studies matters a lot - when devices like randomization are utilized, fewer adjustments and assumptions are necessary to interpret the data. (See my three-part talk to learn more about research methods.)

***

Much of the second paper deals with the complications of the analysis even though the team utilized a random sample. The study interviewed participants at 3-week intervals. There were two rounds of blood samples (April 2020, and October 2020).

In April 2020, the reported cases (about 700) was 0.03% of the Slovenian population. With serological testing, they estimated seroprevalence to be roughly 0.8-0.9%, so roughly 18,000 people have been infected. More precisely, they estimated that 0.8-0.9% of their random sample have evidence of SARS-Cov-2 antibodies (that are true positives), and generalized that result to the entire population.

The raw data actually showed 3 percent, which was revised down to 0.8-0.9%. This is because of two factors that artificially inflated the unadjusted number.

The first is test inaccuracy. All tests are inaccurate to some degree - which is a key message of Chapter 4 of Numbers Rule Your World (link). In this situation, because the vast majority of Slovenians have not been infected, false-positive errors can be devastating to any analysis. Even a small false-postive error rate (e.g. 2 or 3 percent) gets amplified by the large base population into an abundance of incorrect positive test results. Correcting for test inaccuracy therefore pulls down the number of positive cases, and thus seroprevalence.

We have encountered this issue before - when pointing out the methodological weaknesses of the 2020 Stanford study. Their initial analysis did not correct for false-positive results. But it's actually a really hard problem. To know that a positive result is definitely a false positive implies that we know the true result - if we did, we would not need to run a test. In practice, the researchers must obtain an estimate of the probability of false positives from other studies. 

For example, they gather blood samples collected for other reasons - preferably from before first half of 2019 when they are sure the blood samples did not have any SARS-Cov-2 antibodies (or RNA, depending on what test). Then they apply the test to these blood samples, and check what proportion of them give positive results despite clearly being negative.

Such definitively negative blood samples are hard to procure and so our knowledge of test accuracy is limited. Many researchers just trust the test manufacturer's claims, which is a bit shocking since they market their tests, in essence, as 100% accurate. The Slovenian team did a literature search, and discovered that the accuracy of the antibody test they were using was not 100%, and the accuracy measures varied widely from study to study. Thus, the second factor the researchers adjusted for was the variability of test accuracy. (*)

***

Because of concerns about the first testing platform, when they analyzed the second set of blood samples In October 2020, they switched to a different testing platform, which appeared to have better and less variable accuracy statistics.

Despite random sampling, there may be non-response bias - the people who chose to participate in this study are not necessarily the same as those who opted out of the study. If the non-responders are different from responders, the result from this study cannot be generalized to the entire population without more corrections.

If you notice, this study is much more rigorous than the real-world vaccine effectiveness study that I have recently featured - those studies sweep most of the biases under the rug.

Nevertheless, the responders and non-responders look similar enough - by the variables they analyzed. It is therefore not surprising when the authors reported that a further correction for non-response bias did not materially change their seroprevalence estimates.

***

This Slovenian study is a good example of proper science that doesn't cut corners. What's notable is that seroprevalence grows at a much lower rate than reported cases. While cases have jumped almost 40 times during the six months of the study, seroprevalence has risen about 4 times (call it 10 times if we allow for the error bar).

 

P.S. [*] The study employed Bayesian methods throughout. Here is how they adjusted for variability of test accuracy. Different prior studies have come up with a range of estimates for test accuracy. If one doesn't adjust for this, one would take the average value from these studies, and apply this to the analysis. For example, if the test specificity (true negative rate) is 98%, then you assume exactly 2% of the negative blood samples would show up as false positives.

In a Bayesian model, the test specificity is treated as a variable: its average value is 98% but it can vary within some reasonable range (as informed by the variability observed across those prior studies).