Big Data, Plainly Spoken (aka Numbers Rule Your World)

Two side issues as we hurry to a coronavirus vaccine

My review of the Moderna trial protocol (parts 1, 2) shows that by and large, the design follows established standards, with only a few places where the researchers have selected less than the gold standard, which is understandable given the urgency of finding a vaccine.

That said, a rushed job with the desired outcome of a vaccine is going to increase the small odds of a false-positive result. This may come about in unconventional ways.

***

Within each trial, as I pointed out in part 2 of the Moderna protocol review, the threshold for statistical significance has been made drastically more rigorous to account for the interim analysis based on a much smaller sample of data. That's the proper thing to do. The design is generated so that the overall probability of a false-positive across all attempted analyses and endpoints is under control.

Of note, the world is running a dozen or more trials of competing vaccines at the same time. Even when each trial controls the chance of a false positive, the overall probability of a false positive coming from any of the slate of trials is not under control.

Think about elevator maintenance. For any single elevator, the manufacturer might promise a breakdown rate of below one percent. Now, across all elevators of all brands in a single town, the chance that at least one elevator breaks down in a period of time is much higher than one percent. That's because each elevator has a one-percent chance of breaking down, and any elevator may break down at any moment in time. The more elevators there are in the town, the higher the chance of at least one breakdown.

The more vaccine trials are started, the higher the chance that a positive result is a false alarm. No individual clinical trial is controlling for this kind of risk; only the FDA can address this.

***

Another rarely addressed issue is statistical validity, that is to say, how representative are the participants of the trial relative to the people who are likely to receive the vaccine once approved.

The vaccine manufacturers are engaged in a complex competition. The first vaccine to get approved has a special status because it becomes the standard of care. Other trials may have to use the first vaccine as the comparator, rather than the placebo. Because of the provision for interim analysis, the sooner a trial completes enrollment of participants, the sooner it can begin analysis, and the approval application. Interim analyses are not triggered by the total enrollment, as in most clinical trials, but by the total number of infections.

Thus, the incentive is to get as many people enrolled as quickly as possible, and to dip into higher-risk areas as much as possible. In the extreme situation, if all enrollment is focused on only a few highest-risk areas, the trial may become the first to accumulate enough case targets. (A possible dent in this strategy is if people's willingness to participate is inversely proportional to how much risk they face.)

While this competition plays out, the trials that have the most biased samples, concentrating on the regions with the highest "yields", will likely win the race for time. This creates a problem for the FDA because the results would score lower on statistical validity, and have to be de-biased. (Such a sample would however provide better data on higher-risk subgroups. The trade-off is that the vaccine is supposed to be for everyone.)

Perhaps the enrollment can be skewed towards severe cases at the start, and then broadened over time to encapsulate the general population. This mimics the likely plan for rolling out the vaccine.

***

I should underline that these are not show-stopping issues. They are interesting statistical side-notes that may seem theoretical when you read about them in a textbook -- until the pandemic happens.

Behind the proposed strengthening of FDA vaccine approval rules

News just broke that the FDA may announce more "rigorous" standards for a coronavirus vaccine to get approved. This indicates a willingness to listen to some of the critics of the perceived rushed job.

The specific rules mentioned in this Washington Post article are:

1) Participants must be followed for a median of at least two months, starting from when they receive the second dose

2) The analysis must be conducted after at least five severe cases of Covid19 have been observed in the placebo arm.

3) The analysis must be conducted after some minimum cases of Covid19 in older people has been recorded.

***

Rules 2 and 3 are fine in spirit. It's important that the vaccine works for severe cases and older people. These rules ensure that all trial analyses include the two groups that are of great concern to the medical community.

Recall that the case definition uses mild symptoms instead of severe symptoms. Since a lot more people develop mild symptoms, it's possible that the interim analysis contains only mild cases. It's also possible because of self-reporting that the cases used in the interim analysis involve only younger people.

I suspect those scenarios are unlikely anyway. Because testing is triggered by self-reporting, severe cases are more likely to surface than mild cases. So long as they have sufficient older participants in the trial, they should show up in the interim analysis because older people tend to have more severe illnesses.

Nevertheless, if the vaccine is approved by the end of 2020, it will be on the strength of an interim analysis. Such analyses are never designed to make statistically reliable statements about subgroups, such as older people or severe cases. This is another reason why the trial should run to completion even if the FDA grants interim approval.

***

Revised rule 1 is misleading and potentially worrying. 

Let's talk about its good intention first. Rule 1 sets up a second criterion for triggering the interim analysis. Previously, the only criterion is to exceed a prespecified total number of cases. This creates the potentially undesirable scenario in which enrollment is supercharged (either by pushing marketing, or relaxing admission standards), or the true case rate is markedly higher than assumed, such that the first interim analysis is triggered when most participants have barely taken the second dose.

By setting observation time as a second criterion, the new rule provides more confidence that the protection afforded by an approved vaccine lasts at least 2 months.

As I explained in this previous post, Moderna does not count cases until 2 weeks after the second dose. So if they track people for two months, it's really only 1.5 months of observation time. That isn't a lesser worry. What perturbs me is the use of the word "median".

Median is the middle person so the statement actually allows up to half of the participants to be observed for fewer than 2 months. Further, it is highly unusual and in fact statistically fallacious to perform analysis in which different participants are observed for different amounts of time. I hope this is a communications faux pas, a misunderstanding on the part of the journalist. The sensible rule is to require every participant (not the median participant) to be observed for at least 2 months (preferably longer) after the second dose.

 

Notes on the Moderna vaccine trial protocol 2

This is Part 2 of my notes about the Moderna vaccine trial. Read Part 1 here.

***

Durability of Protection

As described in the last post, the full analysis will not take place until middle of 2021, so the vaccine developers are hoping that their vaccine would show enough during the first or second interim analysis to accelerate the time line. Early reads increase the chance of false-positive findings, as I outlined in a previous post. To counter this, the study drastically lowers the p-value threshold to reach statistical significance in the interim analyses. (Instead of p = 0.05, the first interim analysis in effect sets p = 0.0002.) This is a proper thing to do.

However, if the first interim analysis results in a vaccine approval, then the participants will have been observed for at most 2.5 months from the second dose. We will not know if the vaccine offers protection beyond 2.5 months.

According to the protocol, Moderna "intends" to continue the study after if it receives interim approval. There will be an ethical debate about whether those who received placebos should be given the vaccine should it win approval. Some may oppose withholding a "proven" treatment from these participants. If they are given the vaccine, we lose the ability to measure protection beyond 2.5 months.

Sample Size

If the first interim analysis is successful, then the vaccine would receive approval based on about 50 total infections. Assuming that the vaccine effectiveness is 50%, then we expect half the number of infections on the vaccine arm relative to the placebo arm. So the vaccine arm should account for one-third, or about 17 infections. That's an awfully small number.

Nevertheless, if approved, the vaccine would have met the more stringent statistical significance threshold so the small sample size is not a serious problem. But the required number at the first interim analysis is small enough as to make many statisticians nervous. Continued tracking beyond the first analysis will be key to make sure the early result is not a mirage.

Defining Trial Outcomes

In the primary analysis, the "endpoint" (outcome) is infection, which requires a PCR test confirmation plus some pre-specified level of symptoms.

Some experts are arguing that they should count severe cases only because we don't really need to stop the mild cases. The chosen endpoint includes both mild and severe cases but we expect that most of the cases will be mild so there is concern that the signal (severe) to noise (mild) ratio is not high. Of course, if only severe cases are counted, the trial would last longer.

The opposite criticism is that they are by and large missing asymptomatic cases because almost all the testing are triggered by symptom reporting. One can argue that we need to stop asymptomatic spread because any viral transmission may eventually produce severe cases.

Moderna is taking the middle ground here.

Unusual Sources of Noise

In running any experiment, we'd love to eliminate as many sources of noise (variability) as possible so that any observed differences can be attributed to the vaccine. Some practical issues for this vaccine trial may produce noise, which affects our interpretation of the results.

The FDA permits accommodations due to the pandemic. This means that in-person visits may be replaced by phone calls. That itself is not a problem if all visits are converted to phone calls. The problem is that swaps occur based on request so they'd end up with some in-person visits and some phone calls. This creates a possibility that the method of the follow-up may be associated with, or even cause, different outcomes.

Besides, the protocol lists a variety of tests used to confirm infection. The nasal swab is preferred but saliva and respiratory samples are also used. These tests have different accuracies and characteristics so the mix of test types may affect the count of confirmed cases. For example, people living in high-risk areas are more likely to submit saliva samples by mail versus nasal swabs from in-person visits.

Uncertainty Around the Case Rate

The design assumes a 0.75% baseline case rate within six months but the researchers acknowledge uncertainty around this number - that's why they end the enrollment based on the number of cases rather than the number of participants. What if this assumption is wrong?

In particular, this base rate seems fairly low. Those who hope for "herd immunity" might be shocked to learn that the virus is expected to infect less than 1 percent of a community over six months. At that sluggish rate, when will we reach 70 percent or so prevalence?

All sample sizing requires guessing at the base rate, and sometimes we can get it wrong. If the base rate is somewhat higher than 0.75%, then the trial will get infections faster than expected, and thus end enrollment earlier than expected. The real case rate among those participants can be, say, 2%.

Here's the rub. If 2 percent had been used as the base rate in the design of the clinical trial, statisticians would have stipulated more than 53 infections to trigger the first interim analysis. This means, if the observed case rate is higher than expected, the first interim analysis suffers from insufficient data. Smaller-than-expected sample size makes it more challenging to beat the significance threshold.

***

The Moderna protocol is reasonable: there are some unconventional elements, but in an emergency, we'd have to accept some accommodations. It's still hard to imagine how any of these trials will get to a strong result by end of October or November.

News about the vaccine trials is coming fast. Just this weekend, the Washington Post reported that the FDA may tighten the vaccine approval requirements a little to pacify critics and build trust. I have another post up discussing these modifications.

 

 

 

 

 

Good news from Cornell

Source: Redbubble.com

Some colleges and universities have re-opened for in-person classes for the fall term. Previously, I examined the Cornell model (link), which outlines a very aggressive plan for preventing the virus from entering the campus, and then snipping transmission chains during the semester. Also, we learned that several schools that opened in August quickly ran into trouble as hundreds if not thousands of cases surfaced, and some decided to return to online teaching. (link)

I just attended a fantastic webinar by Peter Frazier, who's part of the Cornell team that did the statistical modeling work supporting the re-opening. The webinar was sponsored by Columbia.

It's about three weeks after Cornell reopened for in-person instruction, and the situation in Ithaca is significantly better than on most campuses. The following chart best illustrates why the team is taking a victory lap, although they absolutely realize that there are still risks ahead.

[Applause]

[Note: I expunged a number of Excel defaults from the above chart, including line colors, legend, and date labels. Also, replaced infections with cases, which is a more precise description of the metric.]

As I indicated in the previous post, if Cornell's plan proves successful, it would repudiate what the U.S. government did in the early phase of the pandemic. It shows that extremely aggressive surveillance - and in particular, broad-based testing - will dampen case rate to around zero, and keep it there. That's what other countries like China and South Korea managed to do.

At the end of the talk, Prof. Frazier offered advice to data scientists who want to make real-world impact. I agree with every bullet point and am reiterating the whole list here.

1. Focus on solving the real-world problem, rather than what can be published. He discloses that they have not written a paper about the work they did here. This is advice I give to researchers who want to transition into industry positions.

2. Create mental models of your computer models. A different way of expressing this point is that the modeler should develop some intuition behind how the models work, even if the methodologies are complex and opaque.

3. Communications. He spent a lot of time explaining the model to the public, and the team was open to feedback and continuously listening to the community. People are emotional about the subject, and some of the commentary were harsh but it's part of the job. Frazier's presentation today is impressive evidence of his communications skills.

4. Understand the political environment. Different groups have different objectives and biases.

5. He calls this "luck." The key decision-makers at Cornell are scientists by training, so they have an easier time persuading them to believe in science. That's an important point when it comes to "choosing your boss." For many data scientists, jobs have been plentiful, and job satisfaction will depend on whether you're fighting with your boss all the time. Imagine the scientists at FDA or CDC who had to deal with political appointees who are reportedly pressuring them to alter their language. Those meetings would be a completely different beast than what the Cornell team had to deal with.

Throughout the presentation, Prof. Frazier pointed to other "luck" factors that favor Cornell's response. For example, Ithaca is very isolated. He said that the nearest towns are an hour's drive away. Also, their veterinary school has abundant expertise in PCR testing, and has been extremely helpful.

He also reported that pessimists were much more vocal than optimists, and so assumptions may have been driven more conservative than warranted. Modelers may also have a conservative bias because better safe than sorry.

***

In his presentation, Prof. Frazier demonstrated the power of statistical modeling. The core simulation model is highly complex, with many moving parts and assumptions, as I discussed in the earlier post. The assumptions are necessary because most of those parameters are not directly observable, and highly uncertain. While the model was not "accurate" in the sense of forecasting precise outcomes, it accurately predicted the general trends. Making assumptions is not a bad thing!

Then, we saw Occam's razor in action. Statisticians have long preached that models should be as simple as possible, but not simpler. The core model is actually built on top of simple epidemiological models, and those models alone were able to capture the general trends.

***

Since the time I reviewed the Cornell plan, there were a couple of important additions that probably contributed to the on-the-ground situation being better than the original projection.

First, the frequency of surveillance testing was doubled for undergraduates from once a week to twice a week. The insight is to tailor the testing rate to the amount of contacts expected in different subgroups. The mathematical models help calibrate what level of testing is required to keep the infection rate below a critical threshold.

Second, to work around the limited capacity for contact tracing, and to take advantage of their expertise in PCR testing (via the vet school), they are testing the entire social circle for each positive case. They call this "adaptive testing". The goal is to reveal as many hidden cases as possible before the virus has a chance to spread further.

Testing is a huge commitment in terms of money, resources and time. Cornell is conducting over 5,000 tests per day - obviously, they are returning results very fast. It appears also that the students are cooperating, and grasping the key issues. This pandemic is a test of community spirit.

***

Perhaps lost in our national conversation is the decisiveness of early, aggressive action. Public-health experts have stressed this point from the start, and the Cornell experience (relative to other universities) shows they were right.

Take compliance to testing as an example. When there isn't much virus in circulation, we can accommodate some degree of non-compliance because the non-compliant person will rarely run into an infectious contact. When the epidemic is in full swing, the chance of interfacing with infectious people is much higher, and non-compliant members are more likely to spread the virus around.

We can't think linearly. Any community is moving toward one of two final states: almost no virus in the community, or almost everyone gets infected at which point the virus also spreads slowly because most contacts are between infected people. Without intervention, the second state is inevitable. Once an epidemic gets out of control, it's very hard to revert to the first state.

Cornell's success to this point is showing us that places like China and South Korea can indeed eliminate the virus with early aggressive actions. These countries don't have to fake the data if they did the right things.

Whether the success will last is an open question. It's only three weeks into the term. Prof. Frazier noted that they strongly discourage students from traveling outside Ithaca. I'm not sure this is possible during breaks and Thanksgiving. Similarly, countries that successfully fought off the first wave still face risks if the virus is re-imported from other countries that failed in their initial response. For Cornell, I'm guessing that they will have to roll out arrival testing after each long break.