Big Data, Plainly Spoken (aka Numbers Rule Your World)

Behind the proposed strengthening of FDA vaccine approval rules

News just broke that the FDA may announce more "rigorous" standards for a coronavirus vaccine to get approved. This indicates a willingness to listen to some of the critics of the perceived rushed job.

The specific rules mentioned in this Washington Post article are:

1) Participants must be followed for a median of at least two months, starting from when they receive the second dose

2) The analysis must be conducted after at least five severe cases of Covid19 have been observed in the placebo arm.

3) The analysis must be conducted after some minimum cases of Covid19 in older people has been recorded.

***

Rules 2 and 3 are fine in spirit. It's important that the vaccine works for severe cases and older people. These rules ensure that all trial analyses include the two groups that are of great concern to the medical community.

Recall that the case definition uses mild symptoms instead of severe symptoms. Since a lot more people develop mild symptoms, it's possible that the interim analysis contains only mild cases. It's also possible because of self-reporting that the cases used in the interim analysis involve only younger people.

I suspect those scenarios are unlikely anyway. Because testing is triggered by self-reporting, severe cases are more likely to surface than mild cases. So long as they have sufficient older participants in the trial, they should show up in the interim analysis because older people tend to have more severe illnesses.

Nevertheless, if the vaccine is approved by the end of 2020, it will be on the strength of an interim analysis. Such analyses are never designed to make statistically reliable statements about subgroups, such as older people or severe cases. This is another reason why the trial should run to completion even if the FDA grants interim approval.

***

Revised rule 1 is misleading and potentially worrying. 

Let's talk about its good intention first. Rule 1 sets up a second criterion for triggering the interim analysis. Previously, the only criterion is to exceed a prespecified total number of cases. This creates the potentially undesirable scenario in which enrollment is supercharged (either by pushing marketing, or relaxing admission standards), or the true case rate is markedly higher than assumed, such that the first interim analysis is triggered when most participants have barely taken the second dose.

By setting observation time as a second criterion, the new rule provides more confidence that the protection afforded by an approved vaccine lasts at least 2 months.

As I explained in this previous post, Moderna does not count cases until 2 weeks after the second dose. So if they track people for two months, it's really only 1.5 months of observation time. That isn't a lesser worry. What perturbs me is the use of the word "median".

Median is the middle person so the statement actually allows up to half of the participants to be observed for fewer than 2 months. Further, it is highly unusual and in fact statistically fallacious to perform analysis in which different participants are observed for different amounts of time. I hope this is a communications faux pas, a misunderstanding on the part of the journalist. The sensible rule is to require every participant (not the median participant) to be observed for at least 2 months (preferably longer) after the second dose.

 

Notes on the Moderna vaccine trial protocol 2

This is Part 2 of my notes about the Moderna vaccine trial. Read Part 1 here.

***

Durability of Protection

As described in the last post, the full analysis will not take place until middle of 2021, so the vaccine developers are hoping that their vaccine would show enough during the first or second interim analysis to accelerate the time line. Early reads increase the chance of false-positive findings, as I outlined in a previous post. To counter this, the study drastically lowers the p-value threshold to reach statistical significance in the interim analyses. (Instead of p = 0.05, the first interim analysis in effect sets p = 0.0002.) This is a proper thing to do.

However, if the first interim analysis results in a vaccine approval, then the participants will have been observed for at most 2.5 months from the second dose. We will not know if the vaccine offers protection beyond 2.5 months.

According to the protocol, Moderna "intends" to continue the study after if it receives interim approval. There will be an ethical debate about whether those who received placebos should be given the vaccine should it win approval. Some may oppose withholding a "proven" treatment from these participants. If they are given the vaccine, we lose the ability to measure protection beyond 2.5 months.

Sample Size

If the first interim analysis is successful, then the vaccine would receive approval based on about 50 total infections. Assuming that the vaccine effectiveness is 50%, then we expect half the number of infections on the vaccine arm relative to the placebo arm. So the vaccine arm should account for one-third, or about 17 infections. That's an awfully small number.

Nevertheless, if approved, the vaccine would have met the more stringent statistical significance threshold so the small sample size is not a serious problem. But the required number at the first interim analysis is small enough as to make many statisticians nervous. Continued tracking beyond the first analysis will be key to make sure the early result is not a mirage.

Defining Trial Outcomes

In the primary analysis, the "endpoint" (outcome) is infection, which requires a PCR test confirmation plus some pre-specified level of symptoms.

Some experts are arguing that they should count severe cases only because we don't really need to stop the mild cases. The chosen endpoint includes both mild and severe cases but we expect that most of the cases will be mild so there is concern that the signal (severe) to noise (mild) ratio is not high. Of course, if only severe cases are counted, the trial would last longer.

The opposite criticism is that they are by and large missing asymptomatic cases because almost all the testing are triggered by symptom reporting. One can argue that we need to stop asymptomatic spread because any viral transmission may eventually produce severe cases.

Moderna is taking the middle ground here.

Unusual Sources of Noise

In running any experiment, we'd love to eliminate as many sources of noise (variability) as possible so that any observed differences can be attributed to the vaccine. Some practical issues for this vaccine trial may produce noise, which affects our interpretation of the results.

The FDA permits accommodations due to the pandemic. This means that in-person visits may be replaced by phone calls. That itself is not a problem if all visits are converted to phone calls. The problem is that swaps occur based on request so they'd end up with some in-person visits and some phone calls. This creates a possibility that the method of the follow-up may be associated with, or even cause, different outcomes.

Besides, the protocol lists a variety of tests used to confirm infection. The nasal swab is preferred but saliva and respiratory samples are also used. These tests have different accuracies and characteristics so the mix of test types may affect the count of confirmed cases. For example, people living in high-risk areas are more likely to submit saliva samples by mail versus nasal swabs from in-person visits.

Uncertainty Around the Case Rate

The design assumes a 0.75% baseline case rate within six months but the researchers acknowledge uncertainty around this number - that's why they end the enrollment based on the number of cases rather than the number of participants. What if this assumption is wrong?

In particular, this base rate seems fairly low. Those who hope for "herd immunity" might be shocked to learn that the virus is expected to infect less than 1 percent of a community over six months. At that sluggish rate, when will we reach 70 percent or so prevalence?

All sample sizing requires guessing at the base rate, and sometimes we can get it wrong. If the base rate is somewhat higher than 0.75%, then the trial will get infections faster than expected, and thus end enrollment earlier than expected. The real case rate among those participants can be, say, 2%.

Here's the rub. If 2 percent had been used as the base rate in the design of the clinical trial, statisticians would have stipulated more than 53 infections to trigger the first interim analysis. This means, if the observed case rate is higher than expected, the first interim analysis suffers from insufficient data. Smaller-than-expected sample size makes it more challenging to beat the significance threshold.

***

The Moderna protocol is reasonable: there are some unconventional elements, but in an emergency, we'd have to accept some accommodations. It's still hard to imagine how any of these trials will get to a strong result by end of October or November.

News about the vaccine trials is coming fast. Just this weekend, the Washington Post reported that the FDA may tighten the vaccine approval requirements a little to pacify critics and build trust. I have another post up discussing these modifications.

 

 

 

 

 

Good news from Cornell

Source: Redbubble.com

Some colleges and universities have re-opened for in-person classes for the fall term. Previously, I examined the Cornell model (link), which outlines a very aggressive plan for preventing the virus from entering the campus, and then snipping transmission chains during the semester. Also, we learned that several schools that opened in August quickly ran into trouble as hundreds if not thousands of cases surfaced, and some decided to return to online teaching. (link)

I just attended a fantastic webinar by Peter Frazier, who's part of the Cornell team that did the statistical modeling work supporting the re-opening. The webinar was sponsored by Columbia.

It's about three weeks after Cornell reopened for in-person instruction, and the situation in Ithaca is significantly better than on most campuses. The following chart best illustrates why the team is taking a victory lap, although they absolutely realize that there are still risks ahead.

[Applause]

[Note: I expunged a number of Excel defaults from the above chart, including line colors, legend, and date labels. Also, replaced infections with cases, which is a more precise description of the metric.]

As I indicated in the previous post, if Cornell's plan proves successful, it would repudiate what the U.S. government did in the early phase of the pandemic. It shows that extremely aggressive surveillance - and in particular, broad-based testing - will dampen case rate to around zero, and keep it there. That's what other countries like China and South Korea managed to do.

At the end of the talk, Prof. Frazier offered advice to data scientists who want to make real-world impact. I agree with every bullet point and am reiterating the whole list here.

1. Focus on solving the real-world problem, rather than what can be published. He discloses that they have not written a paper about the work they did here. This is advice I give to researchers who want to transition into industry positions.

2. Create mental models of your computer models. A different way of expressing this point is that the modeler should develop some intuition behind how the models work, even if the methodologies are complex and opaque.

3. Communications. He spent a lot of time explaining the model to the public, and the team was open to feedback and continuously listening to the community. People are emotional about the subject, and some of the commentary were harsh but it's part of the job. Frazier's presentation today is impressive evidence of his communications skills.

4. Understand the political environment. Different groups have different objectives and biases.

5. He calls this "luck." The key decision-makers at Cornell are scientists by training, so they have an easier time persuading them to believe in science. That's an important point when it comes to "choosing your boss." For many data scientists, jobs have been plentiful, and job satisfaction will depend on whether you're fighting with your boss all the time. Imagine the scientists at FDA or CDC who had to deal with political appointees who are reportedly pressuring them to alter their language. Those meetings would be a completely different beast than what the Cornell team had to deal with.

Throughout the presentation, Prof. Frazier pointed to other "luck" factors that favor Cornell's response. For example, Ithaca is very isolated. He said that the nearest towns are an hour's drive away. Also, their veterinary school has abundant expertise in PCR testing, and has been extremely helpful.

He also reported that pessimists were much more vocal than optimists, and so assumptions may have been driven more conservative than warranted. Modelers may also have a conservative bias because better safe than sorry.

***

In his presentation, Prof. Frazier demonstrated the power of statistical modeling. The core simulation model is highly complex, with many moving parts and assumptions, as I discussed in the earlier post. The assumptions are necessary because most of those parameters are not directly observable, and highly uncertain. While the model was not "accurate" in the sense of forecasting precise outcomes, it accurately predicted the general trends. Making assumptions is not a bad thing!

Then, we saw Occam's razor in action. Statisticians have long preached that models should be as simple as possible, but not simpler. The core model is actually built on top of simple epidemiological models, and those models alone were able to capture the general trends.

***

Since the time I reviewed the Cornell plan, there were a couple of important additions that probably contributed to the on-the-ground situation being better than the original projection.

First, the frequency of surveillance testing was doubled for undergraduates from once a week to twice a week. The insight is to tailor the testing rate to the amount of contacts expected in different subgroups. The mathematical models help calibrate what level of testing is required to keep the infection rate below a critical threshold.

Second, to work around the limited capacity for contact tracing, and to take advantage of their expertise in PCR testing (via the vet school), they are testing the entire social circle for each positive case. They call this "adaptive testing". The goal is to reveal as many hidden cases as possible before the virus has a chance to spread further.

Testing is a huge commitment in terms of money, resources and time. Cornell is conducting over 5,000 tests per day - obviously, they are returning results very fast. It appears also that the students are cooperating, and grasping the key issues. This pandemic is a test of community spirit.

***

Perhaps lost in our national conversation is the decisiveness of early, aggressive action. Public-health experts have stressed this point from the start, and the Cornell experience (relative to other universities) shows they were right.

Take compliance to testing as an example. When there isn't much virus in circulation, we can accommodate some degree of non-compliance because the non-compliant person will rarely run into an infectious contact. When the epidemic is in full swing, the chance of interfacing with infectious people is much higher, and non-compliant members are more likely to spread the virus around.

We can't think linearly. Any community is moving toward one of two final states: almost no virus in the community, or almost everyone gets infected at which point the virus also spreads slowly because most contacts are between infected people. Without intervention, the second state is inevitable. Once an epidemic gets out of control, it's very hard to revert to the first state.

Cornell's success to this point is showing us that places like China and South Korea can indeed eliminate the virus with early aggressive actions. These countries don't have to fake the data if they did the right things.

Whether the success will last is an open question. It's only three weeks into the term. Prof. Frazier noted that they strongly discourage students from traveling outside Ithaca. I'm not sure this is possible during breaks and Thanksgiving. Similarly, countries that successfully fought off the first wave still face risks if the virus is re-imported from other countries that failed in their initial response. For Cornell, I'm guessing that they will have to roll out arrival testing after each long break.

Notes on the Moderna vaccine trial protocol 1

Bending to outside pressure, Moderna has made public one of the documents submitted to the FDA that provide more details on how they are running the vaccine trial. (PDF here) This is a reassuring move, and I applaud them for taking this step.

In a previous post, I raised two key questions I would like to find answers to, and after reading all 135 pages of the protocol, I can tell you the answers.

How is efficacy measured?

Vaccine effectiveness (technically, efficacy) is the reduction in infection rate among the vaccinated relative to the baseline infection rate among the unvaccinated. A 50% effective vaccine is one that halves the baseline infection rate. (For more on vaccine effectiveness, see this other post.)

An important detail is how this infection rate is measured. According to the trial design documentation, Moderna will confirm infection using PCR nasal swab tests (primarily). To be precise, what they measure is confirmed cases rather than infections.

What is the analysis schedule?

Until the documentation is made public, there has been a lot of sloppy reporting about when results will be available. Now we know. The primary analysis will be conducted when the total number of confirmed cases (across both test and placebo arms) exceeds 151. The expected time of this analysis is six and a half months after the last trial participant receives the second dose. By my estimate, this will be mid June 2021.

The trial started enrolling participants at the start of August. Assume that enrollment is complete by the end of September, the second dose of the last participant should occur at the end of October, the data for the primary analysis will be frozen by mid May 2021, and the trial result will be ready by mid June 2021.

Why do pharma CEOs keep saying they expect results by the end of 2020? That's because they are hoping one of two interim analyses might produce results so stellar that the FDA will have seen enough. The first interim analysis is triggered when the number of confirmed cases exceeds 53, which is 35 percent of the full sample. If it takes 6.5 months to accumulate 151 cases, it would be 2.5 months to get to 53. (In both instances, the half month is added because they don't start counting cases until 14 days after the second dose.)

For the first interim analysis, assume they only analyze participants through the end of August, the second doses should be completed by the end of September, the data will be frozen by mid December 2020, and so the earliest date by which the vaccine might be declared effective is mid January 2021, if everything else falls into place.

***

What to Look Out For

I read through the entire 135-page document. I prepared a list of notes about the clinical trial, mostly about statistical matters. These are placeholders for now. How much these issues matter depends on what happens during the trial.

Blinding

The trial is observer-blind, less than the gold standard of double-blind. Blinding is hiding whether a participant has gotten the vaccine (or drug) or the placebo. Double-blind means neither the researchers nor the participants are told who got what shot, until the data are ready for analysis. Blinding prevents humans from misbehaving as a result of knowing the treatment assignment. In this Moderna trial, the person administering the shots is not blinded. There may be a scientific reason for why some researchers cannot be blinded for this study but I didn't find an explanation in the documentation. (I checked the Pfizer-Biontech trial, and it is also observer-blind, not double-blind.)

Many outside scientists and reporters have missed this detail, and mis-reported the Moderna trial as double-blind (e.g. here, here, here).

Analysis Population

The primary analysis is "per-protocol", which is less than the gold standard of intent to treat. This is always a controversial subject so let me explain the disagreement. In a canonical clinical trial, half the participants get the vaccine (or drug), and the other half receive the placebo (or comparator). Researchers take great care to randomize assignment to these two arms. If each participant has equal chance of being assigned to either arm, then we have created the condition of "all else being equal" so that when we compare the average person across the two arms, any difference can be attributed to the vaccine (or drug).

In practice, some participants will drop out of a trial. It is quite possible that the dropout rate is not the same across the two arms. For example, if the treatment causes irritation, more people in the treatment arm might quit. The statistical debate is whether to include the dropouts in the analysis. The gold standard of intent-to-treat analysis says to stick as close to the original population as possible. Dropouts are therefore included (up to the time of dropout). In a per-protocol analysis, the dropouts may be excluded as if they never existed. The proponents of per-protocol argue that the final outcomes of these dropouts are not known, and so including them might be "unfair" to the treatment arm.

By contrast, supporters of intent-to-treat (including me) don't buy that the act of dropping out is independent of treatment, and worry that the loss of randomization invalidates the "all else equal" condition. For me, the most important argument is the practical point of view: the per-protocol findings apply only to people who take the treatment without dropping out!

Dropouts are not the only subset of participants who might be excluded in a per-protocol analysis. It's just easier to explain.

Back to the Moderna trial. It appears that dropouts prior to the second dose will be excluded from the per-protocol analysis but those who discontinue after the second dose will be included.

Infections Before Finishing Treatment

In the Moderna trial, another interesting subset consists of participants who get infected and then are excluded from the per-protocol analysis. In the primary analysis, Moderna starts counting infections 14 days after the participants get their second dose.

The trial design stipulates that people who have confirmed infection before the second dose will be ineligible for the second dose. All such people therefore will not complete the protocol and be excluded from the per-protocol analysis. The good news is that Moderna has committed to performing an intent-to-treat analysis for comparison. However, the final decision will be based on per-protocol, and there is no guarantee that the intent-to-treat analysis will be published for outside evaluation.

Let's say we're behind at the end of a(n American) football match, and our coach decides to go for an on-side kick and then drive for a field goal. To win the game requires executing both steps. If the on-side kick fails, the other team will run out the clock. (Last year, on-side kick success rate in the NFL was only 6 percent.) If the on-side kick fails, the per-protocol analysis excludes this match because the team didn't execute the entire plan. If the on-side kick is successful and the team gets into field-goal position, the per-protocol analysis includes this match. So, the estimate of the success rate of the two-step plan will be over-optimistic.

Here's the practical point of view. If the vaccine requires two doses, and someone gets sick after the first dose, the vaccine is ineffective for that individual. It doesn't matter that the said individual has not received the second dose; it is meaningless to administer the second dose.

Excluded Infections

Then, there is the subset of participants who get infected during the first 14 days after the second dose. For an unexplained reason, Moderna isn't counting infections until two weeks after the second dose.

This subset will be included in the per-protocol analysis because they took the full two-dose program. However, the outcome for these participants will be "censored," meaning instead of being treated as infected, they are treated as if they have dropped out of the study just before their infections are confirmed.

Again, I take a practical view on these matters. If, after the vaccine is approved, someone gets infected a few days after receiving the second dose, the vaccine would have failed for this individual.

So when the results come out, keep an eye on whether there is a differential likelihood to get early infections on the vaccine arm relative to the placebo arm. The good news is Moderna has committed to conducting additional analysis that counts infections from the first day after the second dose for comparison. However, the final decision does not have to take into account this analysis, and it may not be publicly available.

Counting Infections

Measuring infections in a vaccine trial is always challenging because the participants are healthy. Contrast this with a clinical trial testing a treatment on sick people. Since it's unethical to inject virus into healthy subjects, we have to wait for infections to crop up via community spread.

Moderna isn't very proactive in monitoring for infections. They rely on self-reported symptoms, which then trigger confirmatory PCR tests. For example, the only way researchers will learn about suspected infections during the first 14 days after the second dose is if the participants report symptoms. Not everyone who report symptoms will get tested; the researchers have some discretion for certain types of symptoms.

That's why we will never know the true infection rate. The reported rate should be considered the rate of confirmed cases.

***

These are the most salient information I learned from the just-released documentation on the Moderna vaccine trial. I have several more notes that will be written up in my next post.