Big Data, Plainly Spoken (aka Numbers Rule Your World)

Good news from Cornell

Source: Redbubble.com

Some colleges and universities have re-opened for in-person classes for the fall term. Previously, I examined the Cornell model (link), which outlines a very aggressive plan for preventing the virus from entering the campus, and then snipping transmission chains during the semester. Also, we learned that several schools that opened in August quickly ran into trouble as hundreds if not thousands of cases surfaced, and some decided to return to online teaching. (link)

I just attended a fantastic webinar by Peter Frazier, who's part of the Cornell team that did the statistical modeling work supporting the re-opening. The webinar was sponsored by Columbia.

It's about three weeks after Cornell reopened for in-person instruction, and the situation in Ithaca is significantly better than on most campuses. The following chart best illustrates why the team is taking a victory lap, although they absolutely realize that there are still risks ahead.

[Applause]

[Note: I expunged a number of Excel defaults from the above chart, including line colors, legend, and date labels. Also, replaced infections with cases, which is a more precise description of the metric.]

As I indicated in the previous post, if Cornell's plan proves successful, it would repudiate what the U.S. government did in the early phase of the pandemic. It shows that extremely aggressive surveillance - and in particular, broad-based testing - will dampen case rate to around zero, and keep it there. That's what other countries like China and South Korea managed to do.

At the end of the talk, Prof. Frazier offered advice to data scientists who want to make real-world impact. I agree with every bullet point and am reiterating the whole list here.

1. Focus on solving the real-world problem, rather than what can be published. He discloses that they have not written a paper about the work they did here. This is advice I give to researchers who want to transition into industry positions.

2. Create mental models of your computer models. A different way of expressing this point is that the modeler should develop some intuition behind how the models work, even if the methodologies are complex and opaque.

3. Communications. He spent a lot of time explaining the model to the public, and the team was open to feedback and continuously listening to the community. People are emotional about the subject, and some of the commentary were harsh but it's part of the job. Frazier's presentation today is impressive evidence of his communications skills.

4. Understand the political environment. Different groups have different objectives and biases.

5. He calls this "luck." The key decision-makers at Cornell are scientists by training, so they have an easier time persuading them to believe in science. That's an important point when it comes to "choosing your boss." For many data scientists, jobs have been plentiful, and job satisfaction will depend on whether you're fighting with your boss all the time. Imagine the scientists at FDA or CDC who had to deal with political appointees who are reportedly pressuring them to alter their language. Those meetings would be a completely different beast than what the Cornell team had to deal with.

Throughout the presentation, Prof. Frazier pointed to other "luck" factors that favor Cornell's response. For example, Ithaca is very isolated. He said that the nearest towns are an hour's drive away. Also, their veterinary school has abundant expertise in PCR testing, and has been extremely helpful.

He also reported that pessimists were much more vocal than optimists, and so assumptions may have been driven more conservative than warranted. Modelers may also have a conservative bias because better safe than sorry.

***

In his presentation, Prof. Frazier demonstrated the power of statistical modeling. The core simulation model is highly complex, with many moving parts and assumptions, as I discussed in the earlier post. The assumptions are necessary because most of those parameters are not directly observable, and highly uncertain. While the model was not "accurate" in the sense of forecasting precise outcomes, it accurately predicted the general trends. Making assumptions is not a bad thing!

Then, we saw Occam's razor in action. Statisticians have long preached that models should be as simple as possible, but not simpler. The core model is actually built on top of simple epidemiological models, and those models alone were able to capture the general trends.

***

Since the time I reviewed the Cornell plan, there were a couple of important additions that probably contributed to the on-the-ground situation being better than the original projection.

First, the frequency of surveillance testing was doubled for undergraduates from once a week to twice a week. The insight is to tailor the testing rate to the amount of contacts expected in different subgroups. The mathematical models help calibrate what level of testing is required to keep the infection rate below a critical threshold.

Second, to work around the limited capacity for contact tracing, and to take advantage of their expertise in PCR testing (via the vet school), they are testing the entire social circle for each positive case. They call this "adaptive testing". The goal is to reveal as many hidden cases as possible before the virus has a chance to spread further.

Testing is a huge commitment in terms of money, resources and time. Cornell is conducting over 5,000 tests per day - obviously, they are returning results very fast. It appears also that the students are cooperating, and grasping the key issues. This pandemic is a test of community spirit.

***

Perhaps lost in our national conversation is the decisiveness of early, aggressive action. Public-health experts have stressed this point from the start, and the Cornell experience (relative to other universities) shows they were right.

Take compliance to testing as an example. When there isn't much virus in circulation, we can accommodate some degree of non-compliance because the non-compliant person will rarely run into an infectious contact. When the epidemic is in full swing, the chance of interfacing with infectious people is much higher, and non-compliant members are more likely to spread the virus around.

We can't think linearly. Any community is moving toward one of two final states: almost no virus in the community, or almost everyone gets infected at which point the virus also spreads slowly because most contacts are between infected people. Without intervention, the second state is inevitable. Once an epidemic gets out of control, it's very hard to revert to the first state.

Cornell's success to this point is showing us that places like China and South Korea can indeed eliminate the virus with early aggressive actions. These countries don't have to fake the data if they did the right things.

Whether the success will last is an open question. It's only three weeks into the term. Prof. Frazier noted that they strongly discourage students from traveling outside Ithaca. I'm not sure this is possible during breaks and Thanksgiving. Similarly, countries that successfully fought off the first wave still face risks if the virus is re-imported from other countries that failed in their initial response. For Cornell, I'm guessing that they will have to roll out arrival testing after each long break.

Notes on the Moderna vaccine trial protocol 1

Bending to outside pressure, Moderna has made public one of the documents submitted to the FDA that provide more details on how they are running the vaccine trial. (PDF here) This is a reassuring move, and I applaud them for taking this step.

In a previous post, I raised two key questions I would like to find answers to, and after reading all 135 pages of the protocol, I can tell you the answers.

How is efficacy measured?

Vaccine effectiveness (technically, efficacy) is the reduction in infection rate among the vaccinated relative to the baseline infection rate among the unvaccinated. A 50% effective vaccine is one that halves the baseline infection rate. (For more on vaccine effectiveness, see this other post.)

An important detail is how this infection rate is measured. According to the trial design documentation, Moderna will confirm infection using PCR nasal swab tests (primarily). To be precise, what they measure is confirmed cases rather than infections.

What is the analysis schedule?

Until the documentation is made public, there has been a lot of sloppy reporting about when results will be available. Now we know. The primary analysis will be conducted when the total number of confirmed cases (across both test and placebo arms) exceeds 151. The expected time of this analysis is six and a half months after the last trial participant receives the second dose. By my estimate, this will be mid June 2021.

The trial started enrolling participants at the start of August. Assume that enrollment is complete by the end of September, the second dose of the last participant should occur at the end of October, the data for the primary analysis will be frozen by mid May 2021, and the trial result will be ready by mid June 2021.

Why do pharma CEOs keep saying they expect results by the end of 2020? That's because they are hoping one of two interim analyses might produce results so stellar that the FDA will have seen enough. The first interim analysis is triggered when the number of confirmed cases exceeds 53, which is 35 percent of the full sample. If it takes 6.5 months to accumulate 151 cases, it would be 2.5 months to get to 53. (In both instances, the half month is added because they don't start counting cases until 14 days after the second dose.)

For the first interim analysis, assume they only analyze participants through the end of August, the second doses should be completed by the end of September, the data will be frozen by mid December 2020, and so the earliest date by which the vaccine might be declared effective is mid January 2021, if everything else falls into place.

***

What to Look Out For

I read through the entire 135-page document. I prepared a list of notes about the clinical trial, mostly about statistical matters. These are placeholders for now. How much these issues matter depends on what happens during the trial.

Blinding

The trial is observer-blind, less than the gold standard of double-blind. Blinding is hiding whether a participant has gotten the vaccine (or drug) or the placebo. Double-blind means neither the researchers nor the participants are told who got what shot, until the data are ready for analysis. Blinding prevents humans from misbehaving as a result of knowing the treatment assignment. In this Moderna trial, the person administering the shots is not blinded. There may be a scientific reason for why some researchers cannot be blinded for this study but I didn't find an explanation in the documentation. (I checked the Pfizer-Biontech trial, and it is also observer-blind, not double-blind.)

Many outside scientists and reporters have missed this detail, and mis-reported the Moderna trial as double-blind (e.g. here, here, here).

Analysis Population

The primary analysis is "per-protocol", which is less than the gold standard of intent to treat. This is always a controversial subject so let me explain the disagreement. In a canonical clinical trial, half the participants get the vaccine (or drug), and the other half receive the placebo (or comparator). Researchers take great care to randomize assignment to these two arms. If each participant has equal chance of being assigned to either arm, then we have created the condition of "all else being equal" so that when we compare the average person across the two arms, any difference can be attributed to the vaccine (or drug).

In practice, some participants will drop out of a trial. It is quite possible that the dropout rate is not the same across the two arms. For example, if the treatment causes irritation, more people in the treatment arm might quit. The statistical debate is whether to include the dropouts in the analysis. The gold standard of intent-to-treat analysis says to stick as close to the original population as possible. Dropouts are therefore included (up to the time of dropout). In a per-protocol analysis, the dropouts may be excluded as if they never existed. The proponents of per-protocol argue that the final outcomes of these dropouts are not known, and so including them might be "unfair" to the treatment arm.

By contrast, supporters of intent-to-treat (including me) don't buy that the act of dropping out is independent of treatment, and worry that the loss of randomization invalidates the "all else equal" condition. For me, the most important argument is the practical point of view: the per-protocol findings apply only to people who take the treatment without dropping out!

Dropouts are not the only subset of participants who might be excluded in a per-protocol analysis. It's just easier to explain.

Back to the Moderna trial. It appears that dropouts prior to the second dose will be excluded from the per-protocol analysis but those who discontinue after the second dose will be included.

Infections Before Finishing Treatment

In the Moderna trial, another interesting subset consists of participants who get infected and then are excluded from the per-protocol analysis. In the primary analysis, Moderna starts counting infections 14 days after the participants get their second dose.

The trial design stipulates that people who have confirmed infection before the second dose will be ineligible for the second dose. All such people therefore will not complete the protocol and be excluded from the per-protocol analysis. The good news is that Moderna has committed to performing an intent-to-treat analysis for comparison. However, the final decision will be based on per-protocol, and there is no guarantee that the intent-to-treat analysis will be published for outside evaluation.

Let's say we're behind at the end of a(n American) football match, and our coach decides to go for an on-side kick and then drive for a field goal. To win the game requires executing both steps. If the on-side kick fails, the other team will run out the clock. (Last year, on-side kick success rate in the NFL was only 6 percent.) If the on-side kick fails, the per-protocol analysis excludes this match because the team didn't execute the entire plan. If the on-side kick is successful and the team gets into field-goal position, the per-protocol analysis includes this match. So, the estimate of the success rate of the two-step plan will be over-optimistic.

Here's the practical point of view. If the vaccine requires two doses, and someone gets sick after the first dose, the vaccine is ineffective for that individual. It doesn't matter that the said individual has not received the second dose; it is meaningless to administer the second dose.

Excluded Infections

Then, there is the subset of participants who get infected during the first 14 days after the second dose. For an unexplained reason, Moderna isn't counting infections until two weeks after the second dose.

This subset will be included in the per-protocol analysis because they took the full two-dose program. However, the outcome for these participants will be "censored," meaning instead of being treated as infected, they are treated as if they have dropped out of the study just before their infections are confirmed.

Again, I take a practical view on these matters. If, after the vaccine is approved, someone gets infected a few days after receiving the second dose, the vaccine would have failed for this individual.

So when the results come out, keep an eye on whether there is a differential likelihood to get early infections on the vaccine arm relative to the placebo arm. The good news is Moderna has committed to conducting additional analysis that counts infections from the first day after the second dose for comparison. However, the final decision does not have to take into account this analysis, and it may not be publicly available.

Counting Infections

Measuring infections in a vaccine trial is always challenging because the participants are healthy. Contrast this with a clinical trial testing a treatment on sick people. Since it's unethical to inject virus into healthy subjects, we have to wait for infections to crop up via community spread.

Moderna isn't very proactive in monitoring for infections. They rely on self-reported symptoms, which then trigger confirmatory PCR tests. For example, the only way researchers will learn about suspected infections during the first 14 days after the second dose is if the participants report symptoms. Not everyone who report symptoms will get tested; the researchers have some discretion for certain types of symptoms.

That's why we will never know the true infection rate. The reported rate should be considered the rate of confirmed cases.

***

These are the most salient information I learned from the just-released documentation on the Moderna vaccine trial. I have several more notes that will be written up in my next post. 

Masks and vaccines

United Nations Covid-19 Response

Wasn't planning on blogging today but the news cycle won't leave me alone :)

CDC Director Dr. Redfield made a bit of a stir yesterday when he told Congress - and I paraphrase - that masks may offer individuals greater protection than vaccines.

This is a very thoughtful statement which rewards some further comments.

***

The way medicine is practiced, we are often misled to over-estimate the impact of drugs and vaccines. When the doctor says take pill X to treat condition Y, we think if we take pill X, condition Y will be cured. In reality, for most conditions and meds, if we take pill X, there is a chance C that pill X will cure condition Y. C is not 100 percent, and is usually way below 100 percent.

Flu vaccines are said to be 50 percent effective or less. What does that mean? Perhaps, that half the people who've gotten flu shots will be protected while the other half will still catch the flu. This isn't how experts use the word "effectiveness" but let me stay with this intuitive definition for now. (I'll turn to the technical definition in the last part of the post.)

This situation reminds me of the famous put-down of TV advertising: half the budget is wasted but we don't know which half.

The trouble of course is that if you halve the TV spending, you will still waste half the money while also losing half the benefit. Same with the vaccines. If only half the people get vaccinated, a quarter of the people will be protected.

The FDA has announced that a SARS-Cov-2 vaccine with 50% effectiveness is acceptable. Staying with the intuitive definition, this means getting the vaccine shot is like buying a lottery ticket with a 50 percent chance of winning. This is not news though. Taking any pill X or undergoing any surgery  has always been a lottery with C percent chance of winning. The value of C depends on the treatment.

One of the reasons why some vaccinated people will still catch the flu is biological variation. Different people respond to the vaccine differently.

***

What about wearing a mask? The mask works as a physical barrier and its function is not affected by anything biological. The same mask worn properly confers the same degree of protection on anyone. Any decent mask blocks some fraction of virus from entering one's respiratory system. Imagine an infectious person coughing directly at me. The mask reduces the amount of viral projectile even if it is not eradicated.

In other words, wearing a mask has a high C. Even though I am a skeptic of many things, mask wearing is not one of them.

Whether the vaccine will work for me or you depends on biology. I have no control over that. Either my body responds to the vaccine or it doesn't. The mask, on the other hand, works on everyone who wears one. That's what the CDC Director is saying, and he's right.

***

Now that I've gotten your attention, let's switch to a proper interpretation of vaccine effectiveness. We have a coronavirus vaccine that demonstrates 50% effectiveness in a clinical trial. One possible scenario is it reduces the infection rate from 1% to 0.5%. (For reference, in the Moderna trial, they assume the baseline infection rate with no vaccine is 0.75% over a six-month period.) That's a 50% drop, and statistically significant.

For every 1000 patients receiving the vaccine shot, five patients will benefit from it as they would have caught Covid-19 without the vaccine. Another five will get sick despite being vaccinated. So C is 5/1000 = 0.5 percent for the vaccine. Getting vaccinated is like buying a lottery ticket with 5 percent chance of winning.

In medical jardon, we say the number needed to treat is 1000/5 = 200. For one person to benefit from vaccination, we must administer 200 vaccine shots. Similar to TV advertising, we can't know who would benefit ahead of time. Statistics tell us that on average, one out of 200 will.

Notice that C isn't 50 percent but only 0.5 percent. It can't be 50 percent because most people won't catch Covid-19 even without the vaccine. For every 1000 people vaccinated, 995 of them won't get sick. Of those 995, 990 would have stayed healthy regardless of vaccination. The 50% effective vaccine really only benefits 0.5% of those who took the shot. (Note again that I'm using the assumption from the Moderna clinical trial design.)

But this is not the full story. The practical effectiveness of the vaccine depends also on what proportion of the population takes the shots. Let's pretend that half the population get vaccinated. Then the 0.5% benefit of vaccination is halved. The infection rate in the population is reduced from 1% to 0.75% since the infection rate for the unvaccinated will remain at 1%.

Herd immunity now complicates this picture. If enough people get the vaccine, typically 60% or higher, the amount of virus circulating in the population is suppressed enough that the infection rate for the unvaccinated will dip below 1% so collective action creates a positive externality.

 

 

 

Confusion over case trends and positivity ratios, an aftertaste of targeted testing

We are now experiencing the aftertaste of targeted (aka triage) testing, a policy that has unsavory consequences that were predictable (link). Targeted testing is the idea that only people with severe symptoms or at high risk of infection should get tested for the novel coronavirus. Compared to a broad-based testing plan (whether comprehensive or randomized), targeting economises the number of test kits. The basis of any targeting strategy is the skill to predict who is infected. The group that is targeted should have a higher infection rate than the group that is not targeted. It's important to bear in mind that targeting not only suppresses how many people are tested but also affects who gets tested.

One way in which targeted testing hurts us is that it ruins our datasets from which we obtain insights about the pandemic. This post is prompted by Ken's comment on my previous post on casual presumptions of randomness:

The number of daily new cases in some countries doubled as fast as every 2 days. This didn't mean a greater rate of infections, it was simply a result of catching up all the non-tested cases.

The connection between infections, tests, and cases is complicated, and the media shift back and forth between the rate of cases (positive test findings) and the positivity ratio (proportion of tests that came back positive). I spent some time explicating these relationships, and hope that this post will succeed in helping you gain some intuition around those concepts.

We start by decomposing the positive test findings. In the flow chart below, starting with the dark gray box on the left side, we examine a fixed population of people. At any given time, the people are divided into three "compartments," the infectious people who currently have the virus, the susceptibles who have not been infected, and the recovered (which by convention, includes those who died from Covid-19).

Also by convention, anyone can be infected only once, and we thus ignore recovered people from further consideration. From the dark gray box emerge two paths that terminate at positive tests: the upper path consists of infectious people while the lower path, susceptibles. Some portion of the population are infectious; within that, some proportion have been tested for Covid-19, and within that, some proportion have tested positive. This last tally is one part of the confirmed cases. The other part of the confirmed cases comes from susceptibles, represented by the lower path.

The number of confirmed cases is the number of positive tests. This is a key metric on all Covid-19 dashboards. The other important statistic is the positivity ratio. This is the number of positive tests (represented by the dark red box on the right) divided by the number of tests conducted (the sum of the two yellow boxes).

The next diagram describes the drivers of the two dashboard statistics:

• the weights of infectious or susceptible compartments in the population (excluding recovered)
• the probability of getting tested among the infectious or susceptible group
• the weights of infectious or susceptible people within the tested subpopulation
• the positivity ratio for the infectious or susceptible people who got tested (call these the group-level positivity ratios to distinguish from the overall positivity ratio)

The overall positivity ratio is affected by three signals. First up is the rate of infection in the population, reflected in the relative weights of the I and S compartments. Second, the testing regime imposes a selection effect on these weights, producing the second set of weights. The notorious triage or targeted testing protocol restricts testing to patients showing severe symptoms or at highest risk - these disproportionately arise from the infectious group, and so the second set of weights are more skewed toward infectious people, compared to the first set of weights. Thirdly, test accuracy affects the number of positive test results, given any tested subpopulation.

Test accuracy is typically measured by sensitivity and specificity. On the upper path, the positivity ratio of the infectious-tested group is just the sensitivity, which we assume to be 90 percent. On the lower path, the positivity ratio for the susceptibles-tested is the rate of false positives (as this group is not infected), which is 1 minus the specificity, which we assume to be 5 percent. (For the sake of this post, we assume that the performance of the diagnostic tests has not changed during the pandemic and thus the group-level positivity ratios are fixed. This simplifies the following examples.)

 

What's special about 5% positivity ratio?

The media keep reminding us that the CDC wants the positivity ratio to fall below 5%. What is the significance of the 5-percent threshold? I can explain using the flow chart.

We hope for the epidemic to dissipate. When this happens, the infectious compartment dwindles to zero while the susceptibles stabilize. This means there will be no more flow through the upper path. All of the positive test results will come from the lower path (and are false positives).

The overall positivity ratio is a weighted average of the group-level positivity levels, the weights being the proportions of those tested who are infectious or susceptible. In this extreme case, the weight is entirely skewed towards the susceptible-tested, for which the positivity ratio is 5 percent. Thus, the overall positivity ratio will be 5 percent.

In other words, when the positivity ratio dips to 5 percent or below, new infections should be dying out.

 

What's the relationship between rate of cases and rate of infections?

I will now use this setup to investigate Ken's comment. He was reacting to the naive interpretation of case growth as infection growth, popularized by the media. Such a conclusion is invalidated because of targeted testing.

Note that infection growth cannot be directly observed. By contrast, case growth is reported on every dashboard. In Ken's scenario, the number of cases (i.e. positive test results) are growing rapidly.

Start from the dark red box on the right of the flow chart, and work backwards. The jump in positive cases may come from either of the two paths.

First, consider the lower path representing susceptibles. During the course of an epidemic, more people get infected, which reduces the number of susceptibles in the population, which in turn decreases the number of susceptibles testing positive. So if the cases from this lower path have to go up, then it must be that susceptibles are suddenly more likely to get tested.

One such scenario is if politicians expand targeted testing to those with mild or no symptoms. This is Ken's hypothesis.

Such a policy shift also affects the upper path. In fact, infectious people are more likely to experience mild symptoms. Even if the rate of infection is stable, the number of positives tests arising from the upper path grows. If infections are also accelerating, both factors contribute to case growth.

Ken argues that the entire case growth can be attributed to the expanded testing capacity while the media generally interprets the case growth as infection growth, ignoring possible shifts in testing policy.

The trouble is that both factors are at play, and we don't know the relative sizes of their contributions. This confounding is brought to you by targeted testing! (I can't resist: it's free and comes with a hidden cost.) This example demonstrates the aftertaste of targeted testing. It messes up the data so we can't interpret them properly.

It doesn't appear that Ken's scenario is happening in the U.S. as the federal government has continued its misguided targeted testing policy. Some politicians even grumble that we are conducting "too many" tests. (This can only happen if we were conducting too few tests, and thus under-reporting cases early on.)

 

How does targeted testing affect the positivity ratio?

In Ken's scenario, he assumes that the increase in testing accounts for the entire jump in cases. So, let's stipulate that the rate of infections is stable, and look at how the testing policy affects the positivity ratio.

The overall positivity ratio is the weighted average of group-level positivity ratios, so the key is to understand what happens to the second set of weights, which describes the mix of infectious and susceptibles in the tested subpopulation.

Under a randomized protocol, the second set of weights just represents the underlying mix of infectious and susceptibles since both groups have equal probability of getting tested. Under a targeted protocol, the second set of weights is biased toward infectious who are more likely to experience symptoms. Thus, early on, the positivity rate is over-stated. If targeting restriction is relaxed, the bias toward infectious is reduced. Thus, the relative weight of susceptibles-tested is pushed up, and so the positivity rate is driven down.

Since the rate of infections is assumed stable, the shift in positivity rate tells us only about the testing policy. We've just discovered how politics enter the data.

***

Targeted testing introduces bias into the data, which pollutes the link between the rate of cases and the rate of infections. Remember: we use the rate of cases to infer the rate of infections because in the real world, no one knows the rate of infections.

Targeted testing is the aftertaste that you can't seem to get rid of. It polluted the data when first implemented. Then, every subsequent change in policy, and the gradual behavioral adaptation to these changes, add to the confusion. The effect on number of tests is observable but the effect on the types of people taking the test is not.

This tragedy on the national level is now being replicated on individual school campuses. Many schools are doing targeted testing only on students who self-report symptoms. Their dashboards are reporting confirmed cases, the trend of which is being interpreted as the trend of infection. What is the point of data-driven decision-making when the data are known to be bad?