Big Data, Plainly Spoken (aka Numbers Rule Your World)

We're told almost nothing about the Moderna vaccine trial

Crisis opens the door to wishful thinking. On this door may hang the tag "Operation Warp Speed," the code name for the US government's effort to fast-track a vaccine for the novel coronavirus.

News arrived last week that Moderna is to begin the Phase 3 human trial of its candidate vaccine. I am curious how we are streamlining this process, so I pulled up the document that lays out the design of this clinical trial. (link)

A vaccine trial presents many unique challenges. We cannot ethically inject someone with the coronavirus, especially those who are assigned the placebo at random. Instead, we have to round up people who live or work in high-risk environments, such as hospital staff or communities with a lot of virus. (This presents a different ethical issue, as poorer communities with a high density of minorities have been hit hard by Covid-19.)

Half the trial participants are given the vaccine shot while the other half receive a placebo. It's crucial that no one knows who has gotten the vaccine, and that the assignment to vaccine or placebo is determined by a coin flip. After administering the shots - really two shots a month apart, as Moderna learned that one shot was not enough, the scientists must wait to see what proportions of each treatment arm eventually get infected with the novel coronavirus.

***

The general design is simple enough but the operational challenges are substantial. I'll focus on three issues.

Measuring the outcome

How do we know if and when someone gets infected? Like any clinical trial, there needs to be follow up. Testing a drug such as remdesivir on patients is much easier since the participants are staying at the hospitals. In a vaccine trial, you rely on the participants roaming around the high-risk communities. Each participant must be monitored until s/he gets infected (or till end of trial, whichever occurs first). How frequently are the participants tested for infection? What test is used? How are test results reported to the trial managers?

There's more. The two doses of the vaccine are administered one month apart. How will they deal with people who tested positive after the first dose before receiving the second dose? (In my view, these people should be counted as infected.)

Timing the analysis

When will the data be analysed? This is extremely tricky requiring a lot of guesswork. How long does it take for the vaccine to do its job? We don't even have a solid handle on how many contacts people have each day with infected people, how likely it is to get infected upon contacting an infectious person, how long it takes for an infection to become detected, etc. We also need to have a firm grasp of how much virus there is in these high-risk neighborhoods. One element of any clinical trial design is a specific analysis plan. For example, in Gilead's remdesivir trial, the scientists analyzed the mortality rate at day 15, meaning each patient is tracked for 15 days after they received treatment. There may be multiple "endpoints"; another analysis was planned after 30 days for remdesivir.

Controlling for behavioral changes

Someone might have opted out of attending a birthday party if they weren't participating in the vaccine trial but after receiving the shots, they might decide to risk it. Of course, people will be told not to adjust their everyday life; for a trial lasting many months, that's a big ask. To measure such behavioral change due to trial participation, a third group can be included in the clinical trial, which will not receive an injection at all.

***

Those were three questions I was hoping to find answers to from the clinical trial design documentation. So, what did I learn?

Measuring the outcome

Surprisingly, there is nothing on the follow-up plan in the design document. We don't know what diagnostic test will be used, who will be testing the participants, how frequently are they tested, etc. This is a worrying omisison. Also omitted is how they intend to analyze people who test positive between the two doses.

Timing the analysis

The analysis plan is also largely absent. One hint is that the key metric of the infection rate has an observation window described as "Time Frame: Day 29 (second dose) up to Day 759 (2 years after second dose)". This implies that each trial participant is monitored for up to two years.

The phrase "up to" will prove pivotal. It means Moderna will not declare failure until two years have passed but they may announce mission accomplished much earlier. In fact, the leader of Operation Warp Speed predicted on CNN last night that they could have a working vaccine with over 90% effectiveness by the end of the year.

Such a statement almost defies belief. Enrollment in the trial starts for practical purposes in August. For someone who joins the trial on August 1, s/he gets the second dose on August 29, if the scientists monitor her or him for 60 days, this patient can have an outcome by November 1. No trial can enroll the entire cohort in one day. Assume it takes 30 days to find enough participants. Those who start the trial on August 31 will be ready for analysis on Nov 30. Even if the data analysts work quickly, it will be around Christmas when the first analysis can be reported.

If a vaccine has to be approved and commercially available by year's end, then they are banking on the vaccine showing a benefit within one month of the second dose. Is it reasonable or wishful thinking?

Controlling for behavioral changes

Nothing in the design document addresses this issue. It's highly unlikely they will set up a third test group as that will use up some of the trial participants, lengthening the time they'd need to report results.

***

Not specifying the time points of the analysis is a deeply worrying omission in the trial design. As I explained above, this might reflect a lack of confidence in our understanding of the virus. However, such an omission opens the door to a statistical fallacy known as testing to significance.

The following analysis procedure is the statistician's worst nightmare: every 30 days for up to two years, the Moderna analyst computes the infection rates for the vaccine and the placebo, and tests the difference for statistical significance; if the difference is not significant, the analyst waits another month and repeats the process; if the difference is significant, the analyst declares victory, ending the trial after approval from the review board.

Such a procedure sounds reasonable but it is the source of a lot of spurious statistical findings.

I'm preparing another post to explain this statistical fallacy. What you need to know is that if one waits around long enough, anything can happen. In the context of a clinical trial, if one checks for a statistically significant difference between the vaccine and placebo arms every 30 days for years, at some point, such a difference will appear - even if the vaccine is useless.

In order to prevent such spurious results, statisticians recommend nailing down an analysis plan prior to running an experiment. This is precisely why clinical trials are "pre-registered" - why Moderna is required to submit the trial design prior to the start of this clinical trial. The analysis plan should not be influenced by peeking at the interim results. This possibility is present in the Moderna vaccine trial since the design document is missing some crucial details.

 

 

 

 

It's high time to call out those who got it right

During this pandemic, epidemiologists have been portrayed as clueless idiots who "got everything wrong". That makes sense only if one ignores the evidence. I can go so far as to say that epidemiologists have gotten everything right - they have not predicted every detail but they have nailed the big picture. (They told us about re-opening too soon, second waves, viruses not chased away by summer heat, etc. etc.)

Remember just a month ago, when cases started to surge in Texas in June, many people looked at the glass half full - they said deaths weren't increasing even as cases exploded. This was the picture they were looking at:

I wrote a blog about this statistical artifact at the end of June. Deaths always lag cases. I said you've got to have patience. Stop interpreting daily tracking charts, and wait a few weeks.

It's more than a month later. What does the chart look like?

By early July, deaths from Covid-19 started to rise rapidly in Texas. In fact, the slope of the growth of deaths in July is roughly the same as the slope of the growth in cases in mid-June.

To see what happened more clearly, I shift the gray line (cases) to the right by three weeks. You can see that deaths and (shifted) cases have changed in lock step.

This chart reveals a couple of insights.

First, the time lag is sufficient to explain everything. There have been speculation about other contributing factors, such as better treatments, and younger patients but those factors are not necessary to explain the trends in cases and deaths. It's important for any insights to be validated by the experience on the ground. Hospitals should know how long an average patient who eventually dies from Covid-19 stays hospitalized.

Second, notice the weird spike in deaths on July 27 (the spike has already been tamed by a 7-day moving average); that's due to a change in how deaths are counted. Here are some notes from the Covid Tracking Project about all the recent changes to how they count things in Texas:

The recent switch of requiring deaths to be confirmed by death certificates, for example, introduces a delay in reporting. It may take a week or two for a death to appear in the data. It would take quite a bit of effort to go back and switch the entire series of data to reflect this new definition. Meanwhile, it's impossible to trust the dates of deaths.

Every note listed above should concern the data analyst. That's the key reason I cannot have confidence in the 3-week time-lag estimate described above. It must come with a margin of error, probably of at least 1 week, to account for data deficiencies, known and unknown.

It's very sad to see that the Covid Tracking Project team has rated Texas's "data quality" as grade A. I can't even imagine how bad the data quality is to earn a B or C grade!

 

 

Another Covid19 victim is science practice

[Yes, that's a mango. It's not a random picture. Keep reading.]

Gilead's remdesivir has grabbed a few headlines lately. The company just issued a press release saying a new analysis showed that the drug saves lives. This might surprise some readers - hasn't the drug been approved already? Does this news mean Gilead won approval without showing remdesivir saves lives?

In this pandemic world, lots of things have turned upside down, and the practice of science has become a casualty as well. This post focuses on the "emergency use" practices that have creeped into science.

The Gold Standard of Science Loses Luster

Getting immediately to the question I raised above. Indeed, remdesivir was approved even though the clinical trial did not show a statistically significant effect on mortality when compared to a placebo. The approval was based on an improvement on a different metric, the so-called "recovery ratio". Patients are graded on an 8-point scale, with a cutoff level below which they are considered "recovered". Thus, at the end of the analysis period, each patient is classified as either dead, recovered, or still in treatment; a certain proportion of patients have "recovered".

The press release from Gilead cites yet another metric, median recovery time of 15 days for recovered patients on placebo versus 11 days for those on remdesivir. Probably this metric is considered fit for public consumption. This result is not very exciting, when one considers the just-announced price point of over US$3,000 per patient. (Actually, in the clinical trial, patients received a 10-dose treatment, which maps to a price point of US$5,800. While the 5-dose treatment is priced lower, it's not clear to me that the clinical trial result would hold using the lower dosage.)

Against this background, the news that remdesivir did save lives after all is newsworthy. The casualty I alluded to earlier is of science practice. This new conclusion isn't based on a new clinical trial but a "deeper dive" analysis of the previous data. Lots of data analysts are asked to do "deep-dives," which frequently amounts to a salvage operation. I'm reminded of the lemon of a mango I just sliced open (it did taste like a lemon). Parts of the flesh have already turned black, I should have thrown the whole fruit out but I carefully carved out the little orange bits, and savored them.

Of course, Gilead already compared mortality rates of treatment versus placebo groups in the preliminary analysis and that did not reach statistical significance so where's the magic?

We don't have much information. In a press release, Gilead described the new analysis as a "Comparative Analysis of Clinical Recovery and Mortality Outcomes from the Phase 3 SIMPLE Trial Versus Real-World Cohort of Severe COVID-19 Patients Receiving Standard of Care." In other words, they abandoned the clinical trial - the gold standard of medical statistics, and adopted a methodology for observational studies, usually resorted to when we don't have the gold standard. 

Effectively, Gilead ditched the placebo group in the trial and replaced it with a self-selected placebo group, constructed by finding patients not in the trial, did not receive remdesivir and match the characteristics of the patients in the trial. This is comedic, actually, the kind of thing that comes with a serious health warning in every statistics class. Why? Because the placebo group in the trial already met all those criteria. The placebo patients did not receive remdesivir, and matched characteristics of those patients who received remdesivir (by design via randomization). These patients were just as "real world" as those others. In addition, the randomly assigned placebo group does not suffer from unknown selection bias, neither by patients nor by analysts.

Don't get me wrong - I love matching methodologies as a principled, explainable way to analyze observational data. But I can't think of a reason to use a matched cohort to overturn the conclusion of a clinical trial. Just as we're approving drugs for "emergency use", lowering the usual standards, we're practicing emergency-use science.

 

Press Releases Topple Peer Reviews in Scientific Communications

The highly unorthodox analysis was announced to the science community in an equally unorthodox method. A press release, with barely any details. Science in the pandemic era has been communicated as a series of leaks and press releases. The previous result was leaked to the press by Dr. Fauci, whose organization is a sponsor of the remdesivir study, about a month before the scientific study was published in the New England Journal of Medicine. The journal dropped the Gilead study Friday evening on Memorial Day weekend in the U.S.

Notably, the published study described the treatment as "Remdesivir was administered intravenously as a 200-mg loading dose on day 1, followed by a 100-mg maintenance dose administered daily on days 2 through 10 or until hospital discharge or death." This corresponds to a 10-dose treatment. If this is the full disclosure of how the trial was conducted, then the patients who received 5 doses were those who recovered early.

A major problem with the specific press release on a matching analysis is that nobody can evaluate the work. Matching is a highly subjective exercise that involves a lot of assumptions. We don't even know what the entire set of variables patients were matched on. So, the situation is similar to New York State antibody testing, for which we still have zero information about how it was conducted.

After selected numbers and headline findings are leaked, the media has no ability to fact check these results and no experts can comment on them without seeing the study. This creates a vaccum in which the results are propagated and unchallenged. When the study is published, it feels like old news but in fact, the results have not previously been vetted by outsiders. Until this day, lots of reporters, and also other scientists are citing Cuomo's numbers even though no data from the antibody testing program has been published anywhere by the scientists running the program.

If the clinical trial has rock-solid results, there is no need to play the PR game. Just get the study through peer review and out in NEJM as soon as possible, and let the world endorse it on its merits.

 

Cherry-picking Become the New Pastime

Prior to the leak of the positive result eventually published by NEJM, the media reported negative results from another study of remdesivir, from Wuhan, China. That study is also a gold-standard, randomized controlled trial. It has now been conveniently forgotten.

Every result adds to the collection of knowledge we have about the drug. It would be sad if the new "comparative analysis" result grabs the headline while the unsatisfactory result from the clinical trial gets the boot.

 

[P.S.

I had the pleasure to be a guest on the CSAI Podcast, hosted by John Reid and Rusen Aktas. We literally chatted for many hours about Covid-19 models, data science, science communications, uncertainty, etc. The best bits are found here: 

Apple: https://podcasts.apple.com/gb/podcast/csai-2-covid-19-and-data-science/id1522519185?i=1000484544918

Spotify: https://open.spotify.com/episode/2Ejv3yZWSmODctu1an2OnG?si=FvSNkkJkRy6io6RDk3ecMg

Anchor fm: https://anchor.fm/csai-podcast/episodes/CSAI-2--Covid-19-and-Data-Science-ege3si

]

 

The email subject line

I completed a survey by a New York theater about planning for eventual re-opening of arts events. It was a nicely designed survey. There are several things to learn from this consumer research effort.

The survey was delivered via email, and they managed to capture my attention with a good subject line. It was "When will you feel comfortable returning to..." I am not sure if this is effective for the majority of people but it worked for me. The theater has definitely sent me emails over the last few months; this might be the first one I actually clicked on.

It was a breezy experience. Within seconds, I was on the survey site and within minutes, I finished the survey.

One obvious goal of the study is to estimate how many of us would be willing to attend live performances. An analyst would have to artificially lower the numbers observed in the survey. This is because the subject line - which I liked so much - is a source of bias. When will you feel comfortable returning to. People who are missing their theater are more likely to take the time to fill out the survey.

There were a number of questions revealing the theater community's concern that the pandemic will change our behavior forever. I'm not one of those people. Pandemics do pass, and I don't understand why I should change the frequency or spending at theaters when Covid-19 is no longer with us.

Surveys continue to play an important role of analytics. For this type of questions, surveillance data are lagging indicators. If we are looking at ticket sales, we're not going to know when people will return to theaters until they have returned to theaters.

 

 

Prediction: demand for off-campus housing near Cornell to booooom

Credit: United Nations

Over the holiday weekend, news came out that Cornell University, one of the Ivies, has decided to re-open for the Fall term 2020. This means no more online instruction, a full-scale return to in-person teaching. The decision is supported by a mathematical model built by a team of professors and graduate students. They published a very detailed report here, and noted that it is a work in progress.

The report makes a strikingly "counterintuitive" claim, that the number of infections within the Cornell community will be one-fifth lower in the "Full Re-open" scenario relative to the "No Re-open" scenario.

 

Outcome is sensible if you know the assumptions

At the superficial level, everyone expects infections to go rampant if a large college community like Cornell (~ 35,000 including faculty, staff and students) opens up. But if one takes account of the assumptions behind the model, carefully laid out in the Cornell report, one must conclude that the research conclusion is entirely non-controversial.

In the Full Re-open scenario, Cornell will successfully implement a full-scale test-trace-isolate program which the U.S. government failed to do at all levels. Every member of the community is assumed to be tested for the novel coronavirus every five days. Anyone who tests positive or self-reports positive will be quarantined. Their contacts will be traced and quarantined as well. On top of all that, mask wearing and social distancing policies will be implemented.

Before anyone is allowed to return to campus, they must take two diagnostic tests. The first test is administered at home, and if someone gets a positive result, they must self-isolate at home and delay their return. The second test is required upon arrival, and anyone testing positive is immediately separated from the community.

With the above assumptions, the researchers have described a community in which almost no one coming back to campus will have the virus, infections are detected early, asymptomatic carriers are identified, and infectious or suspected infectious people are isolated, so it is no surprise that after 16 weeks, only 3.5 percent of the community is expected to have been infected.

In the No Re-open scenario, a test-trace-isolate program is unavailable, community spread is not under control, and so a lot more people end up getting sick.

 

Some unsavory implications

If these modeling results are accepted, they make rather disturbing commentary on several related issues beyond Cornell.

First, the model provides solid evidence that a full-scale test-trace-isolate program constrains community spread and saves lives. This report then is a repudiation of U.S. anti-pandemic policies up to now.

Second, the report also rebukes the previous decisions by American universities, including Ivy League colleges, in March, to shut down their campuses, sending students home. By switching to virtual instruction, these schools have unwittingly contributed to a large growth in infections.

Third, the researchers stress again and again the importance of broad-based, repeated testing to stem infections by asymptomatic carriers, something I've been advocating for a while here. They acknowledge that such testing is not available to the U.S. public at large, which is the primary reason the No Re-open virtual instruction scenario looks so much worse. The report also concludes that masks and social distancing are necessary but not sufficient. This speaks to the continuing failure of U.S. policy.

 

Notes on the modeling framework

The researchers adopted a traditional framework for the analysis. They set up a simulation which divides the community into many "compartments," including not just the usual Susceptibles and Infectious, but also Exposed, Quarantined, Asymptomatics and so on. Every day, people are moved from compartment to another compartment, such as from Susceptible to Exposed, or from Symptomatic to Recovered. After 16 weeks, they observe the proportion of people in each compartment, the outcome. Each run of the simulation represents a possible future.

The pattern of inter-compartment movements is governed by a set of about 15 parameters. These parameters are assumptions of unmeasurable quantities, such as the average number of contacts per day per person, the probability of getting infected upon contact with an infectious person, the proportion of asymptomatics, and so on. 

A subset of these parameters are specified as "stochastic", which is a reason why different simulation runs generate different outcomes. For example, the model assumes an average number of contacts between infectious and susceptible people. Without the stochastic element, given the current mix of infectious and susceptible people, the simulation runs should produce the same number of contacts. The stochastic element introduces variability around this average number so that in some runs, the number is above average, and in others, below average.

This simulation framework is old-school, in the sense that most parameters are fixed to average values. A modern Bayesian model treats all parameters as "random variables" each with an underlying probability distribution. Instead of fixing, for example, the average daily contact rate to 8.3, a Bayesian treats the average rate as a variable, with values that are spread around 8.3. Because of this additional source of variability, a Bayesian model will show wider error bands than the Cornell framework.

The researchers first nailed down a base case scenario ("nominal"). They then did a classical best case and worst case analysis. For instance, the asymptomatic proportion is 48% in the base case, 27% in the best case, and 68% in the worst case. Not all paramaters have three settings. In all three scenarios, they assume that a 50 percent success rate in contact tracing.

 

Key drivers of the model

In this section, I speculate about the key drivers of the outcomes of the Cornell model. Your level of comfort with the model outcomes depends on whether you think these assumptions are reasonable.

Compliance to the test-trace-isolate program is a key structural assumption. The current version of the Cornell model assumes full compliance. This means every community member submits to testing every five days. This means everyone testing or reporting positive is quarantined. This means everyone is following masking and social distancing rules.

Full compliance is crucial to stemming community spread via asymptomatic carriers. One counter-intuitive insight from the model is that infections are reduced when the proportion of asymptomatic carriers is raised. Because of the test-trace-and-isolate protocol, they assume that an asymptomatic case will be detected even earlier than a self-reported case; earlier detection means less time for the virus to spread around. In the No Re-open scenario, cases are detected only through self-reporting so the virus has more time to propagate.

Another strong structural assumption is the on-campus, off-campus living situation of the undergrads. The researchers assume in the Full Re-open scenario, all returning undergrads live on campus while in the No Re-open scenario, most undergrads still move back to Ithaca, where Cornell is located, but all of them live off campus. This is no small matter because the model provides test-trace-isolate only to on-campus students. If some returning undergrads live off-campus in the Full Re-open scenario, they would not be regularly tested and may become a source of infections.

Another important assumption is low prevalence, on and off campus at the start of term. The base-case prevalence for the county outside Cornell is assumed to be 0.28%, an exceptionally low value; and for people's home locations, 2%. The researchers cite a Cornell Vet School source with a single positive test result for the former number. For many weeks, New York State Governor Cuomo has claimed that the prevalence in the state is above 10%, and that it is 3.6% in less populated parts of the state. (I am not arguing for Cuomo's number as readers of my blog know I don't trust Cuomo's data because his state health department that conducted the antibody tests has not made any information available about those tests. I'm juxtaposing these two numbers to show the range of possibilities.)

The double testing requirement regardless of symptoms prior to and upon arrival ensures that few returnees carry the virus with them. Thus, the researchers assume that only 0.09% of the returnees are infectious. They apply this low prevalence rate to people who already reside on the Cornell campus at the start of term "for expedience".

Another assumption concerns the average number of daily contacts between people in the community. This is set to 8.3. I think this is too low. It is defined as "an interaction between two people that has the potential for transmission of infection," and can include multiple contacts with the same person. The value is really a plug in the Cornell model because it's one of several factors leading to the implied value of R0, which they want to match to a CDC-endorsed value. If they increase daily contacts, they will decrease the probability of infection per contact to arrive at the same R0.

As with other epidemiological models used for this pandemic, the Cornell model assumes immunity. In other words, anyone who has recovered becomes immune, taken out of circulation. The researchers are certain there will be no deaths so the only path after infection is recovery and immunity, with a few hospitalizations.

Yet another feature worth noting is the fixed proportion of people assigned to four levels of severity: asymptomatic, mild symptoms, hospitalized, ICU.

Good data to estimate infection rates and severity rates are hard to come by. In the assumptions used in this model, the undergrads fall into the 18-44 age group, which has the greatest chance (52%) of being asymptomatic (by comparison, 65 and above is assumed to have 13% of being asymptomatic). In the No Re-open scenario, the entire off-campus population are assumed to belong to this age group, which means they are mostly asymptomatic carriers, and because they are not tested while off campus, the model assumes a longer time to spread the virus, and this pretty much explains the headline result.

 

Sociology

At this stage, the Cornell model is a feat of engineering. It will be improved with some sociological input.

First is the thorny issue of compliance. Everything from mandatory testing to wearing masks is an inconvenience. A college isn't an autocracy so it's hard to imagine that full compliance is possible. What will the school do when people don't show up for testing? The sensitivity analysis shows 41% more infections when testing is reduced to every seven days.

Second is complacency. In the world predicted by the Cornell model, there are barely any infections. Most tests will come back negative. People will start feeling safe and get complacent. Compliance rate will likely deteriorate over time.

Third is gaming. Recall the double testing for returnees to ensure low prevalence at the start of term. At some point, Cornell will announce a date on which this rule takes effect. At this time, a fraction of people will arrange to arrive just before that date.

I'm not just speculating here. This human behavior is well documented in various countries that have announced a date after which all visitors must be quarantined on arrival. Some visitors then scheduled their flights to arrive just before that date. Journalists were interviewing "disappointed" passengers whose flights were unexpectedly delayed, leading to the "unfortunate" situation in which they had to be quarantined.

Cornell is surely going to have to extend the test-trace-isolate program to students in off-campus housing in Ithaca. Otherwise, the well-intentioned re-opening plan will fail. What is the university going to do when these returning students choose to live off-campus, rather than on-campus as the model assumes? Every such student brings the Full Re-open scenario closer to the No Re-open scenario.

Gaming the rules lets the virus creep onto campus. The current model does not account for this possibility.

Fourth is unhappiness. All these rules and restrictions adversely affect campus life. It also appears that the contact tracing by the county health department may be lip service. They estimate that contact tracing takes only one day. Also, they do not test contacts discovered during the tracing, and thus Cornell will likely quarantine all contacts whether they test positive or not. The model estimates that 700-1,000 people will be quarantined at peak. While this is great for public health, I'm not sure how the community will receive it. Further, given 100% in-class instruction, what's the plan for quarantined students to keep up with school?

These are just a few predictable issues that should be included in the model.

 

The "No Re-open" alternative

The one definitely bad thing to have happened to this study is how the labels of the two scenarios fail to describe what is being modeled. As the study authors repeatedly explain, the biggest difference between the two scenarios is the presence of a strict test-trace-isolate program. Full re-opening without test-trace-isolate will not produce the predicted outcome.

How is the No Re-open (i.e. 100% virtual instruction) scenario defined? The fraction of the community already on campus (all faculty and staff plus half the graduate students) is assumed to continue to stay there. Of the remaining students, mostly undergrads, they assume about half of them will return to Ithaca and attend on-line classes from off-campus housing while the other half will stay at home or not return to school.

The two scenarios are closer than one might expect. In the base case, the Full Re-open scenario assumes 10,000 undergrads return to campus while the No Re-open scenario assumes 9,000 undergrads move back to Ithaca (and live off campus). What sets them apart is that students who live off campus will not participate in the test-trace-isolate program.

This setup is curious to someone admittedly not familiar with the campus housing situation at Cornell. The Full Re-open scenario suggests that they have on-campus housing for at least 10,000 undergrads. The No Re-open scenario suggests that there are at least 9,000 undergrads who can find off-campus housing. With a total of 15,000 undergrads, there must be quite a few vacancies during normal times at both on- and off-campus housing.

The model further assumes that those living off-campus in Ithaca will have zero interaction with the on-campus community. It assumes that the average number of contacts will be the same whether the student lives on campus or off campus. I should note that those two assumptions contradict each other. The former implies that fewer total contacts in the No Re-open scenario (with off-campus undergrads) while the latter implies an equal number of total contacts.

 

Pitfalls

The parameters in the model are by and large determined independently but some of these variables are clearly correlated while others are likely affected by the same lurking variables. This creates difficulty of interpretation and potential for contradiction, especially when looking at sensitivities.

I'll give two examples. In the base case Full Re-open scenario, 10,000 undergrads are expected to return to campus, and the prevalence of Covid-19 in their hometowns is assumed to be 2%. In the worst case scenario, 12,000 will return, and the prevalence is doubled to 4%. The decision to return to campus is likely correlated with the student's perception of risk, which depends partly on hometown prevalence. So those two parameters probably aren't independent.

In Figure 12, they remarked that the model outcome is insensitive to the time delay of contact tracing. The model assume one day's delay in all scenarios while the sensitivity analysis explores up to seven days. Sensitivity analysis is done one factor at a time while holding all other parameters constant. One such parameter is the fraction of contacts identified and traced, which is fixed at 50 percent. Is it plausible that the extra six days of tracing do not increase this proportion?

Speaking of Figure 12, I already put up a blog post about the problematic graphical representation of the sensitivity analysis results. See here.

***

The Cornell model is complicated, and by their account, assembled in haste. While it has become fashionable to damn Covid19 modeling efforts, one should realize the value of the thinking behind models. Building them forces us to think hard and deeply about cause and effect, direction and magnitude, relative importance of different factors, data quality, and human behavior. The output is infinitely better than mouthing off.

The decision to re-open campus is more complex than this type of model can capture. There are many other issues involving the school's operational model, debt load, customer satisfaction, capability to handle an extra class of students, status of international and out-of-state students, parents' impatience with having kids at home, etc. etc.

If the decision is to be based on just this model, I'd not expect the outcome under Full Re-Open to materialize. I think the students will not comply with such stringent rules, and when they game the system, they adulterate the conditions that produce the "counter-intuitive" result that captured the media's attention.

 

 

 

Modeler exceptionalism

Americans exceptionalism is act as we say but not as we do.

It turns out a subclass of modelers also claim exceptionalism.

I'm reacting to several news items from recent weeks. This past post is highly relevant. The post is titled "False belief in true models".

***

The first news item is the giddy response by Wall Street and economists to the latest jobs report in the U.S. Here is the typical news headline:

The consensus forecasts by economists was 2.9 million. If you put out a projection of 2.9 million and the number turns out to be 4.8 million, you'd think the forecasting model is horribly off. But not if you are talking to economists! Every time an economic indicator is reported, the news describes it as being over or below expectations. In other words, the model is always right.

Now, during the last few months, you'd be living under a rock if you have not heard at least one economist criticize Covid-19 projections as wrong. Here is a Vox headline that is typical:

You might be scratching your head. These researchers also build forecasts. When the numbers come out, they differ from the predictions. When the actual number of deaths is below the forecast, shouldn't we rejoice that public health policies have over-performed expectations? And if deaths come in above forecast, the virus is a terrible adversary. Except that the media now tell readers that the modelers are crap.

Some economists went so far as to question the intelligence of epidemiologists who build such models (link).

What do we call modelers who believe their own models to be always true and other people's models to be usually wrong? I like to call them exceptionalists.

 

 

The power of groupthink

In a congressional hearing this week, a leader from the CDC (or was it NIH, I couldn’t recall) mentioned “group testing” as a new tactic to be rolled out. Unfortunately, this may be another case of too little, too late for the United States. But the concept of group testing is rooted in statistics, so let’s talk about why it is an attractive idea.

Imagine we need 20 cc of blood to conduct a diagnostic test. Test kits are short in supply, so we must make some magic. Normally, we use one test kit per person tested. Is it possible to test N people using fewer than N kits?

Statistics enter the picture. We’ll consider 10 people. Take 2 cc of blood from each sample and mix them thoroughly. Use one test kit on the pooled sample. If this pooled sample tests negative for the virus, we conclude none of the 10 are infected, and so save nine test kits. On the other hand, if the pooled sample tests positive, we have to next test each individual, so in this case, we use 11 kits for 10 people.

There is a chance we may use fewer than 10 test kits. But it’s not guaranteed. Whether we will end up with savings depends on the proportion of pooled samples that test negative. A key calculation is the probability that when forming groups of 10 people, all ten are not infected.

Suppose 10 percent of test-takers have the virus. Then, in an independent pool of 10 people selected at random, the chance that no one is infected is (0.9)^10 = 35%. Thus, we expect 35% of these pools of size 10 to test negative, resulting in a saving of nine kits each. For the other 65% of pools, we run one extra test relative to the base case of 10 tests for 10 individuals. On average, we save 2.5 tests for every 10 people. The saving rate is 25%, a quarter fewer test kits. Not bad!

***

For the U.S., is this development coming too late? The group testing strategy confers savings only if the virus is not too widespread. If there are too many infected individuals, then most pools will have at least one infected person, and instead of saving kits, we may burn up even more kits than normal. Let’s repeat the previous calculation but assume 20% of the test-takers have the virus. In this case, about 10% of the pools have no virus while 90% comprise at least one infected sample. On average, we break even – the expected number of tests using group testing of size 10 is ten tests, the same as one test per person. All for naught.

For those interested, you can analyze different pool sizes.

***

Breakeven is not neutral though. If I were the decision-maker, I’d need the group testing strategy to show a clear, positive benefit. That’s because the group testing method has several non-obvious costs.

First is the cost of administration. One test per person is simple to execute. Group testing requires splitting samples, storing samples until the pooled test result is available, matching samples, etc. Much better book-keeping and tracking are required.

A second cost comes from potential errors due to complexity. Samples can get contaminated, mixed up, or lost.

Third is its impact on test-takers. More blood or sputum must be collected in anticipation that some of the samples will be tested twice. Also, it will take longer for the results to be confirmed.

The fourth cost is more false negatives. Imagine a pooled sample with exactly one infected individual. By taking one-tenth of each sample and mixing them together, we dilute the viral load of the infected individual 10 times. It is possible that the pooled sample – and thus all 10 individuals - will be declared negative. If the infected person were tested alone, the viral load would have been high enough to return a positive but the pooling causes a false negative. Such false negatives may make kit savings a pyrrhic victory.

***

Significance magazine has a recent article about group testing.