Big Data, Plainly Spoken (aka Numbers Rule Your World)

A story about poop, and the scientific method

These headlines flooded the media last week. I'm listing the first five results selected by my search engine.

University of Arizona is screening for COVID-19 via dorms poop (New York Post)

University of Arizona says it prevented COVID-19 outbreak by testing students feces (Daily Mail)

Poop test stop COVID-19 outbreak at University of Arizona (Science)

Poop becomes secret weapon to detect COVID-19 cases in dorms, says University of Arizona (National Post)

Researchers at the University of Arizona say they stopped a coronavirus outbreak before it spread by testing students poop (MSN)

***

The sequence of events that reportedly happened is as follows:

1. As part of the re-opening program at University of Arizona, the school tests wastewater from 20 on-campus dorms for presence of the coronavirus.
   
   
2. Last week, one sample came back positive, and was traced to a specific dorm (Likins Hall).

3. Researchers notified school administrators and convinced them to test every resident of that dorm.
   
   
4. Of the 311 samples collected from residents, researchers found 2 cases, implying a positivity rate of 0.6%.

5. The two students, both asymptomatic, were isolated.

6. Researchers concluded that they prevented an outbreak in that dorm.

This is good news for managing campus re-opening. Testing sewage water should be in the toolkit, used in conjunction with other tools. But establishing the cause-effect relationship is not as simple as the Arizona researchers made it out to be.

This story is also a case study of how it is easy to make a scientific claim but it is hard to prove it.

***

The university tested about 10,000 people, as they checked into campus housing about 10 days ago (other returnees and off-campus students are tested only if they request it). They found 46 cases. That's a positivity rate of 0.5%. On average, if we select 311 people randomly, we should expect 1.4 positive results, so it is normal to find one or two cases if that is the prevalence.

Recall that they discovered two cases in 311 tests (0.6% rate) in Likins Hall. The scientific question is whether that positivity rate is abnormal. If they had tested samples from another dorm - one for which the wastewater testing turned out negative, and found no positive cases, that would strengthen the causal claim. Did wastewater testing accurately detect the presence of virus in a specific building? The answer is yes, conditional on the key assumption that the prevalence of virus in dorms that passed sewage testing is zero.

Since the residents of the other dorms were not tested, we don't really know the answer. That's why it is an assumption of the analysis. There are reasons to believe that the prevalence is low but not zero: (a) the residents started checking in on August 21 and the wastewater testing program was announced on August 25 (link) so it is possible that residents have gotten infected in the interim (b) it's likely that at least a few residents have not yet completed arrival testing, and they may be infectious (c) up to two guests per resident are allowed in the dorms, and testing of off-campus students is voluntary (d) vendors (including movers) can bring virus into the building.

***

Now, consider the tracing of sewage water to Likins Hall. How accurately can this determination be made? This depends on where the sewage water is collected. In a notice to on-campus students, the university made this disclosure: "This [wastewater] testing does not identify individuals or even floors of the dorm or building that might have infections." This statement directly contradicts the published story in which pinpointing the building was crucial in the chain of events.

I also wonder how they deal with any mixing of sewage, or possible environmental contamination.

***

The consultant hired to supervise campus re-opening told a reporter: "if we had missed it, if we had waited until they became symptomatic and they stayed in that dorm for days, or a week, or the whole incubation period, how many other people would have been infected?" This interpretation assumes that the virus found in the sewage water came from the two students who subsequently tested positive, and were still infectious.

Wastewater testing is beneficial only if it detects active infections. So establishing timing is essential. It is possible that the infected sewage came from past infections but not from the two active cases - there is still a lot that is unknown about virus shedding in feces, only a fraction of infected people do it. I'm not saying this is what happened but the coincidence of (a) virus found in sewage testing and (b) cases found in building does not prove that the infected sewage came from those specific cases.

***

There are three types of university responses to Covid-19. The most risk-averse group has moved all instruction online. The second group is attempting a return to the classroom but will require comprehensive testing. The third group re-opens but is looking for shortcuts to avoid having to test everyone. Sewage testing is of interest to this final group of schools.

The Arizona case study provides some hope that testing wastewater may reduce the amount of individual testing necessary to monitor and control infections on campus. Conceptually, this is related to group testing (explained here), replacing the usual nasal swab tests with sewage tests in the first stage.

More work needs to be done to establish the cause--effect. For example, controls should be included in the study, testing a dorm where the sewage came back negative. Another reason for controls is to measure false negatives. A deeper understanding of the "schedule" of virus shedding during the infection period is necessary. Measures to rule out contamination help reduce false positives. As with other tests, the scientists should publish the accuracy characteristics of the wastewater test.

Answers on excess deaths, test accuracy and quick testing

The recent posts have sparked some nice discussion here and elsewhere. This post collects additional comments I've since made.

Excess deaths caused by the lockdown

There was a verbal scuffle when reporter Jonathan Swan interviewed President Trump a few weeks ago about how to measure deaths due to Covid-19. Should we look at deaths per case (mortality rate) or deaths per million? In two recent posts (1 and 2), I discussed the visualization and modeling of "excess deaths," which is a better way of measuring the pandemic's impact than both of those popular metrics.

The starting point of an excess death analysis is neutral and apolitical. Based on historical patterns, we know that X people tend to die in the weeks from March to the present. Instead of X, the actual deaths (by any cause) have spiked in most U.S. states by 50% to 700% of the normal level in the worst weeks of the pandemic.

These "excess deaths" are not explained by any of the known causes. A majority of these deaths can be explained by deaths attributed to Covid-19, a novel cause of death. Most scientists believe that the unexplained excess deaths are probably under-counts of Covid-19 deaths. Should convincing data emerge, other causes of deaths could be accepted.

Some readers asked whether "excess deaths" may be caused by the lockdown rather than the virus. The lockdown does not kill anyone directly. What they mean is the side effects of the lockdown, such as more suicides from depression due to lockdown, or more cancer deaths because the patients were turned away from hospitals.

This is a logical argument but let me explain why this argument seems like a dead-end:

• The biggest challenge is the scale of the gap. A 50-percent to 700-percent gap is not easy to reconcile. Cancer is 20 percent of all U.S. deaths in 2017 (source). Let's say half of those occur between March and July. Then, if 2020 is similar to 2017, we should expect 300,000 deaths during those months. If we want cancer to explain 100,000 excess deaths, we're saying the lockdown caused cancer deaths to rise by 33 percent. That's a huge jump for a well-known disease that requires investigation. And we still have another 100,000 excess deaths to explain.
  
  
• Other than rhetoric and anecdotes, I haven't come across any serious attempt to quantify these lockdown side-effects. If some cancer patients have died because they are turned away from hospitals, then they should start investigating people who died at home from late-stage cancer. How many such deaths can we confirm? It's plausible that such deaths exist but I doubt this will make a dent in explaining the 200,000 excess deaths.
  
  
• Most cancer patients who died at home because they couldn't schedule surgery at the hospital during the pandemic were seriously ill, and would have died eventually of cancer. The side effect of the lockdown is accelerated deaths. In the world of "excess deaths," an accelerated death contributes to one positive excess death now and one negative excess (avoided) death later. After lockdown is called off, we should experience a reversal of these excess deaths. Will we find hundreds and thousands fewer deaths from all non-Covid19 causes in the aftermath? I don't think so.

 

False positives in the NFL

News broke that many recent positive test results on NFL players turned out to be false positives. This has been described as a "testing debacle."

First thought: false positives are the price to pay for being able to trust negative results. Notice that the NFL is not re-testing negative results; the NFL retests positive results. In order to afford such a luxury, the negative results must be rock solid. If the test comes back negative, the person is more or less certain to not be infected. This is how a good coronavirus test should behave. A false-negative mistake is an infectious person who spreads the virus unknowingly. For a diagnostic test to have fewer false negatives, it will have more false positives.

Second thought: testing errors, when interpreted in a broad sense, can be caused by operational mishaps. From what has been disclosed so far, it appears that the NFL contracts a single private testing company to perform all coronavirus testing, and that business operates labs in multiple states. All the false positives came from the New Jersey lab. This might be traced to say a contamination of equipment at the lab, just for argument [PS. That's the official reason given by the lab now.]. If true, the inaccuracy has nothing to do with the chemistry; the error came from lab operations.

All accuracy metrics published by test developers assume zero operational mishaps, 100 percent complicance, no gaming, etc. I brought this important point up in Chapter 4 of Numbers Rule Your World (link) - an entire chapter devoted to the complex issue of testing accuracy.

This is why when discussing group testing in this post, I singled out operational mishaps that could happen because group testing adds complexity.

 

Wishful thinking on treatments

The U.S. President has endorsed plasma therapy as the latest best therapy for Covid-19. The FDA has given it emergency use authorization. We should be clear that this is temporary, pending the results of proper scientific testing, with randomized controlled trials that prove plasma works.

I understand the media love the feel-good stories about people who insist that plasma saved their lives. But these survivors don't know. Because if plasma indeed saved their lives, then had they not been given the blood transfusion, they would have died. But no one knows that since they got the plasma!

It is entirely possible that plasma worked for some specific cases. The point of science is to establish that it would make a difference for enough patients to matter. [PS. The FDA Commissioner has already sort of back-tracked.]

Elsewhere, on college campuses and in professional sports, the testing labs are turning to quick saliva tests to scale up testing. This is of course a good thing. There are several such tests developed (I think I've read about those made by Yale, University of Illinois, Rutgers, etc.). The developers of these tests make many magical claims (see here). The quick test is supposed to be just as accurate as the nasal swab PCR test; much faster than that test; can pick up even lower amounts of virus; can provide a numeric scale of the amount of virus, not just its presence or absence; etc.

I find these claims hard to believe, except the efficency claim. Saliva should contain less materials mixed with unrelated fluids. It's probably less accurate. But the benefit of scale is obvious. Since it's used primarily for "surveillance testing", let's hope it doesn't generate too many false negatives, who become silent carriers.

A testing mess across the pond

Controversy has engulfed the A-Level examinations in the U.K. The A-Levels are university entrance level exams of subjects (math, English literature, history, economics, computer science, etc.). Universities conditionally admit students with minimum grade requirements for relevant subjects. The A-Level results are often the most important admissions criterion, by far. Their closest American cousins are the SAT Subject Tests, although in the U.S., these tests are not widely required, and no college conditions admissions on test scores.

The coronavirus pandemic threw a wench into the exams system. Students could not sit for these exams in 2020 thus the authorities concocted a statistical scoring model to assign grades. The model predicts what grade the student would have scored had s/he sat for the exam.

Roughly speaking, each grade has three components: a baseline score that applies to every student, a school factor that applies to all students of the same school, and an individual factor that applies to each student within the same school. The school factor is based on performance of past cohorts of students in the A-Levels. The individual factor is based on the student's prior performance in other public examinations, and on a teacher evaluation.

As part of this exercise, teachers are asked to grade their own students, which have been described by students as their "predicted grades". On results day, a lot of students and their teachers were disappointed by the assigned grades. They found that the model's predicted grades are lower than their teachers' predicted grades. There was a huge outcry about so-called downgrades. The statistical modeling is attacked as "utterly unfair and unfathomable", hurtful to young minds, inconsistent, disappointing, lacking transparency, etc. (See this BBC Live Feed for reactions.)

***

The term "downgrade" interests me a lot. We have two predictions of an unknowable quantity, how the student would have performed had exams not been cancelled. These predictions sometimes don't match.

From the students' perspective, the conditional offers from universities set targets for A-Level grades. Their teachers told them what grades to expect, i.e. the predicted grades. The official grades they receive are evaluated relative to these teacher assessments. If the official grade is lower, it's a "downgrade" and if it's higher, it's an "upgrade".

I believe the exam administrators are shocked by this reaction. They probably regard the teacher assessment as one input in a complex process that generates official grades. They probably see the model's assigned grade as the "truth". It's the teacher assessment that may be over- or under-estimates.

This situation reminds me of my recent post on how the business community interprets metrics about the economy. They take their expectations - potentially generated by models - as the reference level. Official statistics on unemployment, inflation, etc. are evaluated relative to these expecations, rather than the other way round.

***

The media has by and large sided with students and their teachers. In this post, I provide the other side of the story, the challenges faced by the statistical modelers.

There are limited capacity at U.K. universities, and particularly at the more competitive schools. Admissions officers set A-Level grade requirements, the conditional offers, based on historical distribution of grades, and their historical yield rates. If the exams were not cancelled, the administrators are reasonably confident that the grade distribution in 2020 would have been similar to those of recent years. 

Not surprisingly, the distribution of grade predictions by teachers exhibits grade inflation. The exams authority, Ofqual, published the following chart:

The yellow line represents the proportion of A and A+ grades for the last four years. The orange dot labelled CAGs represents teacher assessments for 2020. Ignoring the 2020 yellow dot for the moment, you see that teachers in aggregate assigned 12.5 percent more A and A+ grades than the 2019 results. As a ratio, this is 50 percent more top grades than the year before!

The so-called downgrading describes the gap between the orange dot (37.7%) and yellow dot (27.6%) in 2020. The teacher assessments were fed to the statistical model which brought the proportion of top grades down to roughly the historical level. In fact, the proportion of top grades issued by the model is 1.4 to 2.4% higher than in recent years.

What this means is the attack on the model is rather misguided. I'm sure there are problems with the model, as with any model, but no model can escape the constraint that the proportion of As and A+s should be about 25 percent. That means all models will issue a lot of "downgrades". They differ by which students are downgraded.

Whoever gets downgraded has a grievance. We haven't heard complaints from those who got upgraded (about 2 percent of grades), and probably won't. Such asymmetric complaints tend to make models worse. Deficiencies are remedied along one dimension but not the other, which introduces a "correction bias" to the model.

I cover this issue in Chapter 2 of Numbers Rule Your World (link) in the context of credit scoring models. When the credit bureaus allow us to check the data, we will challenge anything that might reflect negatively on the credit report, but if the bureau makes a mistake that benefits us, for example, if the report failed to record a prior delinquency event, most of us will not volunteer this information (that will certainly lower our credit score).

***

Another factor that no model can escape is the differential quality of schools. Students of some schools have historically done better at A-Levels and other public exams. It is a bit unfair that a student's grade is partially determined by the quality of the school s/he attends.

In fact, standardized testing solves this problem, except the pandemic wiped that away in 2020. To a large extent, all models assume the future is like the past. Existing inequities and biases get propagated.

 

A data communications starter kit for university re-opening

 

It's August, and many universities in the U.S. are attempting to bring some or all of their students back for in-person instruction. Within weeks, sometimes even before the first class, several campuses have already reported dozens or even hundreds of confirmed cases of infection by the contagious novel coronavirus. School administrators are justifiably worried that the virus, once it gets transmitted to older or more vulnerable community members, would cause grave illnesses and even deaths.

It's heartening to see that campus groups are demanding more data transparency from their schools (e.g. this open letter from public health professors at the University of Georgia). In response, some universities have rolled out dashboards, which is just a fancy name for data reports, iconically sprinkled with statistical graphics.

Nevertheless, the standard practice of minimal reporting of campus-wide totals represents much motion but scarce substance. Serious analysts will learn next to nothing from tallies that mix entry testing with ongoing surveillance testing, multiple tests of the same individual with tests of different individuals, etc. In addition, the focus of data reporting must expand from reporting outcomes (how many cases are there?) to reporting operations (are people complying with testnig? are people wearing masks?) in order to reveal opportunities for mid-course correction.

In this post, I outline a comprehensive plan for data reporting that should arm the university communities with the necessary data to assess the level of risk on your campuses.

I call this the aspirational plan because few school administrators will commit to this degree of transparency. Data reporting is political - whether it happens at schools, businesses, governments or organizations. It's quite possible that some officials secretly monitor some of the listed items. That'd be better than not tracking them at all. It falls on campus activists to push their administrators to collect the right data, and release more information.

As you read through this plan, I hope you develop some respect for the profession of data reporting (in business, this usually belong to the business intelligence team). This is not some boring job but a creative and thoughtful endeavor of utmost importance.

***

The ultimate goal of any data reporting plan is to keep the campus safe and open after it re-opens.

In putting together the aspirational data reporting plan, I start by listing the keys to sustained re-opening, based on my interpretation of statistical models built by Cornell and Georgia Tech researchers. Click these previous posts (here and here) to learn more about these models.

(A) Stem the import of virus onto campus.

This can be broken down into two objectives:

(A1) Ensure there is no virus on campus at the start of the semester. This is typically accomplished through comprehensive testing of all returning people, and isolation of the positive cases. [Notice that many Covid19 measures are as aspirational as my data reporting plan. We hope there is no imported virus but everyone assumes leakage is inevitable.]

(A2) Ensure there is no import of virus onto campus during the semester. Various measures are implemented to identify visitors and require self-reporting of health statuses.

(B) Snip transmission chains aggressively:

If the virus seeps onto campus (a certainty), we need to

(B1) find the infected cases quickly (bearing in mind silent spreaders), and

(B2) prevent the virus from spreading widely.

If a campus achieves both of those objectives, the mathematical models predict that infections can be contained during the semester, and students won't be sent home.

***

A data reporting plan must address the above two success factors separately. Over-aggregation of data is an adversary throughout the report design process.

How to monitor the import of virus at the start of the semester

This is one of the simpler tasks. Most dashboards include the number of tests, and the number of positive results. These metrics are not sufficient.

The tallies must not mix together entry testing and ongoing surveillance testing. Entry testing should have a section of its own. In order to interpret the number of tests, we must report the number of returners. It's even better to also count the number of unique individuals tested. Some schools require those already on campus to get tested. If so, those data should be reported in a separate section, not mixed with the data for returners.

[If your school isn't doing any entry testing, or some washed-down version of it, you're dealing with a different problem. Allowing virus to seed itself around campus is the surest way of enabling outbreaks; this is the key epidemiological lesson from the past few months.]

How to track the import of virus throughout the semester

The next task is to track the import of virus during the semester. We've heard many college presidents and deans of student life issue warnings blaming mixing of on-campus and off-campus groups for community spread, and yet we've not seen any university with a data reporting plan capable of tracking this crucial factor. In fact, the Cornell model (see here) assumes, gasp, no mixing of on- and off-campus people so in a sense, this is a blind spot for administrators.

To simplify the language, I'll use the word "visitors" to include anyone showing up on campus who does not reside on campus. The largest groups of visitors are off-campus students and staff, and guests (such as academic collaborators and vendors). At the minimum, a data reporting plan must include an estimated number of visitors circulating on campus. Roughly speaking, this is a function of entries and exits from campus, which are monitored. (Please talk to statisticians on your campus.)

In other words, if the data report contains the number of entries to campus (or campus buildings), the number should be broken down into visitors versus non-visitors. Lest you worry, we don't care about the names of people entering and exiting; for example, if one person enters 10 buildings, we count 10 entrances because we want a measure of contacts, not individuals.

"We don't have the data" is the excuse frequently used by those who prefer not to see the data. Having a reasonable estimate of the number of visitors circulating on campus is so critical to monitoring on- and off-campus mixing and any course correction that administrators should sit down with statisticians to come up with a plan. (Or, hire me or other consultants. We're not that scary!)

How to find virus that has already entered your campus

The next section of the report deals with finding any virus that has entered the campus. This is another tally of test results. The biggest concern is compliance, an issue I raised when discussing the Cornell and Georgia Tech models (see prior posts here and here).

Some schools including Cornell and Georgia Tech have adopted a comprehensive (aka surveillance) testing plan. Most schools take a "see no evil" approach, asking people with symptoms to volunteer for testing (already failed at the national level). Some schools use a limited random testing program. For comprehensive or random testing, the school should report how many invitations were sent out, and how many complied. For self-driven testing, the proportion of unique community members who have been tested in the last X days may be of interest.

It's impossible to interpret aggregate test data. Any serious report about testing must break down the tallies by the type of testing (comprehensive, randomized, self-reported). Other useful breakdowns are type of affiliation (undergrads, grads, faculty, staff, etc.), age groups, gender, subdivisions (e.g. schools). Such analyses allow administrators to see early warnings and take action, such as when they discover that certain communities are flaunting testing requirements.

How to measure community spread

The last section of the reporting is also the most interesting: we want to measure community spread. The standard practice of counting number of confirmed cases is useless. It's like standing by hopelessly watching a runaway train. We focus on the process of spreading, not the outcome of spreading. The goal is to estimate the number of contacts.

The starting point, and easiest, metric is class attendance. The entries and exits from buildings, already mentioned, should be a part of this.

Tried-and-true data collection tools are very useful here. Literally count the number of people walking past selected hotspots on campus. Count the number of masks. I'm sure you can find a few statistics or data science students willing to help out!

In addition, new-fangled AI tools can help. There are dozens of businesses selling (out your) location data collected from cellphones and apps. Surveillance camera footage can also reveal traffic intensity of pedestrians, cyclists, and vehicles. If this is too esoteric, head over to the data science, computer vision, or engineering departments and I'm sure they'll be more than happy to assist.

The data report should not print all these details. It should track a few metrics, such as class attendance, head counts, mask counts, and vehicle counts. These tallies can be samples (such as at hotspots), as we are only interested in trending.

How to monitor operating capacity

For schools that have credible contact tracing and quarantine procedures, there should also be a separate report on relevant metrics. The number of people under quarantine is important but also the average or median length of time spent under quarantine. The number and types of contacts shed light on the transmission chains.

***

The last piece of the puzzle

If any school can roll out the aspirational data report, I'd be overjoyed. The community members will have so much useful data, and administrators will be able to course correct based on trends. As my readers should expect from me, I must point to the elephant in the room.

I must admit that even this aspirational plan has one missing element. It does not measure what economists call the "externalities" of the re-opening. What is the impact of re-opening on the public health of the surrounding community, the people who are not affiliated at all with the university?

People live in areas, or even the same buildings, popular with off-campus students and staff. These community members may come in contact with students or staff at neighborhood restaurants, supermarkets, public transport, or by just walking past each other. From what we know about the coronavirus, outbreaks on campus easily spill over to neighbors. Some coordination with local public health authorities will be needed to address this difficult problem.

To put it bluntly, even if there are no deaths on campus, some hospitalizations and deaths in the surrounding community are by-products of the re-opening. Universities should not turn a blind eye to those.

 

 

Nice explainer from Georgia Tech about their re-opening model

Georgia Tech professors explain how they model school re-opening in this Youtube video. They did a great job simplifying the materials for a general audience.

You'll find that the Gatech model has the same stylized findings as the Cornell model. (However, these professors recommend starting the semester fully online, unlike the Cornell team.) I've already reviewed the key learnings in this prior post - the professors confirm the following:

1. It's super important to suppress imported cases at the start to the lowest possible through entry testing.
2. Then, they prescribe comprehensive testing to nip any community spread. (I've called for broad-based testing from the very start but CDC continues to not recommend it.) Cornell, unlike many other schools, also recognized that random samples are not enough. The Georgia Tech professors explain this nicely - they are using testing as a mitigation tool. (Sampling is a measurement tool.)
3. The testing needs to be extremely aggressive. The model concludes that everyone should be tested at least once a week. This is the same frequency as in the Cornell model.
4. Georgia Tech "cannot" mandate testing thus the comprehensive testing will be voluntary. Like Cornell, Georgia Tech has bottomless faith in its student body and larger community because based on some survey results, the professor said they expect "over 95 percent" compliance.
5. People testing positive will be isolated from the community.
6. They hope there will be close to 100% compliance on mask wearing.
7. Unlike Cornell, they are less invested in contact tracing. They hope that testing and isolation will be sufficient.

You can see that the results of this model - like any model - are based on a host of assumptions, many of which on human behavior. After a week or two of re-opening, the assumptions are already looking shaky. Commenters on Twitter are reporting testing compliance of less than 33% at Georgia Tech, and some Cornell students are said to have moved into dorms without having completed entry testing. (Note: these are unverified.)

As a fellow modeler, all I can say is: innocence lost!

***

There are a few new things I learned from the video:

1. These colleges are spending a fortune on testing, conducting thousands of tests per day, at roughly $10 a day. I get the sense that this price is possible only because Georgia Tech has academic staff with relevant skills, and if they must buy test kits from commercial vendors, the cost would be even higher.
   
   
2. They admit that the goal of testing once a week may not be feasible initially, and once every two weeks is more likely. They explain some of the regulatory and operational hurdles of getting the testing program up and running.
   
   
3. The current estimate for R0 has gone up to 3 (at least for the state of Georgia). This virus is highly contagious. They show that if the virus is allowed to spread through the community (i.e. what the herd immunity advocates want), there will be potentially dozens of deaths among faculty, staff and students.
   
   
4. A calculation of optimal class size shows how daunting it is to operate in-class instruction. In a classroom of 25 students, there is about 50 percent chance that at least one of them is infectious. (R0 of 3 means the first infectious person will transmit the virus to three others.)
   
   
5. For me, the gem of the slides was toward the end (at 43:57). They present recent evidence of pre-symptomatic transmission (work done by a team from UT Austin and others). Of course, we've been talking about silent transmission for a while (the latter part of this link) but somehow to this day, CDC continues to say that only people with symptoms should get tested. The evidence concerns a concept called "negative serial intervals". Imagine a transmission chain: person A transmits the virus to person B, who transmits it to person C. We can measure the time interval between A and B, and between B and C. A negative interval is when B gets sick before A does. This seems impossible.   But that's because we forget the pre-symptomatic carriers. Person A infects B while not showing symptoms; person B shows symptoms quickly after infection. Person A eventually develops symptoms. So the presence of negative serial intervals is proof of pre-symptomatic transmission. (The same dynamic describes asymptomatic transmission, except person A never exhibits symptoms, so without broad-based testing, person A would not be observed.)
   
   
6. Note that the above analysis is made possible by a robust testing and tracing program. If we don't know who's infected and we don't know with whom the infected have been in contact, we can't estimate the serial intervals. The data show person B getting sick before person A; to reverse this order, the researchers must have learned that A got infected first - for example, A attended a campus party, B has stayed in her off-campus apartment, except the one time she invited A over for a dinner.

 

One week was all it took

In my review of the Cornell report justifying full re-opening of college campuses in the Fall, I discussed their rationale - an ambitious strategy to build a bubble on campus through a rigorous test, trace and quarantine program - and also expressed my skepticism about their myriad assumptions about human, especially college-aged, behavior. It seems like college administrators would be anxiously counting the days before they have to shut things down.

***

In the meantime, I occasionally come across the re-opening plans by other schools. Princeton and Harvard both opted for 100% online instruction. Many of the NYC schools take a "see no evil" approach. It appears that their administrators trust students, staff, faculty and visitors to self-report their symptoms to an app. If they say they have no symptoms, they are cleared to enter campus buildings. There are no systematic testing, tracing, or quarantine.

***

The Cornell model has already been proven more responsible, even before most colleges have re-opened.

The University of North Carolina commenced fall classes on August 10 with few of the requirements of the Cornell model. One week was all it took for the school to backtrack, and switch all undergrad classes to online only.

What happened during this week, according to this scathing editorial in the Daily Tar Heel:

• August 10 (Monday), school re-opened.
• By August 14 (Friday), two clusters have surfaced on campus residences.
• By August 15 (the first Saturday), a third cluster has been located at a frat house.
• By August 16 (the first Sunday), a fourth cluster in another dormitory.
• Administrators refused to disclose an official count for the total number of cases, citing "privacy". This sounds familiar. The UK government also said, in the early months when the coronavirus ravaged care home residents, privacy was the reason why it could not disclose the number of deaths at care homes.
• By August 17 (Monday), UNC disclosed that they had 177 students in isolation and 349 other students under quarantine. (Latest report here). Administrators ended the re-opening gambit, moving all undergraduate classes online.

Recall that in the Cornell model, the expected peak capacity for quarantine during the whole semester is 700 students. In one week, UNC has almost reached that volume. (UNC has slightly more undergrads than Cornell.)

Cornell could plan for fewer people under quarantine because of many tactics designed to keep infections contained. UNC deviated from the Cornell model in various ways. People returning to campus were not tested upon arrival. There isn't an ongoing compulsory testing program to find cases.

***

The situation is fluid even for NYC schools. I noticed that NYU added testing and quarantine policies last week (link). But these requirements are far less stringent than the Cornell model, which tests every person once a week, traces contacts of all cases, and quarantines all contacts. NYU tests those who come forward with symptoms, and will also conduct random testing although it has not stated how large that sample is.

What happened to the nation is about to get replayed on college campuses. Those schools that do not take test, trace and quarantine seriously will feel good about themselves for some time, and when cases start to pile up, they will realize the virus has already spread throughout the community. At that time, they have few options but to shut down on-campus instruction, and perhaps housing. The infectious students may then move off campus, taking the virus elsewhere.

 

 

 

 

Excess deaths, reporting delays, statistical adjustments, misleading headline

The New York Times published a very nice presentation on "excess deaths" in the United States from March through July. (link)

The following chart shows how public health policies utterly failed in many Northeastern states in the April and May.

In Connecticut, for example, in the middle of April, the number of deaths exceeded expected deaths by 2.6 times. In Massachusetts, around the same time, actual deaths exceeded expectation by 2.3 times.

A lot of work went into generating these charts - starting with meticulous data collection from each state, and as much as possible, reconciling how metrics are defined differently across states. Then, there is a bit of statistical modeling.

Modeling is required because "excess deaths" is - like most statistical quantities - not directly observable. Excess deaths is the difference between actual deaths (from all causes) in the presence of the Covid-19 pandemic and expected deaths (also from all causes) in its absence. The former is directly measurable but the latter is not.

To establish how many people could have died during a specific week in Connecticut if the novel coronavirus did not exist, statisticians look at deaths during the same week in that state in past years. Over a number of years, one should find some statistical consistency in the death rate. This illustrates the power of averaging.

If 2020 were a normal year, the chart should look something like this:

The above line is for Hawaii, which, because of its remote location, has not been affected much thus far by Covid-19. What you see is the actual weekly deaths hovering around the baseline (corresponding to the historical average), sometimes rising above, and sometimes dipping below.

The line for Connecticut, though, looks nothing like that for Hawaii. The most important feature is that the entire line sits above the baseline, meaning that there have been excess deaths for every week since March.

***

Ready for even more modeling?

For most states, the time line reaches the last week of July, which may surprise you. The first part of excess deaths - the observable part - counts all causes, obviating the need to verify coronavirus test results. The measure relies on death certificates. But it may take weeks to months to receive data on death certificates so how does NYT have estimates of excess deaths up to the week prior?

Does NYT have a crystal ball? In a sense, yes. It uses a statistical model to "top up" the incomplete deaths data in the most recent weeks. It turns out that actual deaths are not completely observable always. There is a reporting delay which means that the data for more recent weeks are incomplete. The data age like fine wine - the older the data, the more complete.

The NYT utilized a similar strategy as the model for projecting expected deaths - assuming historical patterns hold. The CDC keeps track of the amount of reporting delay. It might know that typically, 20 percent of the data showed up with one weekrepo's delay, 50 percent with two weeks' delay, and so on.

There is a crucial difference betweeen these two statistical models, though. The first model projects expected deaths in the absence of the novel coronavirus. The second model tops up the actual deaths reported at the time of analysis, essentially projecting the number of deaths already occurred but not yet reported. These actual deaths are in the presence of the coronavirus. This is the crucial difference. This second model has a further assumption - that reporting delays have not been affected by the pandemic.

The NYT team disclosed this assumption clearly in the text. This is what they mean by the following sentences:

Even with this adjustment, it's possible there could be an underestimate of the complete death toll if increased mortality is causing states to lag more than they have in the past or if states have changed their reporting systems.

It's quite likely that the volume of deaths, the budget pressure, and political interference have changed the pattern of the reporting delay.

The reporter then argued that the adjusted estimates are still better than the unadjusted ones. One can't refute that argument. Of course, this discussion assumes that the chart must show a time line leading to the present. The designer can choose to show only those weeks with virtually complete data.

Or, we can bring in a third model. This one looks at the early weeks of the pandemic, for which we believe almost all of the death-certificate data have been recorded. We can then estimate the error of the second model that serves to top up the actual deaths.

***

NYT's excess-death analysis has found over 200,000 more deaths (from all causes) in the U.S. between March and July than expected based on historical patterns. Given the U.S. counted about 140,000 deaths due to Covid-19 during this period, there have been about 60,000 unexplained deaths so far.

I find the NYT headline, "The true coronavirus toll in the US has already surpassed 200,000," alarmist. The analysis did not support this conclusion - unless they make the further assumption that the only factor causing excess deaths during those months is Covid-19. That is plausible but not fact-based. A more correct headline should be "Deaths due to Covid-19 may have been undercounted by up to 40 percent." That's also shocking and sobering.

Is the U.S. speedboating? The merging of business and politics

In response to the note about the time lag between infection and deaths, Mark P. pointed me to an old blog post of his that features another example from the business world.

Mark's example comes from money lending. In that world, a key metric is the "charge-off rate". This is defined as the amount of new charge-off (i.e. bad loans) divided by the total amount of outstanding loans. Both numbers are computed as of the current date. What's the problem? Loans don't typically go bad right away. In fact, as Mark outlined, the industry won't even write off loans until it is already, say, 90 days past due.

And yet, investors evaluate lending businesses based on this "charge-off rate". How is this being gamed? Mark explains:

Let's say I book an account that goes delinquent after one year. That was a bad deal for the bank – – it lost money due to that decision – – but for one year, having that account actually lowered the bank's charge-off rate. Eventually, of course, this will catch up with the bank, but the reckoning can be delayed if the bank continues to book these bad accounts at an increasing rate.

I'd add that (a) the decision-maker has earned all his/her bonuses, and may not even be around for the reckoning, (b) the reckoning can be very far off so long as the bank has a pipeline of these risky creditors to lend money to, and (c) in the brave new world of massive corporate bailouts, the reckoning might show up as free money, more bonuses, and a stock price spike.

I'd further add that this practice - apparently having a nickname of "speed boating" - (a) is legal and (b) violates no accounting rules.

Of course, the whole mess can be avoided if the investors focus on cohort-adjusted charge-off rates. Computing the adjustment does require better data management.

***

Business executives are extremely good at crunching numbers. They may not know statistical theories, but they figure out what metrics investors are concerned about, and how to dress those numbers up. The U.S. President is a business exeuctive. He wasn't joking and he is not wrong when he said Covid-19 cases go up with more testing. (Don't forget deaths go up with more testing, too. And it's not how many you test but who you test that is the key to understanding those numbers.)

Let's talk about data backlogs

Yesterday, I featured some weird things I saw in Texas's Covid-19 data. A host of issues are officially recognized and flagged by sites like Covid Tracking Project. Reasons for bad data include software upgrades and data backlogs. News also came out that California "found" hundreds of thousands of unreported cases (link). This was blamed on a server outage necessitating code revision that stayed longer than it's welcome resulting in a data backlog. (I'm not joking; this came straight from the official explanation.)

If you haven't already, find a friend who has worked with databases, and ask if s/he believes these stories. Or, just keep reading...

 

Data Backlogs

If your job is to maintain a data pipeline that merges data from many sources, you will at a minimum create an audit report that tells you how much new data have been processed from each data source every day. If that is too much effort, you will make a report that tells you whether each data source has sent their data (yes/no) at the agreed-upon time each day. There is no database administrator who haven't made such a report. If someone hasn't, rest assured that these audit reports appear in the inbox of the CEO, the day after the first data processing error shows up, which probably occurs during the first week on the job.

It's very common, especially in modern, loosely managed distributed databases, that some minute or hour of data went unprocessed. Those are sometimes hard to detect from daily or weekly statistics in the audit reports.

It is simply inconceivable that hundreds of thousands of missing records have escaped attention. I laughed uncontrollably reading this delicious sentence: "the state hadn't realized they weren't receiving data from one of the largest commercial labs for five days." Oh, and the White House press corps didn't realize the President hasn't tweeted for five days. Missing multiple days worth of data goes beyond incompetence.

California is a repeat offender. I previously wrote about another such situation that happened on April 22, when on a single day, the state published 10 times the usual amount of data - to add insult to injury, this batch of data had one-tenth the usual number of positives.

 

Software Upgrades

Texas announced a data blackout on August 2, and then released two days' worth of data on August 3. A scheduled software upgrade was the official reason. But a software upgrade does not cause two days of data to get mixed together.

Data typically come with many dates. Two key dates are the date of the event (e.g. date of death, or date of diagnostic testing), and the date of the report. The reporting date is generally different and later than the date of the event. The systems at hospitals and clinics - the source of testing data - apply a time stamp to mark when the data are reported to the public health agency. The agency adds a time stamp recording when it uploads the data to its databases.

A software upgrade at the Texas public health agency isn't going to erase dates collected at hospitals and clinics. There is no reason why they can't save the data files for August 2 separately from those for August 3. Most likely, the hospitals and clinics probably have those files saved just like that, with the date of reporting as part of the file names. There is no logical reason why the two days of data must be aggregated.

Why couldn't the Texas public health agency divert the August 2 data to a temporary hard drive? Why couldn't it ask hospitals and clinics to send two files? Why couldn't it parse the dates in the data files to separate out the two days? Any of these solutions would have just delayed the release of the data but not scrambled two days of data. This is not rocket science. This behavior goes beyond incompetence.

***

Since the states blame these problems on server outages and software upgrades, they should name the vendors of these technologies. Which database needed an upgrade and why? What server broke down? If the upgrade was "scheduled", why wasn't there a contingency plan for storing the data temporarily in a different place? What measures are taken by the new technologies or vendors to prevent these same problems from happening?

Bad covid data: feature or bug in Texas?

Let me show you how bad the state of Covid-19 data is in the U.S.

Recall my post from two weeks ago about Covid-19 deaths in Texas starting to catch up with the earlier rise in reported cases. When cases started to spike in Texas, talk turned to the declining death rate, which is misleading since we should have been looking at cohort-adjusted death rates from the start. The media conspired to spread this misinformation, ignoring epidemiologists - when given a choice between reporting a misleading statistic today and reporting a more valid number in two weeks, the media cannot help but quench the immediate thirst.

In putting together the prior analysis, I already noticed the sad state of the data. On that day (July 27), Texas just changed how it reports deaths to rely on cause of death on death certificates. This created a single-day death toll of 675, well above the daily average, which presumably corrected for under-counting from previous days. With this correction, the state claimed that it will henceforth provide "more accurate" death counts.

What the state official did not explain is that the change in reporting also ensures a reporting delay for all deaths of at least one week. Let's think through this in the context of rising case rates and "declining" death rates. I make the nonsensical assumption that if patients die from Covid-19, they die immediately, on the day they are reported as new cases. Under the new reporting procedure, these cases will be reported today but the deaths will be reported when death certificates have been issued and can be inspected, which is at least 7-10 days in Texas. Thus, this group of patients (who already died) inflates the number of cases today while suppressing today's death rate since their deaths are not yet reported. This time lag is due to reporting procedures, and has a similar effect as the time lag between infection and death.

The data source, Covid Tracking Project, rated Texas "A" for "data quality" so presumably the state of the data elsewhere is worse.

***

I intended to update the analysis for this post last week but my effort was met with more data mischief. On August 2 (Sunday), Texas did not report any data, blaming a "systems upgrade" and promised that "data for Sunday will be posted with Monday’s data update." So I waited another week before publishing this update.

Just a reminder, two weeks ago, I presented the following chart:

The thin gray line is the growth curve for reported cases in Texas, smoothed using a seven-day window, from mid May through late July. The thick gray line is this growth curve shifted forward in time by three weeks. The red line is the growth curve of deaths in Texas, smoothed using a seven-day window. Except for the single-day anomaly (July 27), it appears that deaths roughly lagged cases by three weeks.

The next chart extends the time-line to August 1, the day before the data blackout:

This chart is disturbing. The growth of deaths in Texas has started to markedly outpace the growth of cases (time-shifted). Using the same time lag of 3 weeks, we should have expected deaths to climb 7 times between mid May to August 1. Instead, deaths from Covid-19 in Texas jumped over 9 times during this period.

This observation opens up a large can of worms. Back in July, the optimistic doctors and people were attributing the "declining" death rate not just to time delay but also to better treatments, younger people getting infected, virus becoming less lethal, etc. The data now suggest that treatments became less effective, or the virus got more lethal, etc. I'm not making such an argument, just pointing out why it is risky to draw premature conclusions from little bits of statistics.

***

Apparently, someone called time-out on these concerning numbers. On August 2, the state took a day off. When the data started flowing again, the death counts fell back to trend, as seen below:

For any data analyst, the alarm bells are deafening.

There are a number of suspicious issues here. Firstly, the official reason for the August 2 black-out was a system upgrade, and that the release for the next day would contain two days' worth of data. It doesn't appear that Monday's data contained two days' worth. The number of new cases released on August 3 was ~11,500. That is supposed to include cases for both August 2 and 3. Between July 27 and August 5, excluding Aug 2-3, daily cases were above 8,000.

Secondly, it should be simple to report deaths daily now that Texas is using death certificates to confirm deaths. I don't see how a system upgrade destroys the ability to know the dates of death. I wonder if they backtracked to the old way of reporting deaths after realizing the new procedure reveals a higher death toll.

***

Here's the problem facing data analysts in the Covid-19 era. The data are what the officials want them to be. They are constantly changing definitions, accounting rules, timing of releases, etc. It is impossible to make sense of the mess unless you maintain the database. Meanwhile, this data catastrophe is severely hampering public health decisions. The question is whether it is a feature or a bug.