A story about poop, and the scientific method

These headlines flooded the media last week. I'm listing the first five results selected by my search engine.

University of Arizona is screening for COVID-19 via dorms poop (New York Post)

University of Arizona says it prevented COVID-19 outbreak by testing students feces (Daily Mail)

Poop test stop COVID-19 outbreak at University of Arizona (Science)

Poop becomes secret weapon to detect COVID-19 cases in dorms, says University of Arizona (National Post)

Researchers at the University of Arizona say they stopped a coronavirus outbreak before it spread by testing students poop (MSN)

***

The sequence of events that reportedly happened is as follows:

1. As part of the re-opening program at University of Arizona, the school tests wastewater from 20 on-campus dorms for presence of the coronavirus.
   
   
2. Last week, one sample came back positive, and was traced to a specific dorm (Likins Hall).

3. Researchers notified school administrators and convinced them to test every resident of that dorm.
   
   
4. Of the 311 samples collected from residents, researchers found 2 cases, implying a positivity rate of 0.6%.

5. The two students, both asymptomatic, were isolated.

6. Researchers concluded that they prevented an outbreak in that dorm.

This is good news for managing campus re-opening. Testing sewage water should be in the toolkit, used in conjunction with other tools. But establishing the cause-effect relationship is not as simple as the Arizona researchers made it out to be.

This story is also a case study of how it is easy to make a scientific claim but it is hard to prove it.

***

The university tested about 10,000 people, as they checked into campus housing about 10 days ago (other returnees and off-campus students are tested only if they request it). They found 46 cases. That's a positivity rate of 0.5%. On average, if we select 311 people randomly, we should expect 1.4 positive results, so it is normal to find one or two cases if that is the prevalence.

Recall that they discovered two cases in 311 tests (0.6% rate) in Likins Hall. The scientific question is whether that positivity rate is abnormal. If they had tested samples from another dorm - one for which the wastewater testing turned out negative, and found no positive cases, that would strengthen the causal claim. Did wastewater testing accurately detect the presence of virus in a specific building? The answer is yes, conditional on the key assumption that the prevalence of virus in dorms that passed sewage testing is zero.

Since the residents of the other dorms were not tested, we don't really know the answer. That's why it is an assumption of the analysis. There are reasons to believe that the prevalence is low but not zero: (a) the residents started checking in on August 21 and the wastewater testing program was announced on August 25 (link) so it is possible that residents have gotten infected in the interim (b) it's likely that at least a few residents have not yet completed arrival testing, and they may be infectious (c) up to two guests per resident are allowed in the dorms, and testing of off-campus students is voluntary (d) vendors (including movers) can bring virus into the building.

***

Now, consider the tracing of sewage water to Likins Hall. How accurately can this determination be made? This depends on where the sewage water is collected. In a notice to on-campus students, the university made this disclosure: "This [wastewater] testing does not identify individuals or even floors of the dorm or building that might have infections." This statement directly contradicts the published story in which pinpointing the building was crucial in the chain of events.

I also wonder how they deal with any mixing of sewage, or possible environmental contamination.

***

The consultant hired to supervise campus re-opening told a reporter: "if we had missed it, if we had waited until they became symptomatic and they stayed in that dorm for days, or a week, or the whole incubation period, how many other people would have been infected?" This interpretation assumes that the virus found in the sewage water came from the two students who subsequently tested positive, and were still infectious.

Wastewater testing is beneficial only if it detects active infections. So establishing timing is essential. It is possible that the infected sewage came from past infections but not from the two active cases - there is still a lot that is unknown about virus shedding in feces, only a fraction of infected people do it. I'm not saying this is what happened but the coincidence of (a) virus found in sewage testing and (b) cases found in building does not prove that the infected sewage came from those specific cases.

***

There are three types of university responses to Covid-19. The most risk-averse group has moved all instruction online. The second group is attempting a return to the classroom but will require comprehensive testing. The third group re-opens but is looking for shortcuts to avoid having to test everyone. Sewage testing is of interest to this final group of schools.

The Arizona case study provides some hope that testing wastewater may reduce the amount of individual testing necessary to monitor and control infections on campus. Conceptually, this is related to group testing (explained here), replacing the usual nasal swab tests with sewage tests in the first stage.

More work needs to be done to establish the cause--effect. For example, controls should be included in the study, testing a dorm where the sewage came back negative. Another reason for controls is to measure false negatives. A deeper understanding of the "schedule" of virus shedding during the infection period is necessary. Measures to rule out contamination help reduce false positives. As with other tests, the scientists should publish the accuracy characteristics of the wastewater test.

Comments

[Paul]

There is some ambiguity in the statement:

"This [wastewater] testing does not identify individuals or even floors of the dorm or building that might have infections."

Is it "does not identify or even or "?

Or is it "does not identify ... "?

I.e., they can identify the dorm, but not the floors of the dorm, and they can identify buildings, but not the floors of the building.

[Ken]

I asked about false positives on a post from CSIRO, Australia's government research organisation, on Facebook about their sewerage testing and didn't get a reply. The positive from Italy in the first half of 2019 (yes, 2019, they used an old sample) would definitely be. What presumably happens is that they use more replication stages in the PCR and this is known to increase the number of false positives. For the nasal swabs the false positive rate seems to be about 2-3 per 100,000 tests.