Big Data, Plainly Spoken (aka Numbers Rule Your World)

Leave the scientists alone, let them do their jobs

This CNBC report on the coronavirus vaccine trial unsettles me because the journalists are not helping the scientists. The report describes the experiences of five participants in the vaccine trials.

First, five participants are an extremely small sample from which to draw any conclusions. This report discusses five cases of suspected mild symptoms, out of five interviewees, as if 100 percent of vaccinations would result in those conditions. That is almost certainly false.

Second, the reporters are surmising that these participants were given the vaccine rather than the placebo. The leading paragraph is written as if the named participant had been given the vaccine, not the placebo: "...woke up in the middle of the night with chills and a fever after taking the Covid-19 booster shot in Moderna’s vaccine trial." Readers are not clearly informed this is a speculation. The guess may be somewhat accurate because it is disclosed that the vaccine shots might cause mild flu-like symptoms while presumably the placebo (saline) shots don't. But a guess is still a guess.

Third, without irony, the first subheader read "double-blind." Patients are not supposed to know which arm they are on. The reason for blinding is that knowing whether one got the vaccine or the placebo might change one's behavior, and this creates a source of uncertainty for the trial outcome. These reporters are encouraging trial participants to speculate about whether they received the vaccine shots or not, and thus usurping the intention of blinding. The named participant is a biologist who stated that he "has a high degree of confidence that [Moderna's methodology] would work, and wanted to contribute to the solution." That's exactly the reason why participants are blinded so that if they took any actions that might "aid" the vaccine, those actions might not work because the participants might be in the placebo arm.

Fourth, if the reporters bothered to pull out the protocols, the titles of both the Moderna and Pfizer protocols said "observer-blind", not double-blind. By design, some of the people involved in running the trial are already unblinded; now, reporters are seeking participants who believe they know which arm they are on - which is an attack on the only blinding used in these trials.

Fifth, participants have no way of knowing that their symptoms are due to the vaccine. Sure, the placebo shot is unlikely to cause mild fever or other mild symptoms but they have plenty of other possible causes, other than the vaccine. CNBC accepts the simplistic assumption that all such symptoms experienced within a few days after taking a dose are caused by the vaccine.

Sixth, the reporters reveal the existence of private Facebook groups in which people in the clinical trials are aggregating and socializing. This is where the reporters are potentially doing science a service. This social-media phenonomen is new, and to my knowledge, there hasn't been research on how to deal with it. The typical statistical analysis of clinical trials assumes that the participants are independent entities, and social media is destroying this assumption.

Take the behavioral change that blinding is supposed to prevent. In the past, individuals may change their behavior if they somehow discover their treatment assignment. Now, groups of participants may make coordinated behavioral changes. Also, unblinding may propagate through the social network. The data used to unblind themselves may even be flawed; people may falsely believe they are in the treatment arm. No matter, behavior still will change.

Seventh, I assume all participants have signed agreements not to talk to the media or other people about the trial. Did they? Are they wantonly breaking these agreements?

***

Please leave the scientists alone, let them do their jobs.

 

 

Notes on the Moderna vaccine trial protocol 2

This is Part 2 of my notes about the Moderna vaccine trial. Read Part 1 here.

***

Durability of Protection

As described in the last post, the full analysis will not take place until middle of 2021, so the vaccine developers are hoping that their vaccine would show enough during the first or second interim analysis to accelerate the time line. Early reads increase the chance of false-positive findings, as I outlined in a previous post. To counter this, the study drastically lowers the p-value threshold to reach statistical significance in the interim analyses. (Instead of p = 0.05, the first interim analysis in effect sets p = 0.0002.) This is a proper thing to do.

However, if the first interim analysis results in a vaccine approval, then the participants will have been observed for at most 2.5 months from the second dose. We will not know if the vaccine offers protection beyond 2.5 months.

According to the protocol, Moderna "intends" to continue the study after if it receives interim approval. There will be an ethical debate about whether those who received placebos should be given the vaccine should it win approval. Some may oppose withholding a "proven" treatment from these participants. If they are given the vaccine, we lose the ability to measure protection beyond 2.5 months.

Sample Size

If the first interim analysis is successful, then the vaccine would receive approval based on about 50 total infections. Assuming that the vaccine effectiveness is 50%, then we expect half the number of infections on the vaccine arm relative to the placebo arm. So the vaccine arm should account for one-third, or about 17 infections. That's an awfully small number.

Nevertheless, if approved, the vaccine would have met the more stringent statistical significance threshold so the small sample size is not a serious problem. But the required number at the first interim analysis is small enough as to make many statisticians nervous. Continued tracking beyond the first analysis will be key to make sure the early result is not a mirage.

Defining Trial Outcomes

In the primary analysis, the "endpoint" (outcome) is infection, which requires a PCR test confirmation plus some pre-specified level of symptoms.

Some experts are arguing that they should count severe cases only because we don't really need to stop the mild cases. The chosen endpoint includes both mild and severe cases but we expect that most of the cases will be mild so there is concern that the signal (severe) to noise (mild) ratio is not high. Of course, if only severe cases are counted, the trial would last longer.

The opposite criticism is that they are by and large missing asymptomatic cases because almost all the testing are triggered by symptom reporting. One can argue that we need to stop asymptomatic spread because any viral transmission may eventually produce severe cases.

Moderna is taking the middle ground here.

Unusual Sources of Noise

In running any experiment, we'd love to eliminate as many sources of noise (variability) as possible so that any observed differences can be attributed to the vaccine. Some practical issues for this vaccine trial may produce noise, which affects our interpretation of the results.

The FDA permits accommodations due to the pandemic. This means that in-person visits may be replaced by phone calls. That itself is not a problem if all visits are converted to phone calls. The problem is that swaps occur based on request so they'd end up with some in-person visits and some phone calls. This creates a possibility that the method of the follow-up may be associated with, or even cause, different outcomes.

Besides, the protocol lists a variety of tests used to confirm infection. The nasal swab is preferred but saliva and respiratory samples are also used. These tests have different accuracies and characteristics so the mix of test types may affect the count of confirmed cases. For example, people living in high-risk areas are more likely to submit saliva samples by mail versus nasal swabs from in-person visits.

Uncertainty Around the Case Rate

The design assumes a 0.75% baseline case rate within six months but the researchers acknowledge uncertainty around this number - that's why they end the enrollment based on the number of cases rather than the number of participants. What if this assumption is wrong?

In particular, this base rate seems fairly low. Those who hope for "herd immunity" might be shocked to learn that the virus is expected to infect less than 1 percent of a community over six months. At that sluggish rate, when will we reach 70 percent or so prevalence?

All sample sizing requires guessing at the base rate, and sometimes we can get it wrong. If the base rate is somewhat higher than 0.75%, then the trial will get infections faster than expected, and thus end enrollment earlier than expected. The real case rate among those participants can be, say, 2%.

Here's the rub. If 2 percent had been used as the base rate in the design of the clinical trial, statisticians would have stipulated more than 53 infections to trigger the first interim analysis. This means, if the observed case rate is higher than expected, the first interim analysis suffers from insufficient data. Smaller-than-expected sample size makes it more challenging to beat the significance threshold.

***

The Moderna protocol is reasonable: there are some unconventional elements, but in an emergency, we'd have to accept some accommodations. It's still hard to imagine how any of these trials will get to a strong result by end of October or November.

News about the vaccine trials is coming fast. Just this weekend, the Washington Post reported that the FDA may tighten the vaccine approval requirements a little to pacify critics and build trust. I have another post up discussing these modifications.

 

 

 

 

 

Notes on the Moderna vaccine trial protocol 1

Bending to outside pressure, Moderna has made public one of the documents submitted to the FDA that provide more details on how they are running the vaccine trial. (PDF here) This is a reassuring move, and I applaud them for taking this step.

In a previous post, I raised two key questions I would like to find answers to, and after reading all 135 pages of the protocol, I can tell you the answers.

How is efficacy measured?

Vaccine effectiveness (technically, efficacy) is the reduction in infection rate among the vaccinated relative to the baseline infection rate among the unvaccinated. A 50% effective vaccine is one that halves the baseline infection rate. (For more on vaccine effectiveness, see this other post.)

An important detail is how this infection rate is measured. According to the trial design documentation, Moderna will confirm infection using PCR nasal swab tests (primarily). To be precise, what they measure is confirmed cases rather than infections.

What is the analysis schedule?

Until the documentation is made public, there has been a lot of sloppy reporting about when results will be available. Now we know. The primary analysis will be conducted when the total number of confirmed cases (across both test and placebo arms) exceeds 151. The expected time of this analysis is six and a half months after the last trial participant receives the second dose. By my estimate, this will be mid June 2021.

The trial started enrolling participants at the start of August. Assume that enrollment is complete by the end of September, the second dose of the last participant should occur at the end of October, the data for the primary analysis will be frozen by mid May 2021, and the trial result will be ready by mid June 2021.

Why do pharma CEOs keep saying they expect results by the end of 2020? That's because they are hoping one of two interim analyses might produce results so stellar that the FDA will have seen enough. The first interim analysis is triggered when the number of confirmed cases exceeds 53, which is 35 percent of the full sample. If it takes 6.5 months to accumulate 151 cases, it would be 2.5 months to get to 53. (In both instances, the half month is added because they don't start counting cases until 14 days after the second dose.)

For the first interim analysis, assume they only analyze participants through the end of August, the second doses should be completed by the end of September, the data will be frozen by mid December 2020, and so the earliest date by which the vaccine might be declared effective is mid January 2021, if everything else falls into place.

***

What to Look Out For

I read through the entire 135-page document. I prepared a list of notes about the clinical trial, mostly about statistical matters. These are placeholders for now. How much these issues matter depends on what happens during the trial.

Blinding

The trial is observer-blind, less than the gold standard of double-blind. Blinding is hiding whether a participant has gotten the vaccine (or drug) or the placebo. Double-blind means neither the researchers nor the participants are told who got what shot, until the data are ready for analysis. Blinding prevents humans from misbehaving as a result of knowing the treatment assignment. In this Moderna trial, the person administering the shots is not blinded. There may be a scientific reason for why some researchers cannot be blinded for this study but I didn't find an explanation in the documentation. (I checked the Pfizer-Biontech trial, and it is also observer-blind, not double-blind.)

Many outside scientists and reporters have missed this detail, and mis-reported the Moderna trial as double-blind (e.g. here, here, here).

Analysis Population

The primary analysis is "per-protocol", which is less than the gold standard of intent to treat. This is always a controversial subject so let me explain the disagreement. In a canonical clinical trial, half the participants get the vaccine (or drug), and the other half receive the placebo (or comparator). Researchers take great care to randomize assignment to these two arms. If each participant has equal chance of being assigned to either arm, then we have created the condition of "all else being equal" so that when we compare the average person across the two arms, any difference can be attributed to the vaccine (or drug).

In practice, some participants will drop out of a trial. It is quite possible that the dropout rate is not the same across the two arms. For example, if the treatment causes irritation, more people in the treatment arm might quit. The statistical debate is whether to include the dropouts in the analysis. The gold standard of intent-to-treat analysis says to stick as close to the original population as possible. Dropouts are therefore included (up to the time of dropout). In a per-protocol analysis, the dropouts may be excluded as if they never existed. The proponents of per-protocol argue that the final outcomes of these dropouts are not known, and so including them might be "unfair" to the treatment arm.

By contrast, supporters of intent-to-treat (including me) don't buy that the act of dropping out is independent of treatment, and worry that the loss of randomization invalidates the "all else equal" condition. For me, the most important argument is the practical point of view: the per-protocol findings apply only to people who take the treatment without dropping out!

Dropouts are not the only subset of participants who might be excluded in a per-protocol analysis. It's just easier to explain.

Back to the Moderna trial. It appears that dropouts prior to the second dose will be excluded from the per-protocol analysis but those who discontinue after the second dose will be included.

Infections Before Finishing Treatment

In the Moderna trial, another interesting subset consists of participants who get infected and then are excluded from the per-protocol analysis. In the primary analysis, Moderna starts counting infections 14 days after the participants get their second dose.

The trial design stipulates that people who have confirmed infection before the second dose will be ineligible for the second dose. All such people therefore will not complete the protocol and be excluded from the per-protocol analysis. The good news is that Moderna has committed to performing an intent-to-treat analysis for comparison. However, the final decision will be based on per-protocol, and there is no guarantee that the intent-to-treat analysis will be published for outside evaluation.

Let's say we're behind at the end of a(n American) football match, and our coach decides to go for an on-side kick and then drive for a field goal. To win the game requires executing both steps. If the on-side kick fails, the other team will run out the clock. (Last year, on-side kick success rate in the NFL was only 6 percent.) If the on-side kick fails, the per-protocol analysis excludes this match because the team didn't execute the entire plan. If the on-side kick is successful and the team gets into field-goal position, the per-protocol analysis includes this match. So, the estimate of the success rate of the two-step plan will be over-optimistic.

Here's the practical point of view. If the vaccine requires two doses, and someone gets sick after the first dose, the vaccine is ineffective for that individual. It doesn't matter that the said individual has not received the second dose; it is meaningless to administer the second dose.

Excluded Infections

Then, there is the subset of participants who get infected during the first 14 days after the second dose. For an unexplained reason, Moderna isn't counting infections until two weeks after the second dose.

This subset will be included in the per-protocol analysis because they took the full two-dose program. However, the outcome for these participants will be "censored," meaning instead of being treated as infected, they are treated as if they have dropped out of the study just before their infections are confirmed.

Again, I take a practical view on these matters. If, after the vaccine is approved, someone gets infected a few days after receiving the second dose, the vaccine would have failed for this individual.

So when the results come out, keep an eye on whether there is a differential likelihood to get early infections on the vaccine arm relative to the placebo arm. The good news is Moderna has committed to conducting additional analysis that counts infections from the first day after the second dose for comparison. However, the final decision does not have to take into account this analysis, and it may not be publicly available.

Counting Infections

Measuring infections in a vaccine trial is always challenging because the participants are healthy. Contrast this with a clinical trial testing a treatment on sick people. Since it's unethical to inject virus into healthy subjects, we have to wait for infections to crop up via community spread.

Moderna isn't very proactive in monitoring for infections. They rely on self-reported symptoms, which then trigger confirmatory PCR tests. For example, the only way researchers will learn about suspected infections during the first 14 days after the second dose is if the participants report symptoms. Not everyone who report symptoms will get tested; the researchers have some discretion for certain types of symptoms.

That's why we will never know the true infection rate. The reported rate should be considered the rate of confirmed cases.

***

These are the most salient information I learned from the just-released documentation on the Moderna vaccine trial. I have several more notes that will be written up in my next post. 

Confusion over case trends and positivity ratios, an aftertaste of targeted testing

We are now experiencing the aftertaste of targeted (aka triage) testing, a policy that has unsavory consequences that were predictable (link). Targeted testing is the idea that only people with severe symptoms or at high risk of infection should get tested for the novel coronavirus. Compared to a broad-based testing plan (whether comprehensive or randomized), targeting economises the number of test kits. The basis of any targeting strategy is the skill to predict who is infected. The group that is targeted should have a higher infection rate than the group that is not targeted. It's important to bear in mind that targeting not only suppresses how many people are tested but also affects who gets tested.

One way in which targeted testing hurts us is that it ruins our datasets from which we obtain insights about the pandemic. This post is prompted by Ken's comment on my previous post on casual presumptions of randomness:

The number of daily new cases in some countries doubled as fast as every 2 days. This didn't mean a greater rate of infections, it was simply a result of catching up all the non-tested cases.

The connection between infections, tests, and cases is complicated, and the media shift back and forth between the rate of cases (positive test findings) and the positivity ratio (proportion of tests that came back positive). I spent some time explicating these relationships, and hope that this post will succeed in helping you gain some intuition around those concepts.

We start by decomposing the positive test findings. In the flow chart below, starting with the dark gray box on the left side, we examine a fixed population of people. At any given time, the people are divided into three "compartments," the infectious people who currently have the virus, the susceptibles who have not been infected, and the recovered (which by convention, includes those who died from Covid-19).

Also by convention, anyone can be infected only once, and we thus ignore recovered people from further consideration. From the dark gray box emerge two paths that terminate at positive tests: the upper path consists of infectious people while the lower path, susceptibles. Some portion of the population are infectious; within that, some proportion have been tested for Covid-19, and within that, some proportion have tested positive. This last tally is one part of the confirmed cases. The other part of the confirmed cases comes from susceptibles, represented by the lower path.

The number of confirmed cases is the number of positive tests. This is a key metric on all Covid-19 dashboards. The other important statistic is the positivity ratio. This is the number of positive tests (represented by the dark red box on the right) divided by the number of tests conducted (the sum of the two yellow boxes).

The next diagram describes the drivers of the two dashboard statistics:

• the weights of infectious or susceptible compartments in the population (excluding recovered)
• the probability of getting tested among the infectious or susceptible group
• the weights of infectious or susceptible people within the tested subpopulation
• the positivity ratio for the infectious or susceptible people who got tested (call these the group-level positivity ratios to distinguish from the overall positivity ratio)

The overall positivity ratio is affected by three signals. First up is the rate of infection in the population, reflected in the relative weights of the I and S compartments. Second, the testing regime imposes a selection effect on these weights, producing the second set of weights. The notorious triage or targeted testing protocol restricts testing to patients showing severe symptoms or at highest risk - these disproportionately arise from the infectious group, and so the second set of weights are more skewed toward infectious people, compared to the first set of weights. Thirdly, test accuracy affects the number of positive test results, given any tested subpopulation.

Test accuracy is typically measured by sensitivity and specificity. On the upper path, the positivity ratio of the infectious-tested group is just the sensitivity, which we assume to be 90 percent. On the lower path, the positivity ratio for the susceptibles-tested is the rate of false positives (as this group is not infected), which is 1 minus the specificity, which we assume to be 5 percent. (For the sake of this post, we assume that the performance of the diagnostic tests has not changed during the pandemic and thus the group-level positivity ratios are fixed. This simplifies the following examples.)

 

What's special about 5% positivity ratio?

The media keep reminding us that the CDC wants the positivity ratio to fall below 5%. What is the significance of the 5-percent threshold? I can explain using the flow chart.

We hope for the epidemic to dissipate. When this happens, the infectious compartment dwindles to zero while the susceptibles stabilize. This means there will be no more flow through the upper path. All of the positive test results will come from the lower path (and are false positives).

The overall positivity ratio is a weighted average of the group-level positivity levels, the weights being the proportions of those tested who are infectious or susceptible. In this extreme case, the weight is entirely skewed towards the susceptible-tested, for which the positivity ratio is 5 percent. Thus, the overall positivity ratio will be 5 percent.

In other words, when the positivity ratio dips to 5 percent or below, new infections should be dying out.

 

What's the relationship between rate of cases and rate of infections?

I will now use this setup to investigate Ken's comment. He was reacting to the naive interpretation of case growth as infection growth, popularized by the media. Such a conclusion is invalidated because of targeted testing.

Note that infection growth cannot be directly observed. By contrast, case growth is reported on every dashboard. In Ken's scenario, the number of cases (i.e. positive test results) are growing rapidly.

Start from the dark red box on the right of the flow chart, and work backwards. The jump in positive cases may come from either of the two paths.

First, consider the lower path representing susceptibles. During the course of an epidemic, more people get infected, which reduces the number of susceptibles in the population, which in turn decreases the number of susceptibles testing positive. So if the cases from this lower path have to go up, then it must be that susceptibles are suddenly more likely to get tested.

One such scenario is if politicians expand targeted testing to those with mild or no symptoms. This is Ken's hypothesis.

Such a policy shift also affects the upper path. In fact, infectious people are more likely to experience mild symptoms. Even if the rate of infection is stable, the number of positives tests arising from the upper path grows. If infections are also accelerating, both factors contribute to case growth.

Ken argues that the entire case growth can be attributed to the expanded testing capacity while the media generally interprets the case growth as infection growth, ignoring possible shifts in testing policy.

The trouble is that both factors are at play, and we don't know the relative sizes of their contributions. This confounding is brought to you by targeted testing! (I can't resist: it's free and comes with a hidden cost.) This example demonstrates the aftertaste of targeted testing. It messes up the data so we can't interpret them properly.

It doesn't appear that Ken's scenario is happening in the U.S. as the federal government has continued its misguided targeted testing policy. Some politicians even grumble that we are conducting "too many" tests. (This can only happen if we were conducting too few tests, and thus under-reporting cases early on.)

 

How does targeted testing affect the positivity ratio?

In Ken's scenario, he assumes that the increase in testing accounts for the entire jump in cases. So, let's stipulate that the rate of infections is stable, and look at how the testing policy affects the positivity ratio.

The overall positivity ratio is the weighted average of group-level positivity ratios, so the key is to understand what happens to the second set of weights, which describes the mix of infectious and susceptibles in the tested subpopulation.

Under a randomized protocol, the second set of weights just represents the underlying mix of infectious and susceptibles since both groups have equal probability of getting tested. Under a targeted protocol, the second set of weights is biased toward infectious who are more likely to experience symptoms. Thus, early on, the positivity rate is over-stated. If targeting restriction is relaxed, the bias toward infectious is reduced. Thus, the relative weight of susceptibles-tested is pushed up, and so the positivity rate is driven down.

Since the rate of infections is assumed stable, the shift in positivity rate tells us only about the testing policy. We've just discovered how politics enter the data.

***

Targeted testing introduces bias into the data, which pollutes the link between the rate of cases and the rate of infections. Remember: we use the rate of cases to infer the rate of infections because in the real world, no one knows the rate of infections.

Targeted testing is the aftertaste that you can't seem to get rid of. It polluted the data when first implemented. Then, every subsequent change in policy, and the gradual behavioral adaptation to these changes, add to the confusion. The effect on number of tests is observable but the effect on the types of people taking the test is not.

This tragedy on the national level is now being replicated on individual school campuses. Many schools are doing targeted testing only on students who self-report symptoms. Their dashboards are reporting confirmed cases, the trend of which is being interpreted as the trend of infection. What is the point of data-driven decision-making when the data are known to be bad?

 

 

 

 

 

 

 

 

 

Story-first thinking

There are two types of people when it comes to using data. I call them "data first" and "story first". The natural mode seems to be "story first". Calling this "narrative fallacy" or "confirmation bias" creates the impression that the default mode is "data first", and being "story first" is an aberration.

I believe it's the opposite. Being "data first" requires training, and conscious effort. This viewpoint may be related to Dan Kahneman's classification of System 1 and 2 thinking. Story first is quick, System 1 while data first is slow, System 2.

***

The recent reporting of first-time unemployment claims data by Marketwatch captures "story first" thinking. The reporter's story is the federal unemployment compensation (the $600 payout) hurts the economy.

On August 13, Marketwatch published an article with the following graphic:

The headline is the story: "when you pay people more to sit at home... they sit at home." What is the evidence (data)? The reporter said "a big drop in people seeking or receiving benefits in the past two weeks after the end of a $600 federal stipend hints the answer might be yes".

So he's pointing to the last two columns, and saying that those two columns are below 1 million while the previous columns exceed 1 million. Simply put, (a) from last week of July to first week of August, federal unemployment payout ended (b) from last week of July to first two weeks of August, initial jobless claims went down, therefore, (a) must be the cause of (b).

In order to sharpen part (a) of the argument, he established a bar of 1 million weekly claims as some kind of magical level separating good from bad. Above 1 million, bad; below 1 million, good. Prior to August, above 1 million; after August, below 1 million.

The reporter proceeds to quote multiple economists all claiming that this conclusion is a slam dunk, "not rocket science", "hear it all the time from business people", etc.

Our System 1 thinking might accept such a logic but given time to consume the data, it's easy to note the fatal flaw in the argument.

The federal benefit lasted from April through end of July (reference). For every single week during this period, the initial jobless claims number has dropped. In fact, the weekly drops were a lot bigger in April than in August. So we have months of counterexamples on the same chart that is used to prove his case.

Besides, the number of initial jobless claims in the second week of August is still five times higher than the level seen in January and February before the pandemic went out of control.

***

Fast forward one week to August 20. The same reporter reported in MarketWatch on initial jobless claims, and updated his chart:

The general trend of this chart mirrors the first chart. The number of initial jobless claims has declined from the peak in early April.

Take a closer look at this chart versus the previous chart. You will notice they are not the same thing. Here are the alterations:

• The first chart shows seasonally adjusted data while the newer chart shows unadjusted data. The footnote even discloses that the seasonally adjusted data are higher (i.e. worse) than the unadjusted data
• The newer chart added a second category of jobless claims (federal)

Despite the data changing, the story hasn't changed.

In the first paragraph, he writes "Initial weekly jobless benefit claims rose in mid-August and topped 1 million again, potentially pointing to an increase in layoffs after a summer surge in the coronavirus epidemic or perhaps to more people applying for benefits after President Trump temporarily added $300 in extra federal payouts."

So, the story remains the same. The previous week, removing the $600 caused the drop in jobless claims. This week, the "addition" of $300 caused the increase in jobless claims.

The President did not "add extra federal payouts". The $300 payout will be half the previous amount. Further, almost no one received a $300 check during the third week of August.

Once again, the contradictory evidence is found on the same graphic. Back in April, when the $600 checks started getting paid, the new jobless claims number was decreasing, not rising. 

Now, let's talk about the 1 million claims threshold that appeared in the first chart. That 1 mllion level was established for the state-level new jobless claims, which is the lower part of the columns in the newer chart. If you look at just the lower parts, all three numbers for August sit below 1 million. So nothing "topped 1 million again." Magically, the 1 million threshold is now applied to the state-plus-federal new jobless claims data. Why? Because it fits the story.

Except... all three numbers for August sit above 1 million.

***

The economists cited in the August 20 article no longer think the link between federal payout and initial jobless claims is a slam dunk. The quotes are a mixed bag.

But the story isn't wrong. What is wrong? It's the data, stupid. "MarketWatch is reporting select jobless claims data using unadjusted figures to give a clearer picture of unemployment. The seasonally adjusted estimates typically expected by Wall Street have become distorted by the pandemic and appear to overstate new claims at times."

MarketWatch presents zero evidence that the seasonal adjustment is incorrect, except their belief that any adjustment that increases the unadjusted figures is wrong.

According to the journalist, "the increase in new claims wasn't as bad as it seemed... based on actual or unadjusted figures. They rose a much smaller 52,776 to 891,510 and remained below 1 million for the third straight week." [As noted, he uses 1 million regardless of whether the data include federal claims.]

This argument is entirely backwards. If the unadjusted change is smaller than the adjusted change, that means the current situation is worse than normal. Hard to argue with that. 

The point of seasonal adjustment is to establish the expected weekly fluctuations in the data during normal times. (I wrote a tutorial on seasonal adjustment years ago, and also in Chapter 6 of Numbersense (link), on employment statistics.)

***

What do we think should happen the following week? Since more states will be paying the "extra $300", the data better behave! We want to see the initial jobless claims at the state level (unadjusted) to rise again since such payout disincentivizes work.

Drum roll.

On August 27, MarketWatch pushed the graphic to the bottom of the page.

The first sentence of the report reads: "New applications for jobless benefits fell again in late August to just above 1 million and resumed a downward trend, perhaps signaling the resumption of a gradual if painfully slow recovery in the U.S. labor market."

The theory says when more people are receiving the $300 checks, they should be staying home pushing up the jobless claims. (One problem all along is the focus on initial jobless claims rather than total jobless claims or continuing jobless claims.) Don't worry: the journalist is holding out hope for next week's data. He said: "most states still aren't offering the money and the extra cash is just starting to be sent out." The week before, he suggested that the $300 checks caused the rise in claims.

The cited statistic in the first paragraph is the seasonally-adjusted, state-level number (the lower sections of those columns) so the reporter reverted to the metric from two weeks before. He goes on to explain that the data problem is limited to that one week which broke the trend. "New claims rose sharply last week.... but most economists suspect problems with the seasonal adjustments or other temporary factors."

***

Story-first thinking frequently leads to cherry-picking specific conforming data, ignoring contradictory data, explaining away or pouring doubt on non-conforming data, branching the story to accommodate non-conformng data, and creating episodes that work individually but do not cohere.

***

There is another jobless claims report out today but I'll save this for a different post because the Bureau of Labor Statistics suddenly changed the methodology used for seasonal adjustment, which at first glance, will make every single seasonally-adjusted number between now and November look better.

 

 

A story about poop, and the scientific method

These headlines flooded the media last week. I'm listing the first five results selected by my search engine.

University of Arizona is screening for COVID-19 via dorms poop (New York Post)

University of Arizona says it prevented COVID-19 outbreak by testing students feces (Daily Mail)

Poop test stop COVID-19 outbreak at University of Arizona (Science)

Poop becomes secret weapon to detect COVID-19 cases in dorms, says University of Arizona (National Post)

Researchers at the University of Arizona say they stopped a coronavirus outbreak before it spread by testing students poop (MSN)

***

The sequence of events that reportedly happened is as follows:

1. As part of the re-opening program at University of Arizona, the school tests wastewater from 20 on-campus dorms for presence of the coronavirus.
   
   
2. Last week, one sample came back positive, and was traced to a specific dorm (Likins Hall).

3. Researchers notified school administrators and convinced them to test every resident of that dorm.
   
   
4. Of the 311 samples collected from residents, researchers found 2 cases, implying a positivity rate of 0.6%.

5. The two students, both asymptomatic, were isolated.

6. Researchers concluded that they prevented an outbreak in that dorm.

This is good news for managing campus re-opening. Testing sewage water should be in the toolkit, used in conjunction with other tools. But establishing the cause-effect relationship is not as simple as the Arizona researchers made it out to be.

This story is also a case study of how it is easy to make a scientific claim but it is hard to prove it.

***

The university tested about 10,000 people, as they checked into campus housing about 10 days ago (other returnees and off-campus students are tested only if they request it). They found 46 cases. That's a positivity rate of 0.5%. On average, if we select 311 people randomly, we should expect 1.4 positive results, so it is normal to find one or two cases if that is the prevalence.

Recall that they discovered two cases in 311 tests (0.6% rate) in Likins Hall. The scientific question is whether that positivity rate is abnormal. If they had tested samples from another dorm - one for which the wastewater testing turned out negative, and found no positive cases, that would strengthen the causal claim. Did wastewater testing accurately detect the presence of virus in a specific building? The answer is yes, conditional on the key assumption that the prevalence of virus in dorms that passed sewage testing is zero.

Since the residents of the other dorms were not tested, we don't really know the answer. That's why it is an assumption of the analysis. There are reasons to believe that the prevalence is low but not zero: (a) the residents started checking in on August 21 and the wastewater testing program was announced on August 25 (link) so it is possible that residents have gotten infected in the interim (b) it's likely that at least a few residents have not yet completed arrival testing, and they may be infectious (c) up to two guests per resident are allowed in the dorms, and testing of off-campus students is voluntary (d) vendors (including movers) can bring virus into the building.

***

Now, consider the tracing of sewage water to Likins Hall. How accurately can this determination be made? This depends on where the sewage water is collected. In a notice to on-campus students, the university made this disclosure: "This [wastewater] testing does not identify individuals or even floors of the dorm or building that might have infections." This statement directly contradicts the published story in which pinpointing the building was crucial in the chain of events.

I also wonder how they deal with any mixing of sewage, or possible environmental contamination.

***

The consultant hired to supervise campus re-opening told a reporter: "if we had missed it, if we had waited until they became symptomatic and they stayed in that dorm for days, or a week, or the whole incubation period, how many other people would have been infected?" This interpretation assumes that the virus found in the sewage water came from the two students who subsequently tested positive, and were still infectious.

Wastewater testing is beneficial only if it detects active infections. So establishing timing is essential. It is possible that the infected sewage came from past infections but not from the two active cases - there is still a lot that is unknown about virus shedding in feces, only a fraction of infected people do it. I'm not saying this is what happened but the coincidence of (a) virus found in sewage testing and (b) cases found in building does not prove that the infected sewage came from those specific cases.

***

There are three types of university responses to Covid-19. The most risk-averse group has moved all instruction online. The second group is attempting a return to the classroom but will require comprehensive testing. The third group re-opens but is looking for shortcuts to avoid having to test everyone. Sewage testing is of interest to this final group of schools.

The Arizona case study provides some hope that testing wastewater may reduce the amount of individual testing necessary to monitor and control infections on campus. Conceptually, this is related to group testing (explained here), replacing the usual nasal swab tests with sewage tests in the first stage.

More work needs to be done to establish the cause--effect. For example, controls should be included in the study, testing a dorm where the sewage came back negative. Another reason for controls is to measure false negatives. A deeper understanding of the "schedule" of virus shedding during the infection period is necessary. Measures to rule out contamination help reduce false positives. As with other tests, the scientists should publish the accuracy characteristics of the wastewater test.

Answers on excess deaths, test accuracy and quick testing

The recent posts have sparked some nice discussion here and elsewhere. This post collects additional comments I've since made.

Excess deaths caused by the lockdown

There was a verbal scuffle when reporter Jonathan Swan interviewed President Trump a few weeks ago about how to measure deaths due to Covid-19. Should we look at deaths per case (mortality rate) or deaths per million? In two recent posts (1 and 2), I discussed the visualization and modeling of "excess deaths," which is a better way of measuring the pandemic's impact than both of those popular metrics.

The starting point of an excess death analysis is neutral and apolitical. Based on historical patterns, we know that X people tend to die in the weeks from March to the present. Instead of X, the actual deaths (by any cause) have spiked in most U.S. states by 50% to 700% of the normal level in the worst weeks of the pandemic.

These "excess deaths" are not explained by any of the known causes. A majority of these deaths can be explained by deaths attributed to Covid-19, a novel cause of death. Most scientists believe that the unexplained excess deaths are probably under-counts of Covid-19 deaths. Should convincing data emerge, other causes of deaths could be accepted.

Some readers asked whether "excess deaths" may be caused by the lockdown rather than the virus. The lockdown does not kill anyone directly. What they mean is the side effects of the lockdown, such as more suicides from depression due to lockdown, or more cancer deaths because the patients were turned away from hospitals.

This is a logical argument but let me explain why this argument seems like a dead-end:

• The biggest challenge is the scale of the gap. A 50-percent to 700-percent gap is not easy to reconcile. Cancer is 20 percent of all U.S. deaths in 2017 (source). Let's say half of those occur between March and July. Then, if 2020 is similar to 2017, we should expect 300,000 deaths during those months. If we want cancer to explain 100,000 excess deaths, we're saying the lockdown caused cancer deaths to rise by 33 percent. That's a huge jump for a well-known disease that requires investigation. And we still have another 100,000 excess deaths to explain.
  
  
• Other than rhetoric and anecdotes, I haven't come across any serious attempt to quantify these lockdown side-effects. If some cancer patients have died because they are turned away from hospitals, then they should start investigating people who died at home from late-stage cancer. How many such deaths can we confirm? It's plausible that such deaths exist but I doubt this will make a dent in explaining the 200,000 excess deaths.
  
  
• Most cancer patients who died at home because they couldn't schedule surgery at the hospital during the pandemic were seriously ill, and would have died eventually of cancer. The side effect of the lockdown is accelerated deaths. In the world of "excess deaths," an accelerated death contributes to one positive excess death now and one negative excess (avoided) death later. After lockdown is called off, we should experience a reversal of these excess deaths. Will we find hundreds and thousands fewer deaths from all non-Covid19 causes in the aftermath? I don't think so.

 

False positives in the NFL

News broke that many recent positive test results on NFL players turned out to be false positives. This has been described as a "testing debacle."

First thought: false positives are the price to pay for being able to trust negative results. Notice that the NFL is not re-testing negative results; the NFL retests positive results. In order to afford such a luxury, the negative results must be rock solid. If the test comes back negative, the person is more or less certain to not be infected. This is how a good coronavirus test should behave. A false-negative mistake is an infectious person who spreads the virus unknowingly. For a diagnostic test to have fewer false negatives, it will have more false positives.

Second thought: testing errors, when interpreted in a broad sense, can be caused by operational mishaps. From what has been disclosed so far, it appears that the NFL contracts a single private testing company to perform all coronavirus testing, and that business operates labs in multiple states. All the false positives came from the New Jersey lab. This might be traced to say a contamination of equipment at the lab, just for argument [PS. That's the official reason given by the lab now.]. If true, the inaccuracy has nothing to do with the chemistry; the error came from lab operations.

All accuracy metrics published by test developers assume zero operational mishaps, 100 percent complicance, no gaming, etc. I brought this important point up in Chapter 4 of Numbers Rule Your World (link) - an entire chapter devoted to the complex issue of testing accuracy.

This is why when discussing group testing in this post, I singled out operational mishaps that could happen because group testing adds complexity.

 

Wishful thinking on treatments

The U.S. President has endorsed plasma therapy as the latest best therapy for Covid-19. The FDA has given it emergency use authorization. We should be clear that this is temporary, pending the results of proper scientific testing, with randomized controlled trials that prove plasma works.

I understand the media love the feel-good stories about people who insist that plasma saved their lives. But these survivors don't know. Because if plasma indeed saved their lives, then had they not been given the blood transfusion, they would have died. But no one knows that since they got the plasma!

It is entirely possible that plasma worked for some specific cases. The point of science is to establish that it would make a difference for enough patients to matter. [PS. The FDA Commissioner has already sort of back-tracked.]

Elsewhere, on college campuses and in professional sports, the testing labs are turning to quick saliva tests to scale up testing. This is of course a good thing. There are several such tests developed (I think I've read about those made by Yale, University of Illinois, Rutgers, etc.). The developers of these tests make many magical claims (see here). The quick test is supposed to be just as accurate as the nasal swab PCR test; much faster than that test; can pick up even lower amounts of virus; can provide a numeric scale of the amount of virus, not just its presence or absence; etc.

I find these claims hard to believe, except the efficency claim. Saliva should contain less materials mixed with unrelated fluids. It's probably less accurate. But the benefit of scale is obvious. Since it's used primarily for "surveillance testing", let's hope it doesn't generate too many false negatives, who become silent carriers.

A data communications starter kit for university re-opening

 

It's August, and many universities in the U.S. are attempting to bring some or all of their students back for in-person instruction. Within weeks, sometimes even before the first class, several campuses have already reported dozens or even hundreds of confirmed cases of infection by the contagious novel coronavirus. School administrators are justifiably worried that the virus, once it gets transmitted to older or more vulnerable community members, would cause grave illnesses and even deaths.

It's heartening to see that campus groups are demanding more data transparency from their schools (e.g. this open letter from public health professors at the University of Georgia). In response, some universities have rolled out dashboards, which is just a fancy name for data reports, iconically sprinkled with statistical graphics.

Nevertheless, the standard practice of minimal reporting of campus-wide totals represents much motion but scarce substance. Serious analysts will learn next to nothing from tallies that mix entry testing with ongoing surveillance testing, multiple tests of the same individual with tests of different individuals, etc. In addition, the focus of data reporting must expand from reporting outcomes (how many cases are there?) to reporting operations (are people complying with testnig? are people wearing masks?) in order to reveal opportunities for mid-course correction.

In this post, I outline a comprehensive plan for data reporting that should arm the university communities with the necessary data to assess the level of risk on your campuses.

I call this the aspirational plan because few school administrators will commit to this degree of transparency. Data reporting is political - whether it happens at schools, businesses, governments or organizations. It's quite possible that some officials secretly monitor some of the listed items. That'd be better than not tracking them at all. It falls on campus activists to push their administrators to collect the right data, and release more information.

As you read through this plan, I hope you develop some respect for the profession of data reporting (in business, this usually belong to the business intelligence team). This is not some boring job but a creative and thoughtful endeavor of utmost importance.

***

The ultimate goal of any data reporting plan is to keep the campus safe and open after it re-opens.

In putting together the aspirational data reporting plan, I start by listing the keys to sustained re-opening, based on my interpretation of statistical models built by Cornell and Georgia Tech researchers. Click these previous posts (here and here) to learn more about these models.

(A) Stem the import of virus onto campus.

This can be broken down into two objectives:

(A1) Ensure there is no virus on campus at the start of the semester. This is typically accomplished through comprehensive testing of all returning people, and isolation of the positive cases. [Notice that many Covid19 measures are as aspirational as my data reporting plan. We hope there is no imported virus but everyone assumes leakage is inevitable.]

(A2) Ensure there is no import of virus onto campus during the semester. Various measures are implemented to identify visitors and require self-reporting of health statuses.

(B) Snip transmission chains aggressively:

If the virus seeps onto campus (a certainty), we need to

(B1) find the infected cases quickly (bearing in mind silent spreaders), and

(B2) prevent the virus from spreading widely.

If a campus achieves both of those objectives, the mathematical models predict that infections can be contained during the semester, and students won't be sent home.

***

A data reporting plan must address the above two success factors separately. Over-aggregation of data is an adversary throughout the report design process.

How to monitor the import of virus at the start of the semester

This is one of the simpler tasks. Most dashboards include the number of tests, and the number of positive results. These metrics are not sufficient.

The tallies must not mix together entry testing and ongoing surveillance testing. Entry testing should have a section of its own. In order to interpret the number of tests, we must report the number of returners. It's even better to also count the number of unique individuals tested. Some schools require those already on campus to get tested. If so, those data should be reported in a separate section, not mixed with the data for returners.

[If your school isn't doing any entry testing, or some washed-down version of it, you're dealing with a different problem. Allowing virus to seed itself around campus is the surest way of enabling outbreaks; this is the key epidemiological lesson from the past few months.]

How to track the import of virus throughout the semester

The next task is to track the import of virus during the semester. We've heard many college presidents and deans of student life issue warnings blaming mixing of on-campus and off-campus groups for community spread, and yet we've not seen any university with a data reporting plan capable of tracking this crucial factor. In fact, the Cornell model (see here) assumes, gasp, no mixing of on- and off-campus people so in a sense, this is a blind spot for administrators.

To simplify the language, I'll use the word "visitors" to include anyone showing up on campus who does not reside on campus. The largest groups of visitors are off-campus students and staff, and guests (such as academic collaborators and vendors). At the minimum, a data reporting plan must include an estimated number of visitors circulating on campus. Roughly speaking, this is a function of entries and exits from campus, which are monitored. (Please talk to statisticians on your campus.)

In other words, if the data report contains the number of entries to campus (or campus buildings), the number should be broken down into visitors versus non-visitors. Lest you worry, we don't care about the names of people entering and exiting; for example, if one person enters 10 buildings, we count 10 entrances because we want a measure of contacts, not individuals.

"We don't have the data" is the excuse frequently used by those who prefer not to see the data. Having a reasonable estimate of the number of visitors circulating on campus is so critical to monitoring on- and off-campus mixing and any course correction that administrators should sit down with statisticians to come up with a plan. (Or, hire me or other consultants. We're not that scary!)

How to find virus that has already entered your campus

The next section of the report deals with finding any virus that has entered the campus. This is another tally of test results. The biggest concern is compliance, an issue I raised when discussing the Cornell and Georgia Tech models (see prior posts here and here).

Some schools including Cornell and Georgia Tech have adopted a comprehensive (aka surveillance) testing plan. Most schools take a "see no evil" approach, asking people with symptoms to volunteer for testing (already failed at the national level). Some schools use a limited random testing program. For comprehensive or random testing, the school should report how many invitations were sent out, and how many complied. For self-driven testing, the proportion of unique community members who have been tested in the last X days may be of interest.

It's impossible to interpret aggregate test data. Any serious report about testing must break down the tallies by the type of testing (comprehensive, randomized, self-reported). Other useful breakdowns are type of affiliation (undergrads, grads, faculty, staff, etc.), age groups, gender, subdivisions (e.g. schools). Such analyses allow administrators to see early warnings and take action, such as when they discover that certain communities are flaunting testing requirements.

How to measure community spread

The last section of the reporting is also the most interesting: we want to measure community spread. The standard practice of counting number of confirmed cases is useless. It's like standing by hopelessly watching a runaway train. We focus on the process of spreading, not the outcome of spreading. The goal is to estimate the number of contacts.

The starting point, and easiest, metric is class attendance. The entries and exits from buildings, already mentioned, should be a part of this.

Tried-and-true data collection tools are very useful here. Literally count the number of people walking past selected hotspots on campus. Count the number of masks. I'm sure you can find a few statistics or data science students willing to help out!

In addition, new-fangled AI tools can help. There are dozens of businesses selling (out your) location data collected from cellphones and apps. Surveillance camera footage can also reveal traffic intensity of pedestrians, cyclists, and vehicles. If this is too esoteric, head over to the data science, computer vision, or engineering departments and I'm sure they'll be more than happy to assist.

The data report should not print all these details. It should track a few metrics, such as class attendance, head counts, mask counts, and vehicle counts. These tallies can be samples (such as at hotspots), as we are only interested in trending.

How to monitor operating capacity

For schools that have credible contact tracing and quarantine procedures, there should also be a separate report on relevant metrics. The number of people under quarantine is important but also the average or median length of time spent under quarantine. The number and types of contacts shed light on the transmission chains.

***

The last piece of the puzzle

If any school can roll out the aspirational data report, I'd be overjoyed. The community members will have so much useful data, and administrators will be able to course correct based on trends. As my readers should expect from me, I must point to the elephant in the room.

I must admit that even this aspirational plan has one missing element. It does not measure what economists call the "externalities" of the re-opening. What is the impact of re-opening on the public health of the surrounding community, the people who are not affiliated at all with the university?

People live in areas, or even the same buildings, popular with off-campus students and staff. These community members may come in contact with students or staff at neighborhood restaurants, supermarkets, public transport, or by just walking past each other. From what we know about the coronavirus, outbreaks on campus easily spill over to neighbors. Some coordination with local public health authorities will be needed to address this difficult problem.

To put it bluntly, even if there are no deaths on campus, some hospitalizations and deaths in the surrounding community are by-products of the re-opening. Universities should not turn a blind eye to those.

 

 

Excess deaths, reporting delays, statistical adjustments, misleading headline

The New York Times published a very nice presentation on "excess deaths" in the United States from March through July. (link)

The following chart shows how public health policies utterly failed in many Northeastern states in the April and May.

In Connecticut, for example, in the middle of April, the number of deaths exceeded expected deaths by 2.6 times. In Massachusetts, around the same time, actual deaths exceeded expectation by 2.3 times.

A lot of work went into generating these charts - starting with meticulous data collection from each state, and as much as possible, reconciling how metrics are defined differently across states. Then, there is a bit of statistical modeling.

Modeling is required because "excess deaths" is - like most statistical quantities - not directly observable. Excess deaths is the difference between actual deaths (from all causes) in the presence of the Covid-19 pandemic and expected deaths (also from all causes) in its absence. The former is directly measurable but the latter is not.

To establish how many people could have died during a specific week in Connecticut if the novel coronavirus did not exist, statisticians look at deaths during the same week in that state in past years. Over a number of years, one should find some statistical consistency in the death rate. This illustrates the power of averaging.

If 2020 were a normal year, the chart should look something like this:

The above line is for Hawaii, which, because of its remote location, has not been affected much thus far by Covid-19. What you see is the actual weekly deaths hovering around the baseline (corresponding to the historical average), sometimes rising above, and sometimes dipping below.

The line for Connecticut, though, looks nothing like that for Hawaii. The most important feature is that the entire line sits above the baseline, meaning that there have been excess deaths for every week since March.

***

Ready for even more modeling?

For most states, the time line reaches the last week of July, which may surprise you. The first part of excess deaths - the observable part - counts all causes, obviating the need to verify coronavirus test results. The measure relies on death certificates. But it may take weeks to months to receive data on death certificates so how does NYT have estimates of excess deaths up to the week prior?

Does NYT have a crystal ball? In a sense, yes. It uses a statistical model to "top up" the incomplete deaths data in the most recent weeks. It turns out that actual deaths are not completely observable always. There is a reporting delay which means that the data for more recent weeks are incomplete. The data age like fine wine - the older the data, the more complete.

The NYT utilized a similar strategy as the model for projecting expected deaths - assuming historical patterns hold. The CDC keeps track of the amount of reporting delay. It might know that typically, 20 percent of the data showed up with one weekrepo's delay, 50 percent with two weeks' delay, and so on.

There is a crucial difference betweeen these two statistical models, though. The first model projects expected deaths in the absence of the novel coronavirus. The second model tops up the actual deaths reported at the time of analysis, essentially projecting the number of deaths already occurred but not yet reported. These actual deaths are in the presence of the coronavirus. This is the crucial difference. This second model has a further assumption - that reporting delays have not been affected by the pandemic.

The NYT team disclosed this assumption clearly in the text. This is what they mean by the following sentences:

Even with this adjustment, it's possible there could be an underestimate of the complete death toll if increased mortality is causing states to lag more than they have in the past or if states have changed their reporting systems.

It's quite likely that the volume of deaths, the budget pressure, and political interference have changed the pattern of the reporting delay.

The reporter then argued that the adjusted estimates are still better than the unadjusted ones. One can't refute that argument. Of course, this discussion assumes that the chart must show a time line leading to the present. The designer can choose to show only those weeks with virtually complete data.

Or, we can bring in a third model. This one looks at the early weeks of the pandemic, for which we believe almost all of the death-certificate data have been recorded. We can then estimate the error of the second model that serves to top up the actual deaths.

***

NYT's excess-death analysis has found over 200,000 more deaths (from all causes) in the U.S. between March and July than expected based on historical patterns. Given the U.S. counted about 140,000 deaths due to Covid-19 during this period, there have been about 60,000 unexplained deaths so far.

I find the NYT headline, "The true coronavirus toll in the US has already surpassed 200,000," alarmist. The analysis did not support this conclusion - unless they make the further assumption that the only factor causing excess deaths during those months is Covid-19. That is plausible but not fact-based. A more correct headline should be "Deaths due to Covid-19 may have been undercounted by up to 40 percent." That's also shocking and sobering.

Working with poorer ingredients

Antonio forwarded this Guardian article to me, with the headline “coronavirus may have been in Wuhan in August, study suggests.” This headline is based on a preprint promoted by Harvard, available here.

The report introduced two pieces of evidence: (a) occupancy of major hospitals increasing and (b) web search volume for Covid-related keywords increasing. The data sources were respectively satellite images of parking lot traffic, and search trend data from the leading search engine in China. This type of data analysis is typical of “big data,” which I define using the OCCAM criteria (see here).  

I have come across this report on a different site, and planned to write about it – as the latest example of what I call “story time”. It’s a feeling I frequently experience in the midst of reading studies. As I start to doze off from the lullaby of the data presentation, the researchers have moved seamlessly into creative narration – they’ve called “story time.” After I wake up, I realize that the story is only minimally connected with the data.

One of the telltale ingredients of “story time” is proxies. The most important sentence in the preprint is this one: “Between 2018 and 2020, there was a general upward trend of increased hospital occupancy as measured by the parking lot volume proxy.” (my emphasis). Everyone should master the skill of “proxy unmasking”.

What must be true in order for parking lot volume to be an effective proxy for hospital occupancy? An incomplete list includes:

• People who are sick drive to one of those six hospitals in Wuhan.
• People who drive to these hospitals park their vehicles in the hospitals’ parking lots.
• People who park in these parking lots are patients, not staff, or random people say visiting places nearby.
• These hospitals have enough capacity in their parking lots.
• People who visit the hospital are themselves sick, not relatives or friends of someone who is sick.
• People who visit these hospitals will occupy hospital beds.
• People who are hospitalized keep their vehicles in the parking lots until they are discharged.
• People who are hospitalized have the novel coronavirus (as yet unknown pathogen).

Notice that the data from satellite images confer none of this information. They are simply assumed without comment. I sensed “story time” when I realized that most of these assumptions are contradicted by my own experience. Even if I believed them, I make a distinction between data and assumptions.

The same proxy unmasking applies to the web search data.

It's really a tough task to prove that one's featured story is the right story as opposed to a story. For example, here is the chart showing the trend in search volume for "diarrhea":

The story of the preprint focuses on the first and third vertical dashed lines. There is a small spike in the red line (diarrhear) around the time of the first line (August) and then a huge spike around the time of the third line (January 23, when the first case was identified). The blue line for "cough" is taken by these researchers as a "control," saying that its trend is explained by seasonal influenza. Given that cough is still a key symptom of Covid-19, one should expect the volume of "cough" searches to become elevated when both influenza and Covid-19 are affecting search volume. Further, the period to the right of the third vertical dashed line was the epidemic period - when more people were worried about community spread, and what can explain the dramatic drop in searches?

***

When the quality of data is lower, the standard of analysis must be set higher. A chef needs a bag of tricks when working with lower-quality ingredients. Lower-quality data cannot be an excuse for lower standards. Quite the contrary, "big data", OCCAM data demand more stringent benchmarks. Is this counterintuitive?