Big Data, Plainly Spoken (aka Numbers Rule Your World)

Two side issues as we hurry to a coronavirus vaccine

My review of the Moderna trial protocol (parts 1, 2) shows that by and large, the design follows established standards, with only a few places where the researchers have selected less than the gold standard, which is understandable given the urgency of finding a vaccine.

That said, a rushed job with the desired outcome of a vaccine is going to increase the small odds of a false-positive result. This may come about in unconventional ways.

***

Within each trial, as I pointed out in part 2 of the Moderna protocol review, the threshold for statistical significance has been made drastically more rigorous to account for the interim analysis based on a much smaller sample of data. That's the proper thing to do. The design is generated so that the overall probability of a false-positive across all attempted analyses and endpoints is under control.

Of note, the world is running a dozen or more trials of competing vaccines at the same time. Even when each trial controls the chance of a false positive, the overall probability of a false positive coming from any of the slate of trials is not under control.

Think about elevator maintenance. For any single elevator, the manufacturer might promise a breakdown rate of below one percent. Now, across all elevators of all brands in a single town, the chance that at least one elevator breaks down in a period of time is much higher than one percent. That's because each elevator has a one-percent chance of breaking down, and any elevator may break down at any moment in time. The more elevators there are in the town, the higher the chance of at least one breakdown.

The more vaccine trials are started, the higher the chance that a positive result is a false alarm. No individual clinical trial is controlling for this kind of risk; only the FDA can address this.

***

Another rarely addressed issue is statistical validity, that is to say, how representative are the participants of the trial relative to the people who are likely to receive the vaccine once approved.

The vaccine manufacturers are engaged in a complex competition. The first vaccine to get approved has a special status because it becomes the standard of care. Other trials may have to use the first vaccine as the comparator, rather than the placebo. Because of the provision for interim analysis, the sooner a trial completes enrollment of participants, the sooner it can begin analysis, and the approval application. Interim analyses are not triggered by the total enrollment, as in most clinical trials, but by the total number of infections.

Thus, the incentive is to get as many people enrolled as quickly as possible, and to dip into higher-risk areas as much as possible. In the extreme situation, if all enrollment is focused on only a few highest-risk areas, the trial may become the first to accumulate enough case targets. (A possible dent in this strategy is if people's willingness to participate is inversely proportional to how much risk they face.)

While this competition plays out, the trials that have the most biased samples, concentrating on the regions with the highest "yields", will likely win the race for time. This creates a problem for the FDA because the results would score lower on statistical validity, and have to be de-biased. (Such a sample would however provide better data on higher-risk subgroups. The trade-off is that the vaccine is supposed to be for everyone.)

Perhaps the enrollment can be skewed towards severe cases at the start, and then broadened over time to encapsulate the general population. This mimics the likely plan for rolling out the vaccine.

***

I should underline that these are not show-stopping issues. They are interesting statistical side-notes that may seem theoretical when you read about them in a textbook -- until the pandemic happens.

Behind the proposed strengthening of FDA vaccine approval rules

News just broke that the FDA may announce more "rigorous" standards for a coronavirus vaccine to get approved. This indicates a willingness to listen to some of the critics of the perceived rushed job.

The specific rules mentioned in this Washington Post article are:

1) Participants must be followed for a median of at least two months, starting from when they receive the second dose

2) The analysis must be conducted after at least five severe cases of Covid19 have been observed in the placebo arm.

3) The analysis must be conducted after some minimum cases of Covid19 in older people has been recorded.

***

Rules 2 and 3 are fine in spirit. It's important that the vaccine works for severe cases and older people. These rules ensure that all trial analyses include the two groups that are of great concern to the medical community.

Recall that the case definition uses mild symptoms instead of severe symptoms. Since a lot more people develop mild symptoms, it's possible that the interim analysis contains only mild cases. It's also possible because of self-reporting that the cases used in the interim analysis involve only younger people.

I suspect those scenarios are unlikely anyway. Because testing is triggered by self-reporting, severe cases are more likely to surface than mild cases. So long as they have sufficient older participants in the trial, they should show up in the interim analysis because older people tend to have more severe illnesses.

Nevertheless, if the vaccine is approved by the end of 2020, it will be on the strength of an interim analysis. Such analyses are never designed to make statistically reliable statements about subgroups, such as older people or severe cases. This is another reason why the trial should run to completion even if the FDA grants interim approval.

***

Revised rule 1 is misleading and potentially worrying. 

Let's talk about its good intention first. Rule 1 sets up a second criterion for triggering the interim analysis. Previously, the only criterion is to exceed a prespecified total number of cases. This creates the potentially undesirable scenario in which enrollment is supercharged (either by pushing marketing, or relaxing admission standards), or the true case rate is markedly higher than assumed, such that the first interim analysis is triggered when most participants have barely taken the second dose.

By setting observation time as a second criterion, the new rule provides more confidence that the protection afforded by an approved vaccine lasts at least 2 months.

As I explained in this previous post, Moderna does not count cases until 2 weeks after the second dose. So if they track people for two months, it's really only 1.5 months of observation time. That isn't a lesser worry. What perturbs me is the use of the word "median".

Median is the middle person so the statement actually allows up to half of the participants to be observed for fewer than 2 months. Further, it is highly unusual and in fact statistically fallacious to perform analysis in which different participants are observed for different amounts of time. I hope this is a communications faux pas, a misunderstanding on the part of the journalist. The sensible rule is to require every participant (not the median participant) to be observed for at least 2 months (preferably longer) after the second dose.

 

 

P.S. [10/6/2020: A group of scientists and medical experts agree! STAT News writes about their pushback on rule #1. They want "a minimum of two months' observation for all participants".]

Notes on the Moderna vaccine trial protocol 2

This is Part 2 of my notes about the Moderna vaccine trial. Read Part 1 here.

***

Durability of Protection

As described in the last post, the full analysis will not take place until middle of 2021, so the vaccine developers are hoping that their vaccine would show enough during the first or second interim analysis to accelerate the time line. Early reads increase the chance of false-positive findings, as I outlined in a previous post. To counter this, the study drastically lowers the p-value threshold to reach statistical significance in the interim analyses. (Instead of p = 0.05, the first interim analysis in effect sets p = 0.0002.) This is a proper thing to do.

However, if the first interim analysis results in a vaccine approval, then the participants will have been observed for at most 2.5 months from the second dose. We will not know if the vaccine offers protection beyond 2.5 months.

According to the protocol, Moderna "intends" to continue the study after if it receives interim approval. There will be an ethical debate about whether those who received placebos should be given the vaccine should it win approval. Some may oppose withholding a "proven" treatment from these participants. If they are given the vaccine, we lose the ability to measure protection beyond 2.5 months.

Sample Size

If the first interim analysis is successful, then the vaccine would receive approval based on about 50 total infections. Assuming that the vaccine effectiveness is 50%, then we expect half the number of infections on the vaccine arm relative to the placebo arm. So the vaccine arm should account for one-third, or about 17 infections. That's an awfully small number.

Nevertheless, if approved, the vaccine would have met the more stringent statistical significance threshold so the small sample size is not a serious problem. But the required number at the first interim analysis is small enough as to make many statisticians nervous. Continued tracking beyond the first analysis will be key to make sure the early result is not a mirage.

Defining Trial Outcomes

In the primary analysis, the "endpoint" (outcome) is infection, which requires a PCR test confirmation plus some pre-specified level of symptoms.

Some experts are arguing that they should count severe cases only because we don't really need to stop the mild cases. The chosen endpoint includes both mild and severe cases but we expect that most of the cases will be mild so there is concern that the signal (severe) to noise (mild) ratio is not high. Of course, if only severe cases are counted, the trial would last longer.

The opposite criticism is that they are by and large missing asymptomatic cases because almost all the testing are triggered by symptom reporting. One can argue that we need to stop asymptomatic spread because any viral transmission may eventually produce severe cases.

Moderna is taking the middle ground here.

Unusual Sources of Noise

In running any experiment, we'd love to eliminate as many sources of noise (variability) as possible so that any observed differences can be attributed to the vaccine. Some practical issues for this vaccine trial may produce noise, which affects our interpretation of the results.

The FDA permits accommodations due to the pandemic. This means that in-person visits may be replaced by phone calls. That itself is not a problem if all visits are converted to phone calls. The problem is that swaps occur based on request so they'd end up with some in-person visits and some phone calls. This creates a possibility that the method of the follow-up may be associated with, or even cause, different outcomes.

Besides, the protocol lists a variety of tests used to confirm infection. The nasal swab is preferred but saliva and respiratory samples are also used. These tests have different accuracies and characteristics so the mix of test types may affect the count of confirmed cases. For example, people living in high-risk areas are more likely to submit saliva samples by mail versus nasal swabs from in-person visits.

Uncertainty Around the Case Rate

The design assumes a 0.75% baseline case rate within six months but the researchers acknowledge uncertainty around this number - that's why they end the enrollment based on the number of cases rather than the number of participants. What if this assumption is wrong?

In particular, this base rate seems fairly low. Those who hope for "herd immunity" might be shocked to learn that the virus is expected to infect less than 1 percent of a community over six months. At that sluggish rate, when will we reach 70 percent or so prevalence?

All sample sizing requires guessing at the base rate, and sometimes we can get it wrong. If the base rate is somewhat higher than 0.75%, then the trial will get infections faster than expected, and thus end enrollment earlier than expected. The real case rate among those participants can be, say, 2%.

Here's the rub. If 2 percent had been used as the base rate in the design of the clinical trial, statisticians would have stipulated more than 53 infections to trigger the first interim analysis. This means, if the observed case rate is higher than expected, the first interim analysis suffers from insufficient data. Smaller-than-expected sample size makes it more challenging to beat the significance threshold.

***

The Moderna protocol is reasonable: there are some unconventional elements, but in an emergency, we'd have to accept some accommodations. It's still hard to imagine how any of these trials will get to a strong result by end of October or November.

News about the vaccine trials is coming fast. Just this weekend, the Washington Post reported that the FDA may tighten the vaccine approval requirements a little to pacify critics and build trust. I have another post up discussing these modifications.

 

 

 

 

 

Good news from Cornell

Source: Redbubble.com

Some colleges and universities have re-opened for in-person classes for the fall term. Previously, I examined the Cornell model (link), which outlines a very aggressive plan for preventing the virus from entering the campus, and then snipping transmission chains during the semester. Also, we learned that several schools that opened in August quickly ran into trouble as hundreds if not thousands of cases surfaced, and some decided to return to online teaching. (link)

I just attended a fantastic webinar by Peter Frazier, who's part of the Cornell team that did the statistical modeling work supporting the re-opening. The webinar was sponsored by Columbia.

It's about three weeks after Cornell reopened for in-person instruction, and the situation in Ithaca is significantly better than on most campuses. The following chart best illustrates why the team is taking a victory lap, although they absolutely realize that there are still risks ahead.

[Applause]

[Note: I expunged a number of Excel defaults from the above chart, including line colors, legend, and date labels. Also, replaced infections with cases, which is a more precise description of the metric.]

As I indicated in the previous post, if Cornell's plan proves successful, it would repudiate what the U.S. government did in the early phase of the pandemic. It shows that extremely aggressive surveillance - and in particular, broad-based testing - will dampen case rate to around zero, and keep it there. That's what other countries like China and South Korea managed to do.

At the end of the talk, Prof. Frazier offered advice to data scientists who want to make real-world impact. I agree with every bullet point and am reiterating the whole list here.

1. Focus on solving the real-world problem, rather than what can be published. He discloses that they have not written a paper about the work they did here. This is advice I give to researchers who want to transition into industry positions.

2. Create mental models of your computer models. A different way of expressing this point is that the modeler should develop some intuition behind how the models work, even if the methodologies are complex and opaque.

3. Communications. He spent a lot of time explaining the model to the public, and the team was open to feedback and continuously listening to the community. People are emotional about the subject, and some of the commentary were harsh but it's part of the job. Frazier's presentation today is impressive evidence of his communications skills.

4. Understand the political environment. Different groups have different objectives and biases.

5. He calls this "luck." The key decision-makers at Cornell are scientists by training, so they have an easier time persuading them to believe in science. That's an important point when it comes to "choosing your boss." For many data scientists, jobs have been plentiful, and job satisfaction will depend on whether you're fighting with your boss all the time. Imagine the scientists at FDA or CDC who had to deal with political appointees who are reportedly pressuring them to alter their language. Those meetings would be a completely different beast than what the Cornell team had to deal with.

Throughout the presentation, Prof. Frazier pointed to other "luck" factors that favor Cornell's response. For example, Ithaca is very isolated. He said that the nearest towns are an hour's drive away. Also, their veterinary school has abundant expertise in PCR testing, and has been extremely helpful.

He also reported that pessimists were much more vocal than optimists, and so assumptions may have been driven more conservative than warranted. Modelers may also have a conservative bias because better safe than sorry.

***

In his presentation, Prof. Frazier demonstrated the power of statistical modeling. The core simulation model is highly complex, with many moving parts and assumptions, as I discussed in the earlier post. The assumptions are necessary because most of those parameters are not directly observable, and highly uncertain. While the model was not "accurate" in the sense of forecasting precise outcomes, it accurately predicted the general trends. Making assumptions is not a bad thing!

Then, we saw Occam's razor in action. Statisticians have long preached that models should be as simple as possible, but not simpler. The core model is actually built on top of simple epidemiological models, and those models alone were able to capture the general trends.

***

Since the time I reviewed the Cornell plan, there were a couple of important additions that probably contributed to the on-the-ground situation being better than the original projection.

First, the frequency of surveillance testing was doubled for undergraduates from once a week to twice a week. The insight is to tailor the testing rate to the amount of contacts expected in different subgroups. The mathematical models help calibrate what level of testing is required to keep the infection rate below a critical threshold.

Second, to work around the limited capacity for contact tracing, and to take advantage of their expertise in PCR testing (via the vet school), they are testing the entire social circle for each positive case. They call this "adaptive testing". The goal is to reveal as many hidden cases as possible before the virus has a chance to spread further.

Testing is a huge commitment in terms of money, resources and time. Cornell is conducting over 5,000 tests per day - obviously, they are returning results very fast. It appears also that the students are cooperating, and grasping the key issues. This pandemic is a test of community spirit.

***

Perhaps lost in our national conversation is the decisiveness of early, aggressive action. Public-health experts have stressed this point from the start, and the Cornell experience (relative to other universities) shows they were right.

Take compliance to testing as an example. When there isn't much virus in circulation, we can accommodate some degree of non-compliance because the non-compliant person will rarely run into an infectious contact. When the epidemic is in full swing, the chance of interfacing with infectious people is much higher, and non-compliant members are more likely to spread the virus around.

We can't think linearly. Any community is moving toward one of two final states: almost no virus in the community, or almost everyone gets infected at which point the virus also spreads slowly because most contacts are between infected people. Without intervention, the second state is inevitable. Once an epidemic gets out of control, it's very hard to revert to the first state.

Cornell's success to this point is showing us that places like China and South Korea can indeed eliminate the virus with early aggressive actions. These countries don't have to fake the data if they did the right things.

Whether the success will last is an open question. It's only three weeks into the term. Prof. Frazier noted that they strongly discourage students from traveling outside Ithaca. I'm not sure this is possible during breaks and Thanksgiving. Similarly, countries that successfully fought off the first wave still face risks if the virus is re-imported from other countries that failed in their initial response. For Cornell, I'm guessing that they will have to roll out arrival testing after each long break.

Notes on the Moderna vaccine trial protocol 1

Bending to outside pressure, Moderna has made public one of the documents submitted to the FDA that provide more details on how they are running the vaccine trial. (PDF here) This is a reassuring move, and I applaud them for taking this step.

In a previous post, I raised two key questions I would like to find answers to, and after reading all 135 pages of the protocol, I can tell you the answers.

How is efficacy measured?

Vaccine effectiveness (technically, efficacy) is the reduction in infection rate among the vaccinated relative to the baseline infection rate among the unvaccinated. A 50% effective vaccine is one that halves the baseline infection rate. (For more on vaccine effectiveness, see this other post.)

An important detail is how this infection rate is measured. According to the trial design documentation, Moderna will confirm infection using PCR nasal swab tests (primarily). To be precise, what they measure is confirmed cases rather than infections.

What is the analysis schedule?

Until the documentation is made public, there has been a lot of sloppy reporting about when results will be available. Now we know. The primary analysis will be conducted when the total number of confirmed cases (across both test and placebo arms) exceeds 151. The expected time of this analysis is six and a half months after the last trial participant receives the second dose. By my estimate, this will be mid June 2021.

The trial started enrolling participants at the start of August. Assume that enrollment is complete by the end of September, the second dose of the last participant should occur at the end of October, the data for the primary analysis will be frozen by mid May 2021, and the trial result will be ready by mid June 2021.

Why do pharma CEOs keep saying they expect results by the end of 2020? That's because they are hoping one of two interim analyses might produce results so stellar that the FDA will have seen enough. The first interim analysis is triggered when the number of confirmed cases exceeds 53, which is 35 percent of the full sample. If it takes 6.5 months to accumulate 151 cases, it would be 2.5 months to get to 53. (In both instances, the half month is added because they don't start counting cases until 14 days after the second dose.)

For the first interim analysis, assume they only analyze participants through the end of August, the second doses should be completed by the end of September, the data will be frozen by mid December 2020, and so the earliest date by which the vaccine might be declared effective is mid January 2021, if everything else falls into place.

***

What to Look Out For

I read through the entire 135-page document. I prepared a list of notes about the clinical trial, mostly about statistical matters. These are placeholders for now. How much these issues matter depends on what happens during the trial.

Blinding

The trial is observer-blind, less than the gold standard of double-blind. Blinding is hiding whether a participant has gotten the vaccine (or drug) or the placebo. Double-blind means neither the researchers nor the participants are told who got what shot, until the data are ready for analysis. Blinding prevents humans from misbehaving as a result of knowing the treatment assignment. In this Moderna trial, the person administering the shots is not blinded. There may be a scientific reason for why some researchers cannot be blinded for this study but I didn't find an explanation in the documentation. (I checked the Pfizer-Biontech trial, and it is also observer-blind, not double-blind.)

Many outside scientists and reporters have missed this detail, and mis-reported the Moderna trial as double-blind (e.g. here, here, here).

Analysis Population

The primary analysis is "per-protocol", which is less than the gold standard of intent to treat. This is always a controversial subject so let me explain the disagreement. In a canonical clinical trial, half the participants get the vaccine (or drug), and the other half receive the placebo (or comparator). Researchers take great care to randomize assignment to these two arms. If each participant has equal chance of being assigned to either arm, then we have created the condition of "all else being equal" so that when we compare the average person across the two arms, any difference can be attributed to the vaccine (or drug).

In practice, some participants will drop out of a trial. It is quite possible that the dropout rate is not the same across the two arms. For example, if the treatment causes irritation, more people in the treatment arm might quit. The statistical debate is whether to include the dropouts in the analysis. The gold standard of intent-to-treat analysis says to stick as close to the original population as possible. Dropouts are therefore included (up to the time of dropout). In a per-protocol analysis, the dropouts may be excluded as if they never existed. The proponents of per-protocol argue that the final outcomes of these dropouts are not known, and so including them might be "unfair" to the treatment arm.

By contrast, supporters of intent-to-treat (including me) don't buy that the act of dropping out is independent of treatment, and worry that the loss of randomization invalidates the "all else equal" condition. For me, the most important argument is the practical point of view: the per-protocol findings apply only to people who take the treatment without dropping out!

Dropouts are not the only subset of participants who might be excluded in a per-protocol analysis. It's just easier to explain.

Back to the Moderna trial. It appears that dropouts prior to the second dose will be excluded from the per-protocol analysis but those who discontinue after the second dose will be included.

Infections Before Finishing Treatment

In the Moderna trial, another interesting subset consists of participants who get infected and then are excluded from the per-protocol analysis. In the primary analysis, Moderna starts counting infections 14 days after the participants get their second dose.

The trial design stipulates that people who have confirmed infection before the second dose will be ineligible for the second dose. All such people therefore will not complete the protocol and be excluded from the per-protocol analysis. The good news is that Moderna has committed to performing an intent-to-treat analysis for comparison. However, the final decision will be based on per-protocol, and there is no guarantee that the intent-to-treat analysis will be published for outside evaluation.

Let's say we're behind at the end of a(n American) football match, and our coach decides to go for an on-side kick and then drive for a field goal. To win the game requires executing both steps. If the on-side kick fails, the other team will run out the clock. (Last year, on-side kick success rate in the NFL was only 6 percent.) If the on-side kick fails, the per-protocol analysis excludes this match because the team didn't execute the entire plan. If the on-side kick is successful and the team gets into field-goal position, the per-protocol analysis includes this match. So, the estimate of the success rate of the two-step plan will be over-optimistic.

Here's the practical point of view. If the vaccine requires two doses, and someone gets sick after the first dose, the vaccine is ineffective for that individual. It doesn't matter that the said individual has not received the second dose; it is meaningless to administer the second dose.

Excluded Infections

Then, there is the subset of participants who get infected during the first 14 days after the second dose. For an unexplained reason, Moderna isn't counting infections until two weeks after the second dose.

This subset will be included in the per-protocol analysis because they took the full two-dose program. However, the outcome for these participants will be "censored," meaning instead of being treated as infected, they are treated as if they have dropped out of the study just before their infections are confirmed.

Again, I take a practical view on these matters. If, after the vaccine is approved, someone gets infected a few days after receiving the second dose, the vaccine would have failed for this individual.

So when the results come out, keep an eye on whether there is a differential likelihood to get early infections on the vaccine arm relative to the placebo arm. The good news is Moderna has committed to conducting additional analysis that counts infections from the first day after the second dose for comparison. However, the final decision does not have to take into account this analysis, and it may not be publicly available.

Counting Infections

Measuring infections in a vaccine trial is always challenging because the participants are healthy. Contrast this with a clinical trial testing a treatment on sick people. Since it's unethical to inject virus into healthy subjects, we have to wait for infections to crop up via community spread.

Moderna isn't very proactive in monitoring for infections. They rely on self-reported symptoms, which then trigger confirmatory PCR tests. For example, the only way researchers will learn about suspected infections during the first 14 days after the second dose is if the participants report symptoms. Not everyone who report symptoms will get tested; the researchers have some discretion for certain types of symptoms.

That's why we will never know the true infection rate. The reported rate should be considered the rate of confirmed cases.

***

These are the most salient information I learned from the just-released documentation on the Moderna vaccine trial. I have several more notes that will be written up in my next post. 

Masks and vaccines

United Nations Covid-19 Response

Wasn't planning on blogging today but the news cycle won't leave me alone :)

CDC Director Dr. Redfield made a bit of a stir yesterday when he told Congress - and I paraphrase - that masks may offer individuals greater protection than vaccines.

This is a very thoughtful statement which rewards some further comments.

***

The way medicine is practiced, we are often misled to over-estimate the impact of drugs and vaccines. When the doctor says take pill X to treat condition Y, we think if we take pill X, condition Y will be cured. In reality, for most conditions and meds, if we take pill X, there is a chance C that pill X will cure condition Y. C is not 100 percent, and is usually way below 100 percent.

Flu vaccines are said to be 50 percent effective or less. What does that mean? Perhaps, that half the people who've gotten flu shots will be protected while the other half will still catch the flu. This isn't how experts use the word "effectiveness" but let me stay with this intuitive definition for now. (I'll turn to the technical definition in the last part of the post.)

This situation reminds me of the famous put-down of TV advertising: half the budget is wasted but we don't know which half.

The trouble of course is that if you halve the TV spending, you will still waste half the money while also losing half the benefit. Same with the vaccines. If only half the people get vaccinated, a quarter of the people will be protected.

The FDA has announced that a SARS-Cov-2 vaccine with 50% effectiveness is acceptable. Staying with the intuitive definition, this means getting the vaccine shot is like buying a lottery ticket with a 50 percent chance of winning. This is not news though. Taking any pill X or undergoing any surgery  has always been a lottery with C percent chance of winning. The value of C depends on the treatment.

One of the reasons why some vaccinated people will still catch the flu is biological variation. Different people respond to the vaccine differently.

***

What about wearing a mask? The mask works as a physical barrier and its function is not affected by anything biological. The same mask worn properly confers the same degree of protection on anyone. Any decent mask blocks some fraction of virus from entering one's respiratory system. Imagine an infectious person coughing directly at me. The mask reduces the amount of viral projectile even if it is not eradicated.

In other words, wearing a mask has a high C. Even though I am a skeptic of many things, mask wearing is not one of them.

Whether the vaccine will work for me or you depends on biology. I have no control over that. Either my body responds to the vaccine or it doesn't. The mask, on the other hand, works on everyone who wears one. That's what the CDC Director is saying, and he's right.

***

Now that I've gotten your attention, let's switch to a proper interpretation of vaccine effectiveness. We have a coronavirus vaccine that demonstrates 50% effectiveness in a clinical trial. One possible scenario is it reduces the infection rate from 1% to 0.5%. (For reference, in the Moderna trial, they assume the baseline infection rate with no vaccine is 0.75% over a six-month period.) That's a 50% drop, and statistically significant.

For every 1000 patients receiving the vaccine shot, five patients will benefit from it as they would have caught Covid-19 without the vaccine. Another five will get sick despite being vaccinated. So C is 5/1000 = 0.5 percent for the vaccine. Getting vaccinated is like buying a lottery ticket with 5 percent chance of winning.

In medical jardon, we say the number needed to treat is 1000/5 = 200. For one person to benefit from vaccination, we must administer 200 vaccine shots. Similar to TV advertising, we can't know who would benefit ahead of time. Statistics tell us that on average, one out of 200 will.

Notice that C isn't 50 percent but only 0.5 percent. It can't be 50 percent because most people won't catch Covid-19 even without the vaccine. For every 1000 people vaccinated, 995 of them won't get sick. Of those 995, 990 would have stayed healthy regardless of vaccination. The 50% effective vaccine really only benefits 0.5% of those who took the shot. (Note again that I'm using the assumption from the Moderna clinical trial design.)

But this is not the full story. The practical effectiveness of the vaccine depends also on what proportion of the population takes the shots. Let's pretend that half the population get vaccinated. Then the 0.5% benefit of vaccination is halved. The infection rate in the population is reduced from 1% to 0.75% since the infection rate for the unvaccinated will remain at 1%.

Herd immunity now complicates this picture. If enough people get the vaccine, typically 60% or higher, the amount of virus circulating in the population is suppressed enough that the infection rate for the unvaccinated will dip below 1% so collective action creates a positive externality.

 

 

 

Confusion over case trends and positivity ratios, an aftertaste of targeted testing

We are now experiencing the aftertaste of targeted (aka triage) testing, a policy that has unsavory consequences that were predictable (link). Targeted testing is the idea that only people with severe symptoms or at high risk of infection should get tested for the novel coronavirus. Compared to a broad-based testing plan (whether comprehensive or randomized), targeting economises the number of test kits. The basis of any targeting strategy is the skill to predict who is infected. The group that is targeted should have a higher infection rate than the group that is not targeted. It's important to bear in mind that targeting not only suppresses how many people are tested but also affects who gets tested.

One way in which targeted testing hurts us is that it ruins our datasets from which we obtain insights about the pandemic. This post is prompted by Ken's comment on my previous post on casual presumptions of randomness:

The number of daily new cases in some countries doubled as fast as every 2 days. This didn't mean a greater rate of infections, it was simply a result of catching up all the non-tested cases.

The connection between infections, tests, and cases is complicated, and the media shift back and forth between the rate of cases (positive test findings) and the positivity ratio (proportion of tests that came back positive). I spent some time explicating these relationships, and hope that this post will succeed in helping you gain some intuition around those concepts.

We start by decomposing the positive test findings. In the flow chart below, starting with the dark gray box on the left side, we examine a fixed population of people. At any given time, the people are divided into three "compartments," the infectious people who currently have the virus, the susceptibles who have not been infected, and the recovered (which by convention, includes those who died from Covid-19).

Also by convention, anyone can be infected only once, and we thus ignore recovered people from further consideration. From the dark gray box emerge two paths that terminate at positive tests: the upper path consists of infectious people while the lower path, susceptibles. Some portion of the population are infectious; within that, some proportion have been tested for Covid-19, and within that, some proportion have tested positive. This last tally is one part of the confirmed cases. The other part of the confirmed cases comes from susceptibles, represented by the lower path.

The number of confirmed cases is the number of positive tests. This is a key metric on all Covid-19 dashboards. The other important statistic is the positivity ratio. This is the number of positive tests (represented by the dark red box on the right) divided by the number of tests conducted (the sum of the two yellow boxes).

The next diagram describes the drivers of the two dashboard statistics:

• the weights of infectious or susceptible compartments in the population (excluding recovered)
• the probability of getting tested among the infectious or susceptible group
• the weights of infectious or susceptible people within the tested subpopulation
• the positivity ratio for the infectious or susceptible people who got tested (call these the group-level positivity ratios to distinguish from the overall positivity ratio)

The overall positivity ratio is affected by three signals. First up is the rate of infection in the population, reflected in the relative weights of the I and S compartments. Second, the testing regime imposes a selection effect on these weights, producing the second set of weights. The notorious triage or targeted testing protocol restricts testing to patients showing severe symptoms or at highest risk - these disproportionately arise from the infectious group, and so the second set of weights are more skewed toward infectious people, compared to the first set of weights. Thirdly, test accuracy affects the number of positive test results, given any tested subpopulation.

Test accuracy is typically measured by sensitivity and specificity. On the upper path, the positivity ratio of the infectious-tested group is just the sensitivity, which we assume to be 90 percent. On the lower path, the positivity ratio for the susceptibles-tested is the rate of false positives (as this group is not infected), which is 1 minus the specificity, which we assume to be 5 percent. (For the sake of this post, we assume that the performance of the diagnostic tests has not changed during the pandemic and thus the group-level positivity ratios are fixed. This simplifies the following examples.)

 

What's special about 5% positivity ratio?

The media keep reminding us that the CDC wants the positivity ratio to fall below 5%. What is the significance of the 5-percent threshold? I can explain using the flow chart.

We hope for the epidemic to dissipate. When this happens, the infectious compartment dwindles to zero while the susceptibles stabilize. This means there will be no more flow through the upper path. All of the positive test results will come from the lower path (and are false positives).

The overall positivity ratio is a weighted average of the group-level positivity levels, the weights being the proportions of those tested who are infectious or susceptible. In this extreme case, the weight is entirely skewed towards the susceptible-tested, for which the positivity ratio is 5 percent. Thus, the overall positivity ratio will be 5 percent.

In other words, when the positivity ratio dips to 5 percent or below, new infections should be dying out.

 

What's the relationship between rate of cases and rate of infections?

I will now use this setup to investigate Ken's comment. He was reacting to the naive interpretation of case growth as infection growth, popularized by the media. Such a conclusion is invalidated because of targeted testing.

Note that infection growth cannot be directly observed. By contrast, case growth is reported on every dashboard. In Ken's scenario, the number of cases (i.e. positive test results) are growing rapidly.

Start from the dark red box on the right of the flow chart, and work backwards. The jump in positive cases may come from either of the two paths.

First, consider the lower path representing susceptibles. During the course of an epidemic, more people get infected, which reduces the number of susceptibles in the population, which in turn decreases the number of susceptibles testing positive. So if the cases from this lower path have to go up, then it must be that susceptibles are suddenly more likely to get tested.

One such scenario is if politicians expand targeted testing to those with mild or no symptoms. This is Ken's hypothesis.

Such a policy shift also affects the upper path. In fact, infectious people are more likely to experience mild symptoms. Even if the rate of infection is stable, the number of positives tests arising from the upper path grows. If infections are also accelerating, both factors contribute to case growth.

Ken argues that the entire case growth can be attributed to the expanded testing capacity while the media generally interprets the case growth as infection growth, ignoring possible shifts in testing policy.

The trouble is that both factors are at play, and we don't know the relative sizes of their contributions. This confounding is brought to you by targeted testing! (I can't resist: it's free and comes with a hidden cost.) This example demonstrates the aftertaste of targeted testing. It messes up the data so we can't interpret them properly.

It doesn't appear that Ken's scenario is happening in the U.S. as the federal government has continued its misguided targeted testing policy. Some politicians even grumble that we are conducting "too many" tests. (This can only happen if we were conducting too few tests, and thus under-reporting cases early on.)

 

How does targeted testing affect the positivity ratio?

In Ken's scenario, he assumes that the increase in testing accounts for the entire jump in cases. So, let's stipulate that the rate of infections is stable, and look at how the testing policy affects the positivity ratio.

The overall positivity ratio is the weighted average of group-level positivity ratios, so the key is to understand what happens to the second set of weights, which describes the mix of infectious and susceptibles in the tested subpopulation.

Under a randomized protocol, the second set of weights just represents the underlying mix of infectious and susceptibles since both groups have equal probability of getting tested. Under a targeted protocol, the second set of weights is biased toward infectious who are more likely to experience symptoms. Thus, early on, the positivity rate is over-stated. If targeting restriction is relaxed, the bias toward infectious is reduced. Thus, the relative weight of susceptibles-tested is pushed up, and so the positivity rate is driven down.

Since the rate of infections is assumed stable, the shift in positivity rate tells us only about the testing policy. We've just discovered how politics enter the data.

***

Targeted testing introduces bias into the data, which pollutes the link between the rate of cases and the rate of infections. Remember: we use the rate of cases to infer the rate of infections because in the real world, no one knows the rate of infections.

Targeted testing is the aftertaste that you can't seem to get rid of. It polluted the data when first implemented. Then, every subsequent change in policy, and the gradual behavioral adaptation to these changes, add to the confusion. The effect on number of tests is observable but the effect on the types of people taking the test is not.

This tragedy on the national level is now being replicated on individual school campuses. Many schools are doing targeted testing only on students who self-report symptoms. Their dashboards are reporting confirmed cases, the trend of which is being interpreted as the trend of infection. What is the point of data-driven decision-making when the data are known to be bad?

 

 

 

 

 

 

 

 

 

I feel sad for those who had to roll out the methodology change announced by BLS for seasonal adjustment

Imagine running a McDonald's franchise. We need lots and lots of potatoes. Over a year, we order 2400 cases, an average of 200 per month. Each December, more people dine out and our sales jump. We order 300 cases, which is 50 percent more than an average month. (For simplicity, let's say February sales dip to 100 cases which gets us back to the monthly average.)

If we run through 300 cases of potatoes in December, that's a normal year. If we end up consuming 330 cases, that's 10% better than normal. 360 cases, that's a special year, 20% above normal. Conversely, if our diners only devour 250 cases in December, that's bad. It's bad despite the 25% month-on-month growth from 200 in November.

We should not directly compare month-to-month data. We want to break up the data into interpretable parts: current December sales = normal December sales x growth factor = normal monthly sales x seasonal factor for December x growth factor. Normal monthly sales is 200. December sales are typically 50% above normal monthly sales. This gets us to 300 cases. Since we ordered 330 cases this year, the implied growth factor is 1.10. We used 10% more potatoes than the typical December.

This calculation is known as a seasonal adjustment. The seasonally-adjusted December sales is 330/1.50 = 220 cases. This number is 10% above the normal monthly rate of 200.

***

Seasonal adjustment returns to the front pages of the business press last week when the Bureau of Labor Statistics made a consequential change to the methodology used to compute the seasonally-adjusted initial jobless claims.

The following chart shows the strong seasonal pattern of new jobless claims for the last decade. They peak in January every year. (2020 will be added later.)

A jump in January does not mean that the employment market has deteriorated. To know whether the job market was better or worse, we must first take out the seasonal factor.

The following chart shows the seasonal factors used for seasonal adjustment from 2010-2020. The line for 2020 is shown in orange.

The number shown on the vertical axis is our seasonal factor. For the first week of the year, the seasonal factor is 1.60, meaning that typically the U.S. experiences a 60% surge in initial jobless claims in the first week of the year compared to the average week.

The orange line for 2020 hugs the other lines until the most recent week. That's when BLS suddenly switched the methodology for seasonal adjustment. Instead of adjusting the claims number up by roughly 25 perent, the new methodology adjusts up by only about 5 percent.

***

The methodology change is from a multiplicative model to an additive model. Instead of measuring the seasonal factor as a percentage of average claims, the new methodology estimates the seasonal factor as an absolute number of claims.

The chart below shows the number of claims added or substracted due to seasonal adjustments using the multiplicative model. For the first week of the year, the adjustment is 1/1.60 = 0.63, meaning the adjusted claims number is 63% of the unadjusted number. The expunged 37% represent about 123,000 jobless claims, shown below.

The first part of the orange curve up to the present strays away from the gray lines because the same percentage of claims translates to a larger number of claims when the level of claims surged during the pandemic. (Recall the previous chart, which shows that the seasonal adjustments used in 2020 fall right in line with previous years when viewed as a percentage of claims.)

BLS published the scheduled adjustments for the rest of 2020 using the additive model. These are shown as the dashed line below:

The new metholodogy takes the absolute numbers of adjusted claims from past years and use those numbers to add or subtract. The reasoning behind this methodology switch is muddled. By August, the level of initial jobless claims in the U.S. is still five times higher than the typical amount in the last years. Why should the seasonal adjustments not scale with the level of claims?

Given the run-up to the November elections, an obvious explanation is a political decision. Notice that between September and November, the typical seasonal adjustments add to the raw counts (see the first chart to confirm); that's because initial jobless claims are usually lower than average in those months. By switching to the additive model, the size of these up-adjustments is smaller than required by the multiplicative model.

***

Back to our McDonald's restaurant. Imagine that our business is doing very well, and we expanded to occupy the next-door storefront, doubling our sales. We know from the past that December potato orders are typically 50 percent above an average month. That's the multiplicative model. Our new average monthly order has doubled to 400 cases. December continues to be the busiest month of the year, and we should expect orders to expand to 600 cases, using the 1.50 seasonal factor as before.

The BLS is throwing away the multiplicative model. In the additive model, they'd say in December, we order 100 more cases than the historically monthly average (200). So, our estimate for December after expansion is 200+100 = 300 cases. The absurdity of this is clear. Three hundred cases is below the new average order of 400 cases! The problem is that the additive model does not scale the adjustment with the level of purchases.

I feel sad that someone at BLS has to justify the methodology switch.

 

What to make of the vaccine pledge

Investors have schedules: they want to make a return of at least x percent within y years. Science does not follow schedules. That's why vaccine manufacturers got into trouble when the U.S. government keeps seting deadlines for when a coronavirus vaccine will come to market. The anticipated arrival date of late October defies belief. It implies that the U.S. government is poised to approve a vaccine that has been observed for basically one month. (Anyone who joined the Moderna trial at the beginning is just getting the second dose right now.)

This week, we learned that the top pharmas signed a "pledge" to assure the public (link). "We believe this pledge will help ensure public confidence in the rigorous scientific and regulatory process by which COVID-19 vaccines are evaluated and may ultimately be approved."

But those are empty words as they have not committed to any specific action. For example, the information submitted to ClinicalTrials.gov by Moderna, predicted as one of the first to cross the finish line, is as sparse today as it was in late July when I first looked at it (link). The document continues to omit the most crucial pieces of information needed to judge its scientific rigor.

First, the submitted documentation does not explain how the scientists will establish whether someone is infected. It appears that they are relying on self-reported symptoms, rather than diagnostic testing, to determine infection. I say this based on what I read at Moderna's recruiting website (link), noting that this is a marketing website, not a scientific website. If true, that's not what one would describe as rigorous science.

Second, it does not state when the scientists plan on conducting analyses. If they analyze the data after one month, the best they can say is that the vaccine provides immunity for one month (and only for some proportion of people, as is the case with any medicine).

For this pledge to have any value, the pharma CEOs must commit to releasing more details of the clinical trials. What I listed above are not fancy requirements; they are rudimentary and fundamental to the practice of pre-registering clinical trials. It is shocking that those documents omit the information; it is insulting after signing a pledge to use rigorous science.

***

See this previous post to learn about the dangers of not pre-registering the analysis plan, and selecting some short observation period based on initial outcomes.

PS. The Oxford/AstraZeneca trial is catching attention as they have paused due to a severe side effect. I must say that the design document they submitted to ClinicalTrials.gov addresses the two questions listed above. They plan on doing an analysis after 6 months, and they will confirm infections using a PCR diagnostic test (although it sounds like they will count asymptomatic infections as not infections - the devil is always in the details).

Story-first thinking

There are two types of people when it comes to using data. I call them "data first" and "story first". The natural mode seems to be "story first". Calling this "narrative fallacy" or "confirmation bias" creates the impression that the default mode is "data first", and being "story first" is an aberration.

I believe it's the opposite. Being "data first" requires training, and conscious effort. This viewpoint may be related to Dan Kahneman's classification of System 1 and 2 thinking. Story first is quick, System 1 while data first is slow, System 2.

***

The recent reporting of first-time unemployment claims data by Marketwatch captures "story first" thinking. The reporter's story is the federal unemployment compensation (the $600 payout) hurts the economy.

On August 13, Marketwatch published an article with the following graphic:

The headline is the story: "when you pay people more to sit at home... they sit at home." What is the evidence (data)? The reporter said "a big drop in people seeking or receiving benefits in the past two weeks after the end of a $600 federal stipend hints the answer might be yes".

So he's pointing to the last two columns, and saying that those two columns are below 1 million while the previous columns exceed 1 million. Simply put, (a) from last week of July to first week of August, federal unemployment payout ended (b) from last week of July to first two weeks of August, initial jobless claims went down, therefore, (a) must be the cause of (b).

In order to sharpen part (a) of the argument, he established a bar of 1 million weekly claims as some kind of magical level separating good from bad. Above 1 million, bad; below 1 million, good. Prior to August, above 1 million; after August, below 1 million.

The reporter proceeds to quote multiple economists all claiming that this conclusion is a slam dunk, "not rocket science", "hear it all the time from business people", etc.

Our System 1 thinking might accept such a logic but given time to consume the data, it's easy to note the fatal flaw in the argument.

The federal benefit lasted from April through end of July (reference). For every single week during this period, the initial jobless claims number has dropped. In fact, the weekly drops were a lot bigger in April than in August. So we have months of counterexamples on the same chart that is used to prove his case.

Besides, the number of initial jobless claims in the second week of August is still five times higher than the level seen in January and February before the pandemic went out of control.

***

Fast forward one week to August 20. The same reporter reported in MarketWatch on initial jobless claims, and updated his chart:

The general trend of this chart mirrors the first chart. The number of initial jobless claims has declined from the peak in early April.

Take a closer look at this chart versus the previous chart. You will notice they are not the same thing. Here are the alterations:

• The first chart shows seasonally adjusted data while the newer chart shows unadjusted data. The footnote even discloses that the seasonally adjusted data are higher (i.e. worse) than the unadjusted data
• The newer chart added a second category of jobless claims (federal)

Despite the data changing, the story hasn't changed.

In the first paragraph, he writes "Initial weekly jobless benefit claims rose in mid-August and topped 1 million again, potentially pointing to an increase in layoffs after a summer surge in the coronavirus epidemic or perhaps to more people applying for benefits after President Trump temporarily added $300 in extra federal payouts."

So, the story remains the same. The previous week, removing the $600 caused the drop in jobless claims. This week, the "addition" of $300 caused the increase in jobless claims.

The President did not "add extra federal payouts". The $300 payout will be half the previous amount. Further, almost no one received a $300 check during the third week of August.

Once again, the contradictory evidence is found on the same graphic. Back in April, when the $600 checks started getting paid, the new jobless claims number was decreasing, not rising. 

Now, let's talk about the 1 million claims threshold that appeared in the first chart. That 1 mllion level was established for the state-level new jobless claims, which is the lower part of the columns in the newer chart. If you look at just the lower parts, all three numbers for August sit below 1 million. So nothing "topped 1 million again." Magically, the 1 million threshold is now applied to the state-plus-federal new jobless claims data. Why? Because it fits the story.

Except... all three numbers for August sit above 1 million.

***

The economists cited in the August 20 article no longer think the link between federal payout and initial jobless claims is a slam dunk. The quotes are a mixed bag.

But the story isn't wrong. What is wrong? It's the data, stupid. "MarketWatch is reporting select jobless claims data using unadjusted figures to give a clearer picture of unemployment. The seasonally adjusted estimates typically expected by Wall Street have become distorted by the pandemic and appear to overstate new claims at times."

MarketWatch presents zero evidence that the seasonal adjustment is incorrect, except their belief that any adjustment that increases the unadjusted figures is wrong.

According to the journalist, "the increase in new claims wasn't as bad as it seemed... based on actual or unadjusted figures. They rose a much smaller 52,776 to 891,510 and remained below 1 million for the third straight week." [As noted, he uses 1 million regardless of whether the data include federal claims.]

This argument is entirely backwards. If the unadjusted change is smaller than the adjusted change, that means the current situation is worse than normal. Hard to argue with that. 

The point of seasonal adjustment is to establish the expected weekly fluctuations in the data during normal times. (I wrote a tutorial on seasonal adjustment years ago, and also in Chapter 6 of Numbersense (link), on employment statistics.)

***

What do we think should happen the following week? Since more states will be paying the "extra $300", the data better behave! We want to see the initial jobless claims at the state level (unadjusted) to rise again since such payout disincentivizes work.

Drum roll.

On August 27, MarketWatch pushed the graphic to the bottom of the page.

The first sentence of the report reads: "New applications for jobless benefits fell again in late August to just above 1 million and resumed a downward trend, perhaps signaling the resumption of a gradual if painfully slow recovery in the U.S. labor market."

The theory says when more people are receiving the $300 checks, they should be staying home pushing up the jobless claims. (One problem all along is the focus on initial jobless claims rather than total jobless claims or continuing jobless claims.) Don't worry: the journalist is holding out hope for next week's data. He said: "most states still aren't offering the money and the extra cash is just starting to be sent out." The week before, he suggested that the $300 checks caused the rise in claims.

The cited statistic in the first paragraph is the seasonally-adjusted, state-level number (the lower sections of those columns) so the reporter reverted to the metric from two weeks before. He goes on to explain that the data problem is limited to that one week which broke the trend. "New claims rose sharply last week.... but most economists suspect problems with the seasonal adjustments or other temporary factors."

***

Story-first thinking frequently leads to cherry-picking specific conforming data, ignoring contradictory data, explaining away or pouring doubt on non-conforming data, branching the story to accommodate non-conformng data, and creating episodes that work individually but do not cohere.

***

There is another jobless claims report out today but I'll save this for a different post because the Bureau of Labor Statistics suddenly changed the methodology used for seasonal adjustment, which at first glance, will make every single seasonally-adjusted number between now and November look better.