Big Data, Plainly Spoken (aka Numbers Rule Your World)

Take 2 on the Johnson & Johnson vaccine trial results

In the previous post, I characterized the vaccine efficacy (VE) of Johnson & Johnson's vaccine (J&J) as 67%, which is the headline number in the FDA report.

The VE Constellation

If J&J had offered a single estimate of vaccine efficacy, we'd have been shocked.

They centered our attention on 67%, which is described as the VE for "centrally-confirmed moderate to severe/critical Covid-19 cases" with a case-counting window starting 14 days after the first shot (1D+14).

The following list is a partial run-down of other VE estimates that also appeared in the report, and how they relate to the headline number.

If the case-counting window is shifted forward to 1D+28, the VE becomes 66% (lower by mere decimals). This estimate throws out another two weeks of cases, and was added to the clinical trial protocol midway through the trial. I can't think of a reason to justify this metric, other than to compare it directly to Pfizer's headline number (2D+7 = 1D+28).

If that's the case, we also need J&J's VE starting the count at 1D + 35 to compare to Moderna's headline VE at 2D+14. This is not directly cited in the report but it's still 67%.

The VE is 55% if case counting starts the day after the first shot. This metric is not referenced in the text of the report but can be obtained from the cumulative incidence curve I featured in the last post.

The numbers above aren't quite the same as Pfizer's or Moderna's because J&J removes "mild cases." However, J&J reported only 4 "mild cases with onset >= 14 days post-vaccination". This makes the definition of "mild" pointless (unless there are many mild cases prior to Day 14, which we do not know). The VE including mild cases is still 67%.

The headline VE also does not contain all known cases by the time of data cutoff. That's because J&J requires "central confirmation" of positive cases. About 30% of cases have not been confirmed as of the FDA meeting. These cases were excluded from the analysis - which has the same effect as assuming they are false positives. The report said that if all the unconfirmed cases came back confirmed, the VE will be 66%, decimals fewer than the headline number but on the other side of 0.5.

J&J employs adjudicators to decide if a Covid-19 case is "severe." Therefore, they report another VE that ignores moderate cases. This secondary endpoint was added midway through the trial in an amendment to the trial protocol. When moderate cases are removed, the VE goes up to 77% at 1D+14, and 85% at 1D+28. The definition of "severity" includes ICU admission and deaths but neither is necessary, and the adjudicators have the final say. We do not know how many cases they adjudicated. It is not clear how clinicians can utilize such a finding unless "severity" is predictive of hospitalization, intubation, death, etc.

In describing the above result, the report says "Efficacy against severe disease appears to be greater when cases that occurred before 28 days are excluded." Mathematically, the severe cases on the vaccine arm are more front-loaded, and if the earlier cases were excluded, the VE rises. This exposes the very odd enterprise of reducing case counts through exclusions, which almost always leads to increased VE. This increased VE has a more restrictive interpretation (only severe cases, only cases occurring after 28 days, etc.) so nothing practical is gained. The vaccine is less effective, not ineffective, in those excluded areas.

Next, there is a VE for hospitalizations, which was 82% at 1D+14 and 100% at 1D+28. We are now getting into the territory of rare events, and the error bars on those numbers span 80%. Not surprisingly, this analysis is post-hoc, not originally specified in the trial protocol. There are several inscrutable sentences about what counts as a "hospitalization" on p. 33 of the FDA report, and if interested, I'd direct you there (link to PDF).

There is one other VE estimate that should interest my American readers. The J&J trial has participants from the U.S. (~50%), Brazil (~20%), South Africa (~10%) and other South America. For U.S. participants, the VE was 74% at 1D+14, and 72% at 1D+28. These estimates included the unconfirmed cases, which means it is only comparable to the number that includes those (66%). The size of the trial is not large enough to conclude that VE is higher for Americans (the confidence intervals completely overlap).

Double Zero

Recently, the media started describing the primary purpose of vaccines as preventing deaths, even though none of the clinical trials have been sized to measure the death rate with the required precision. In several trials, including J&J, there have been zero deaths attributed to Covid-19 in the vaccine arm, and people then erroneously concluded that the vaccine provides 100% protection against death.

Most lottery players have never won a lottery. However, all lottery players do not conclude that their chance of winning is 0%.

J&J reported 7 deaths on the placebo arm that have been adjudicated as Covid-19 related. All seven occurred in South Africa. So, in the U.S. section of the trial, the verdict thus far is 0 vs 0. If you're an optimist, that's 100% protection. If you're a fatalist, that's 0%. If you're a statistician, you'd wait for more data.

Double Testing

The central confirmation of positive cases is, as far as I know, unique to the J&J trial. This procedure is effectively a guard against false positives but since negative tests are not checked, the double testing always reduces the number of detected cases. The FDA report discloses that typically about 90% of the positive results are confirmed by the central lab.

The central lab uses the Abbott Alinity RT-PCR test, the result of which over-rides local test results. It appears that this type of tests has a false-negative rate of at least 20% (link). The other trials do not re-confirm positives.

Let's assume 10% of all positive test results are thrown out by the central lab. The report tells us nothing more. It is not at all clear that the confirmation rate is even across all cases. Is the probability of confirmation the same between placebo and vaccine cases? Is the rate of confirmation the same regardless of the date of onset of cases? Any such imbalance affects how we interpret the data.

Double Blind

The J&J trial is double-blind while Pfizer/Biontech, Moderna and Astrazeneca-Oxford were all single-blind. This rules out a potential source of bias, and raises the question of why the other pharmas did not conduct double-blind trials.

Blinding reduces the chance that participants know their vaccination status and thus behave differently. Despite being double-blind, there are hints that some people in the J&J trial may have inferred whether they got the vaccine or the placebo. On Table 4, we learned that there was double the number of people who withdrew from the study on the placebo arm (72) than the vaccine arm (32) - this difference is highly significant, even though the rate of dropout is small.

Double Jeopardy

As with all trials, J&J tests each participant on the day of the first shot before innoculating him or her, and throws out anyone who tests positive. Because of this reason, J&J removed about 4,000 people, which seems quite high in a trial with 40,000 enrollees.

While these participants are no longer part of the trial, they presumably took the shot before their test results were available, and were thus followed up. Surprisingly, the vaccine appeared to be ineffective in this subgroup at both 1D+14 (3 vs 4) and 1D+28 (1 vs 2), and also ineffective against "severe" disease (1 vs 0).

Double Alibi

In a side analysis about "asymptomatic cases," J&J disclosed 3 cases that were originally deemed asymptomatic but later re-classified as symptomatic. The investigators said they analyzed symptomatic cases first since it's the primary endpoint, at which time those cases were deemed asymptomatic and thus excluded. When it came time to look at asymptomatic cases, the cases have been re-classified, and thus were excluded from the secondary analysis. In other words, those cases fell through the cracks, neither here nor there.

Perhaps only statisticians find this little episode amusing. We're talking about a few cases. This shows the high stakes in the vaccine competition among the pharmas; every case matters.

 

How to rise above the cacophony of sometimes phony data analyses

Data analyses are flooding the airwaves. They paint a confusing, contradictory picture of all aspects of the pandemic. Sadly, most of the work are hastily put together, and low quality. Here are some things to keep in mind when you look at these studies.

What happened in an experiment will not happen again

If a vaccine trial returns a 90% vaccine efficacy, we can say for sure the vaccine’s efficacy is precisely 90% for the 30,000 or so participants in that specific vaccine trial. We are not empowered to conclude that the VE for billions of people who will eventually get the shots will be 90% - no matter how many epidemiologists repeat this falsehood on TV.

That’s because the trial involved a small sample of people, and there is a margin of error around the 90% number. If the range estimate is 70% to 98%, then we say “with 95% confidence” that the VE for the general population is above 70%. That’s still an impressive number, and it has the advantage of respecting scientific principles.

Imagine repeating the same clinical trial with different choices of 30,000 participants repeatedly. The margin of error says 95% of the series of VE estimates from these trials will fall inside the 70% to 98% range. The chance of any of these trials repeating the exact 90% VE is practically zero.

Credit: Jeremy Schultz/Flickr

 

The randomized clinical trial (RCT) is the gold standard for establishing cause--effect. The lack of randomization in “real-world” studies opens a can of worms.

In a vaccine trial, we compare the case rates of those who are vaccinated to those who aren’t, which are the primary ingredients of the vaccine efficacy formula. We are currently being served weekly new studies that also compare vaccinated people to unvaccinated people. The media are telling us that these new studies are better because (a) they are more recent (b) they have much larger sample sizes and (c) they constitute “real-world” evidence. A popular theme is that these studies fill in the gaps left by the vaccine trials.

The noise you’re hearing are the chuckles from incredulous statisticians. Since RCTs are regarded as the gold standard for causal inference, no “real world” evidence should be modifying, and certainly not correcting, findings from a scientific experiment. Doing so is like hiring a C student to tutor an A student, and siding with the C student when they disagree.

There is one critically important property missing from all real-world studies: the randomization of treatment. (See, however, so-called natural experiments.) In an RCT, a coin flip determines who gets the vaccine but in the real world, who gets the shots is anything but random. Most countries have priority lists and specific types of people are getting the vaccines earlier than others. In any “real world” study, the unvaccinated group is different from the vaccinated group, not just by vaccination status but also by many other factors, both known and unknown. Any of those other factors could contribute – majorly or minorly – to the observed difference between the two groups. Randomizing treatment ensures that on average, these other factors will not bias the finding in an RCT, a condition that does not exist in real-world studies.

A partial solution is to define a better control group. We’d take a subset of the unvaccinated people who look like those who are vaccinated. This is an effort to manufacture artificially the randomization condition. This matching process is highly subjective, and we can only match people using measurable and influential factors. There is no law that dictates that every important factor is measurable. As an illustrative example, perhaps people with higher-speed Internet connections are more likely to land vaccination slots. Since medical researchers do not have individual data on people’s Internet speeds, we cannot control for this effect. Even the most obvious adjustments have problems. For example, young people can be excluded because they don’t qualify for vaccination yet but young people who are front-line workers are getting shots.

Another common issue is incurable imbalance: if all care home residents have been vaccinated, we won't find care home residents in the unvaccinated group. The biggest problem with matching studies is lack of transparency. The typical disclosure is vague and confusing – I can’t even figure out what they did. None of these studies publish their data or code.

So, a real-world study that corrects for selection bias is better than one that doesn’t but in no case is an observational study superior to an RCT. Larger sample sizes produce more precise estimates (with lower margins of error) but they do not auto-correct biases. More precise estimates derived from biased data pose even greater risk precisely because they feel more solid when the biases are ignored.

The randomized clinical trial is a victim of abuse.

In reviewing the torrent of studies that have come out of the vaccine trials, I can’t help but notice that analysts are mercilessly abusing the RCT framework. I’m going to echo Andrew Gelman’s criticism of the standards of research in psychology. We’re concerned about the high probability of false-positive findings.

The VE numbers from Pfizer, Moderna, Astrazeneca, etc. are not comparable to each other. That’s because each team measures case rates differently. Pfizer drops all cases prior to 7 days after the second shot; Moderna and Astrazeneca drop cases prior to 14 days after the second shot. Then, you have people reanalyzing the data, who use their own case-counting windows. The U.K. government published a post-hoc analysis of the Pfizer data, counting only cases between 15 and 28 days after the first dose; this is echoed in a Canadian re-analysis of the same data, in which they set the window of 14 days to 20 days. A more recent “real-world” study out of Scotland counted from 28 to 34 days after the first dose. This is no small matter as VE is an improvement metric relative to a baseline, and we can’t keep our heads straight which the baseline applies.

Many scientists justified the choice of when they count cases, on record no less, by pointing to the cumulative case curve, arguing for counting when the vaccine’s curve starts flattening because that’s exactly how long it takes for the vaccine to do its magic. This type of post-hoc thinking is what teachers warn students not to do. These clinical trials were sized to measure the endpoint, and not sized to identify a turning point on a case curve. Imagine repeating this trial many times over. Assuming that the VE after 90 days rises to roughly 90 percent in each trial, it is likely that the turning point on the curve is not precisely at day 14 (or whenever it was in the first trial).

Analysts appear to have looked at their individual datasets, identified the period in which the vaccine showed the best performance against the placebo, defined their VE metric to hone in on that time window, and justified the definition with some sciency blather. This is a recipe for over-estimating the true performance.

The proof is the absence of new studies that go through similar post-hoc thinking to adjust down the findings of the respective RCT. In fact, the re-analyses of the Pfizer data, which moved the estimated VE of the first dose from around 50% to 90% is a nice example of a C student correcting an A student.

To prevent the danger of post-hoc theorizing, scientists agree to pre-register their studies so they have to define how they would count cases ahead of time. Unfortunately, those protocols are written loosely to allow a wide range of options, such as 7 days, 14 days, 21 days, etc. as possible evaluation metrics. This maybe exposes how little investigators know about the behavior of their inventions.

If only the start of the case-counting window were the only “researchers’ degree of freedom”. The researchers also follow their noses about (a) the duration of the case-counting window (b) whether a positive test is required to confirm cases (c) which test is allowed for confirmation and the parameters for running the test (d) the maximum length of time between reporting symptoms and testing positive (e) what symptoms are included and excluded as qualifying (f) how many symptoms are qualifying. As I showed in this post, VE drops fast with just a few more cases on the vaccine arm so any of these decisions matters.

The RCT framework does not protect us against these potential abuses. RCTs are the gold standard, but they aren’t fool-proof.

Story time after RCT

The potential problems outlined in the previous section relate mostly to trial design decisions. The vaccine RCTs are also being suffocated by post-hoc hacking. I already devoted an entire post to this phenomenon of “story time”. Not all results obtained from analyzing data from RCTs have the crucial randomization property. That’s because analysts can destroy the property by cherry-picking the data. We’re lulled into thinking these lesser findings have the same status as a proper RCT finding.

The most infamous example of such analysis is the 90% VE attributed to the so-called low-dose, standard-dose subgroup in the Astrazeneca-Oxford trial (link). This VE is indeed based on comparing case rates of a vaccine arm to that of a placebo arm. In a true RCT, those two arms are statistically the same except for what was placed in the two shots. Nevertheless, the low-dose subgroup happened as an accident, and it subsequently emerged that this subgroup contained only younger participants, have earlier enrollment dates, have longer intervals between doses, etc. And yet, on the front page of the Lancet paper summarizing this trial, the investigators stated “In participants who received two standard doses, vaccine efficacy was 62.1%... and in participants who received a low dose followed by a standard dose, efficacy was 90.0%...” By this point, there has been no disclosure about the “accident,” and anyone reading this sentence is likely to assume that there were two randomized dosage schedules used in this trial.

Likewise, much ink has been spilled on dose intervals. Many analysts have compared subgroups that took their second shots later and earlier. Even though the underlying data were collected in an RCT, dose intervals were not randomized. And yet, these results were published and publicized using methods for analyzing RCTs and not methods for analyzing observational studies. In some cases, the researchers even disclosed that these subgroups differed on numerous important factors, and still they plodded on.

**

In this era, even results coming out of RCTs must be scrutinized. Early analyses, post-hoc analyses, deep-dive analyses, and side analyses typically discard the crucial randomization property, and should be treated as observational studies.

Of course, results from observational ("real-world") studies should be scrutinized even more. In a future post, I’ll outline how to review analyses of observational data.

It's Tversky-Kahneman time: the law of small numbers

Amos Tversky and Daniel Kahneman coined the phrase "the law of small numbers" (1971) to describe the fallacy of "asserting that the law of large numbers applies to small numbers as well." Statistical thinking is based on the belief that a random sample is sufficiently representative of the underlying population, which is justified only when the sample size is large enough.

***

In recent weeks, some researchers and their media allies have touted the ability of vaccines to combat "severe disease". This is based on statistics from the vaccine trials for which they have "small numbers" indeed- literally, fewer than a handful of severe cases in the vaccinated arms in each trial.

None of those trials were sized for measuring severe disease. To measure rare events requires huge samples, for the simple fact that if the number of participants is not high enough, the most likely scenario is zero severe cases. When you've observed no events, it's very hard to give a precise estimate of the probability of the event occurring.

We already have evidence that the claim of 100% protection against severe disease is shaky. Of the trial results that I have studied in detail (Pfizer, Moderna and Astrazeneca), Pfizer's boasted the largest sample size of 42,000 participants across treatment and vaccine arms. In the Pfizer trial, the vaccine efficacy when counting only severe cases was 75% (1 case among vaccinated versus 4 in the placebo group). On the surface, this is already worse than what Astrazeneca is touting (100%).

What is the range estimate around that 75% in the Pfizer trial? This is not for the faint of heart, so only keep reading if you feel strong.

With 42,000 participants, we have enough data to arrive at a precision of... -153% to 100%. You read that correctly, the data are consistent with any VE between negative 153% to 100%. Simply put, the estimate of 75% is useless, meaningless, junk science.

Astrazeneca? Their analysis is based on 11,000 participants - pooled across two separate trials. This is one quarter of the sample size of the Pfizer trial. A back-of-the-envelope calculation shows that if eventually, we observed 1 vs 4 cases in the Astrazeneca-Oxford trial (same as for Pfizer), the width of the range estimate would double Pfizer's because of the smaller sample size. If the Pfizer number is junk science, then what is the right word for the Astrazeneca number?

***

Now that the vaccine journey has moved from clinical trials to roll-out, the nature of the data has shifted from carefully collected experimental data to conveniently scavengered observational data, and as a consequence, the media have been flooded with bad takes. I'll have a few more posts on this problem soon.

 

A Look Back at 2020

There is no bigger news in 2020 than the pandemic. This once-in-a-lifetime (hopefully) event has shadowed much of my blog writing last year. I'm glad you've stuck around, and if you've been reading from the start, you'd have been ahead of the curve on all the key issues. Data have been central to navigating this new world, however messy, sparse, and incomplete they are. I'm confident you've picked up more about statistics on these pages than in an entire statistics course.

In this post, I've selected some of the posts that have aged particularly well. If you missed these before, be sure to read them now!

***

In mid March, as New York was poised to enter a lockdown, I predicted that this lockdown would have a different ending than those in China or South Korea.

The second type of lockdown - soon to be exemplified by the U.S. - is based on severely limited data. It is based on the assumption that everyone has been infected. The necessity of this kitchen-sink assumption is driven by the lack of knowledge. It is a position of weakness. The duration of the lockdown will be longer, the spread of infections will be wider, and the long-term costs will be higher. It is not the case that everyone has the coronavirus but we have to assume it because we are not testing nearly enough people.

Nine months later, the U.S. has not tamed the virus while the economy has suffered.
(1) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/two-types-of-lockdowns-why-us-must-rethink-its-testing-policy.html

In March and April, when news outlets were one-upping each other with variations of log-scaled charts showing the "exponential" rise in Covid-19 cases, I pointed out that the growth curves were sub-exponential, which also means the log scale is inappropriate. Today, log-scaled charts have disappeared because they have been proven useless.

(2) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/note-about-fitting-and-visualizing-exponential-models.html
(3) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/new-video-validating-data-science-models-a-case-study-with-covid-19-data.html
(4) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/another-note-on-fitting-exponential-models.html
(5) https://junkcharts.typepad.com/junk_charts/2020/06/presented-without-comment.html

In May, I was already drawing the connection between advocating antibody testing and pushing herd immunity:

In places that have fallen down on rolling out diagnostic testing (the U.S. and the U.K., for example), the officials have started talking up “antibody testing”... Antibody testing is a rear view mirror. If someone is found to have antibodies, it’s because the person has recovered from the disease. It follows that the person was infected, and so the person could have died. Antibody testing without diagnostic testing means death over life.

In the same post, I made the following alarming calculation:

With a 0.1% fatality rate, conservatively set at that of influenza, the U.S. would suffer over 200,000 deaths to get there. The path to herd immunity is paved with body bags.

The current U.S. death toll is over 300,000 and counting.
(6) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/the-shift-to-antibody-testing-is-a-choice-of-death-over-life.html

Nevertheless, several countries (U.S., U.K., and Sweden, being the most prominent) continued to deny advancing herd-immunity policies, while most observers recognized them as such. In June, I pointed out that one cannot both believe the coronavirus is the common flu and support herd immunity through natural infections because influenza is seasonal, signifying that natural infection does not confer immunity.
(7) https://junkcharts.typepad.com/numbersruleyourworld/2020/06/coronavirus-has-not-gone-away-neither-have-these-covid-fallacies.html

In October, I objected to herd immunity as a public health objective as it rewards the wrong behavior. I warned against bystander behavior because "you have tied your health to the actions of people who hold opposite views as you." Herd immunity is mathematical immunity, not biological immunity.
(8) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/i-dont-like-the-term-herd-immunity.html

At the end of October, I pronounced the death of the Swedish experiment. I have seen enough. If your soccer (football) team goes down 0-10 in the first 10 minutes of a match, what choice words do you have for the coach who is preaching patience, waiting for the comeback win? By mid-December, Sweden's King admitted that "we have failed."
(9) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/the-swedish-mirage-the-verdict-is-already-written.html

At the time, I assumed 75 percent of the population would need to be vaccinated to attain herd immunity. Many experts placed the number much lower, about 40-60 percent (link). By end of the year, Dr. Fauci is estimating a range of 70-85%.

In one of my very first posts about the coronavirus in early March, I predicted the thorny data quality issues that would bedevil analysts to this day:

We have very limited amounts of data (the known and suspected cases), incomplete data (under-reporting, possible hiding or manipulation), data of dubious quality (self-reporting, hastily assembled tests, unpreparedness), assumptions that later turn out to be false, the potential black-swan fallacy (failure to imagine what hasn't happened before), and non-stationary data (old data might not be wrong but become outdated as the virus evolves).

(10) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/eight-unanswered-questions-about-the-coronavirus.html

In March, I already warned about "the elephant in the room: people are counted as infected only if they are tested", and asked how deaths are confirmed. It would take weeks of regurgitating official counts before the mainstream media acknowledged these shortcomings.
(11) https://junkcharts.typepad.com/numbersruleyourworld/2020/03/eight-unanswered-questions-about-the-coronavirus.html

In April, I explained the benefit of "excess deaths" while also noting why death statistics are subject to their own set of issues. It is here that I said all excess deaths are accelerated deaths. (I will have another new post about this point in early 2021.)
(12) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/excess-deaths-why-we-want-the-cake-and-why-we-cant-eat-it-yet.html

In May and then in December, while studying the data on excess deaths in Florida, I created the following graphic showing the sale.

(13) https://junkcharts.typepad.com/junk_charts/2020/05/how-covid-19-deaths-sneaked-into-floridas-statistics.html
(14) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/its-time-to-visit-florida-again-graphically.html

In May, when Dr. Birx the politician complained that CDC data was terrible, I responded thus:

The members of the Task Force can immediately help make the data better if they cared about the quality of the data. When they aren't pushing diagnostic testing, while trashing the data, their real perceived enemy is not bad data but data.

(15) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/their-real-enemy-is-not-bad-data-but-data-period.html

By June, the media have gone through the five stages of grief:

In the short history of the Covid pandemic, people started with case statistics. Then, they claimed that death statistics are less manipulated than case statistics, when they learned about testing. Then, they claimed that testing statistics are less manipulated, until they realized governments determined who got tested. Some governments counted tests shipped, not test results. Then, they claimed hospitalization statistics are less manipulated, until they learned that hospitals sent sick patients to nursing homes.

(16) https://junkcharts.typepad.com/numbersruleyourworld/2020/06/coronavirus-has-not-gone-away-neither-have-these-covid-fallacies.html

The antibody testing lobby has a particular disregard for data quality. New York's Governor Cuomo, whom I heard won an Emmy for his press conferences, claimed for weeks that 20 percent of New Yorkers had already been infected by April, citing "randomized" antibody testing at supermarkets (during lockdown). To this day, the only "datum" provided by the state's health department which conducted these tests is this FAQ. (Click with no expectation!)

Around the same time, an Oxford research team made headlines, claiming 40 percent of the U.K. population were already infected based on antibody testing. This finding was laughable at the time, and embarrassing given the latest surge in the U.K. In April, I wrote a series of six posts about the Oxford study that explains how scientists build models based on sparse data to make such predictions.

(17) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-1.html
(18) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-2.html
(19) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-3.html
(20) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-4.html
(21) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-5.html
(22) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/a-primer-to-understanding-statistical-models-such-as-the-oxford-coronavirus-study-6.html

The lead investigator of the Oxford study turns out to be a co-author of the Great Barrington Declaration, which pushes the amoral and dubious theory of herd immunity through natural infections.

Another author of that Declaration is a Stanford professor who is infamous for his own role in the debunked study of antibody testing in Santa Clara, California.
(23) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/the-percent-with-antibodies-could-be-3-percent-or-it-could-be-zero-thats-what-the-stanford-study-rea.html

The first draft of the Stanford study neglected the enormous false-positive problem associated with testing rare events, a topic which I covered in depth in Chapter 4 of Numbers Rule Your World (link).
(24) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/to-justify-zero-false-positive-assumption-stanford-study-needs-a-reference-sample-10-times-bigger.html

The corrected manuscript reveals statistical issues with estimating test accuracy when developing new tests, and failed to quiet the skeptics.
(25) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/the-fda-looks-past-these-statistical-issues-but-maybe-you-shoudnt.html

Most importantly, the antibody testing lobby lost credibility in the face of real-world evidence. They predicted a high level of immunity from natural infections, which would prevent a second wave. Everywhere they made that prediction - Sweden, U.K., California, reality bit back, hard.

Meanwhile, mainstream epidemiology has triumphed. Certain colleges re-opened in the fall, implementing strict test, trace and isolate protocols that were swatted aside by the U.S. government. I wrote about the successes at Cornell and Georgia Tech.
(26) https://junkcharts.typepad.com/numbersruleyourworld/2020/09/good-news-from-cornell.html
(27) https://junkcharts.typepad.com/numbersruleyourworld/2020/08/nice-explainer-from-georgia-tech-about-their-re-opening-model.html

Epidemiologists also have predicted the rise of deaths following rise of cases. This lag is frequently ignored by journalists who are professionally trained to think only in the present, and not one or two weeks ahead, and by politicians who say black is white. In a series of posts, I showed how to see the time delay between cases and deaths, in Lombardia (Italy), Texas, Illinois and Iowa.
(28) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/how-to-act-like-a-data-scientist-9-quantifying-our-intuition.html
(29) https://junkcharts.typepad.com/numbersruleyourworld/2020/06/how-to-act-like-a-data-scientist-12-say-no-to-the-daily-grind.html
(30) https://junkcharts.typepad.com/numbersruleyourworld/2020/07/its-high-time-to-call-out-those-who-got-it-right.html
(31) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-importance-of-matching-data.html

As the year ends, we received good news from several vaccine trials. In July, I complained that Moderna provided almost no useful information in its submission to ClinicalTrials.gov, the supposed center of open science.
(32) https://junkcharts.typepad.com/numbersruleyourworld/2020/07/were-told-almost-nothing-about-the-moderna-vaccine-trial.html

By September, Moderna was pressured to release detailed protocols. So followed other vaccine developers. Once these protocols were released, I computed that the earliest approval date for vaccines would be mid January 2021. At the time, the media reiterated unrealistic timelines suggesting results by Election Day (early November).
(33) https://junkcharts.typepad.com/numbersruleyourworld/2020/09/notes-on-the-moderna-vaccine-trial-protocol-1.html
(34) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/political-headline-decrying-politics.html

If the pharmas adhered to those timelines - the Pfizer CEO was pretending they could, it implied that the trial participants would have been observed for only one month, which would at best prove vaccine efficacy for one month. Such a possibility was squashed when the FDA "strengthened" guidelines around the observation window.
(35) https://junkcharts.typepad.com/numbersruleyourworld/2020/09/behind-the-proposed-strengthening-of-fda-vaccine-approval-rules.html

This guideline was not as strong as it could be. I proposed that all participants - instead of the median participant - be observed for at least two months. A week later, a group of prominent statisticians sent a letter to the FDA, making this same point.

When reviewing the trial results, I noted that neither Moderna nor Pfizer met that FDA guideline for median observation window. This is one of numerous tradeoffs made to rush the approval process.
(36) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-worst-kept-secret-will-be-revealed-today.html
(37) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-second-worst-kept-secret-of-the-pandemic-year-and-some-decimal-mischief.html

The metric of vaccine efficacy is frequently misinterpreted. I wrote about VP Mike Pence's tweet, claiming that 90% VE means the vaccine prevents infections in 90% of volunteers. Not even close. Most of the trial participants have not yet been exposed to the virus so they would not get infected whether they got the vaccine shot or the placebo.
(38) https://junkcharts.typepad.com/numbersruleyourworld/2020/10/cutting-the-infection-rate-by-half-is-not-the-same-as-protecting-half-the-people-from-infection.html
(39) https://junkcharts.typepad.com/numbersruleyourworld/2020/11/watching-statistical-gravity-in-action-elections-and-vaccines.html

These vaccine trial results demonstrate a key tenet of designing experiments: decisions have consequences.

Neither Pfizer nor Moderna has proven vaccine efficacy for two months. Their results are based on less than half the participants reaching the two-month mark. If the FDA rule had been all participants reaching two months of observation, then yes.
(40) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/unexpected-sightings-and-unknowns-from-the-pfizer-result.html

We cannot conclude that one dose gives partial protection for two months because no one tested single doses. Anyone suggesting otherwise makes the assumption that the second dose has no value, creating a tautology, saying the same thing twice.
(41) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/one-dose-pfizer-is-not-happening-and-heres-why.html

Any analysis of subgroups (race, age, severe cases, etc.) should be ignored. The trials were designed to minimize total sample size so as to achieve warp speed; the casualty of this design is subgroup analysis.
(42) https://junkcharts.typepad.com/numbersruleyourworld/2020/12/the-worst-kept-secret-will-be-revealed-today.html

Throughout this pandemic, the U.S. government, apparently supported by CDC experts, ignores the threat of asymptomatic spread. The policy of triage testing discourages people without symptoms from getting tested. The design of the vaccine trials - with case definitions based on self-reported symptoms - produces no knowledge about asymptomatic transmission. In March, I wrote a column in Wired to call for broad-based testing.
(43) https://www.wired.com/story/the-problem-with-trumps-triage-testing/
(44) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/when-will-we-get-more-and-broader-testing.html

The clinical trials provide higher quality data compared to research studies that use observational data. I outlined the many problems of observational studies that were thrown together in haste, and published as preprints. These include a Yale study of sewage, a Harvard study of parking lot images and search engine traffic, and a Kings College (UK) study of mobile tracking app data.
(45) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/yale-meds-please-meet-yale-stats.html
https://junkcharts.typepad.com/numbersruleyourworld/2020/06/a-glimpse-into-surveillance-data-via-the-harvard-study-of-parking-lots.html
(46) https://www.wired.com/story/beware-the-lofty-promises-of-covid-19-tracker-apps/
(47) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/it-begins-with-the-data.html
(48) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/pre-processing-data-to-avoid-putting-your-foot-in-that-puddle.html

All these examples show that a model that explains the past well does not necessarily predict the future.
(49) https://junkcharts.typepad.com/numbersruleyourworld/2020/05/prediction-versus-explanation-simple-to-state-yet-hard-to-grasp.html

Regarding the Covid-19 symptom tracking app, I explained how the analytical methodology for estimating population prevalence can justify any level of prevalence. That was back in April. We have barely heard a pip about Covid apps since then.
(50) https://junkcharts.typepad.com/numbersruleyourworld/2020/04/the-first-major-study-using-covid-tracking-app-data-stumbles-out-of-the-gate.html

Between blogging, I was a guest on the CSAI podcast with John Reid and Rusen Aktas:
(51) https://anchor.fm/csai-podcast/episodes/CSAI-2--Covid-19-and-Data-Science-ege3si

More recently, on Ryan Ray's podcast, I talked about how I wrote my books:
(52) https://junkcharts.typepad.com/junk_charts/2020/11/podcast-highlights.html

On the sister blog (Junk Charts), I featured several nice data visualizations. SCMP published this beautiful picture of contact tracing efforts in Hong Kong:
(53) https://junkcharts.typepad.com/junk_charts/2020/09/unlocking-the-secrets-of-a-marvellous-data-visualization.html

The New York Times visualized excess deaths in the U.S.:
(54) https://junkcharts.typepad.com/junk_charts/2020/08/deaths-as-percent-neither-of-cases-nor-of-population-deaths-as-percent-of-normal.html

The New York Times traced the exodus of the rich from NYC:
(55) https://junkcharts.typepad.com/junk_charts/2020/06/designs-of-two-variables-map-dot-plot-line-chart-table.html

The Visual Capitalist depicted how we changed our consumption habits:
(56) https://junkcharts.typepad.com/junk_charts/2020/05/consumption-patterns-during-the-pandemic.html

***

I wrote 212 posts across the two blogs in 2020. Since I started, people have read posts more than 4 million times. Thank you for reading, come back often, dig into the archives, and recommend me to your friends and colleagues. See you in 2021!

Kaiser

 

[1-1-2020: Added an index to the links, and corrected a link error.]

Two side issues as we hurry to a coronavirus vaccine

My review of the Moderna trial protocol (parts 1, 2) shows that by and large, the design follows established standards, with only a few places where the researchers have selected less than the gold standard, which is understandable given the urgency of finding a vaccine.

That said, a rushed job with the desired outcome of a vaccine is going to increase the small odds of a false-positive result. This may come about in unconventional ways.

***

Within each trial, as I pointed out in part 2 of the Moderna protocol review, the threshold for statistical significance has been made drastically more rigorous to account for the interim analysis based on a much smaller sample of data. That's the proper thing to do. The design is generated so that the overall probability of a false-positive across all attempted analyses and endpoints is under control.

Of note, the world is running a dozen or more trials of competing vaccines at the same time. Even when each trial controls the chance of a false positive, the overall probability of a false positive coming from any of the slate of trials is not under control.

Think about elevator maintenance. For any single elevator, the manufacturer might promise a breakdown rate of below one percent. Now, across all elevators of all brands in a single town, the chance that at least one elevator breaks down in a period of time is much higher than one percent. That's because each elevator has a one-percent chance of breaking down, and any elevator may break down at any moment in time. The more elevators there are in the town, the higher the chance of at least one breakdown.

The more vaccine trials are started, the higher the chance that a positive result is a false alarm. No individual clinical trial is controlling for this kind of risk; only the FDA can address this.

***

Another rarely addressed issue is statistical validity, that is to say, how representative are the participants of the trial relative to the people who are likely to receive the vaccine once approved.

The vaccine manufacturers are engaged in a complex competition. The first vaccine to get approved has a special status because it becomes the standard of care. Other trials may have to use the first vaccine as the comparator, rather than the placebo. Because of the provision for interim analysis, the sooner a trial completes enrollment of participants, the sooner it can begin analysis, and the approval application. Interim analyses are not triggered by the total enrollment, as in most clinical trials, but by the total number of infections.

Thus, the incentive is to get as many people enrolled as quickly as possible, and to dip into higher-risk areas as much as possible. In the extreme situation, if all enrollment is focused on only a few highest-risk areas, the trial may become the first to accumulate enough case targets. (A possible dent in this strategy is if people's willingness to participate is inversely proportional to how much risk they face.)

While this competition plays out, the trials that have the most biased samples, concentrating on the regions with the highest "yields", will likely win the race for time. This creates a problem for the FDA because the results would score lower on statistical validity, and have to be de-biased. (Such a sample would however provide better data on higher-risk subgroups. The trade-off is that the vaccine is supposed to be for everyone.)

Perhaps the enrollment can be skewed towards severe cases at the start, and then broadened over time to encapsulate the general population. This mimics the likely plan for rolling out the vaccine.

***

I should underline that these are not show-stopping issues. They are interesting statistical side-notes that may seem theoretical when you read about them in a textbook -- until the pandemic happens.

Notes on the Moderna vaccine trial protocol 2

This is Part 2 of my notes about the Moderna vaccine trial. Read Part 1 here.

***

Durability of Protection

As described in the last post, the full analysis will not take place until middle of 2021, so the vaccine developers are hoping that their vaccine would show enough during the first or second interim analysis to accelerate the time line. Early reads increase the chance of false-positive findings, as I outlined in a previous post. To counter this, the study drastically lowers the p-value threshold to reach statistical significance in the interim analyses. (Instead of p = 0.05, the first interim analysis in effect sets p = 0.0002.) This is a proper thing to do.

However, if the first interim analysis results in a vaccine approval, then the participants will have been observed for at most 2.5 months from the second dose. We will not know if the vaccine offers protection beyond 2.5 months.

According to the protocol, Moderna "intends" to continue the study after if it receives interim approval. There will be an ethical debate about whether those who received placebos should be given the vaccine should it win approval. Some may oppose withholding a "proven" treatment from these participants. If they are given the vaccine, we lose the ability to measure protection beyond 2.5 months.

Sample Size

If the first interim analysis is successful, then the vaccine would receive approval based on about 50 total infections. Assuming that the vaccine effectiveness is 50%, then we expect half the number of infections on the vaccine arm relative to the placebo arm. So the vaccine arm should account for one-third, or about 17 infections. That's an awfully small number.

Nevertheless, if approved, the vaccine would have met the more stringent statistical significance threshold so the small sample size is not a serious problem. But the required number at the first interim analysis is small enough as to make many statisticians nervous. Continued tracking beyond the first analysis will be key to make sure the early result is not a mirage.

Defining Trial Outcomes

In the primary analysis, the "endpoint" (outcome) is infection, which requires a PCR test confirmation plus some pre-specified level of symptoms.

Some experts are arguing that they should count severe cases only because we don't really need to stop the mild cases. The chosen endpoint includes both mild and severe cases but we expect that most of the cases will be mild so there is concern that the signal (severe) to noise (mild) ratio is not high. Of course, if only severe cases are counted, the trial would last longer.

The opposite criticism is that they are by and large missing asymptomatic cases because almost all the testing are triggered by symptom reporting. One can argue that we need to stop asymptomatic spread because any viral transmission may eventually produce severe cases.

Moderna is taking the middle ground here.

Unusual Sources of Noise

In running any experiment, we'd love to eliminate as many sources of noise (variability) as possible so that any observed differences can be attributed to the vaccine. Some practical issues for this vaccine trial may produce noise, which affects our interpretation of the results.

The FDA permits accommodations due to the pandemic. This means that in-person visits may be replaced by phone calls. That itself is not a problem if all visits are converted to phone calls. The problem is that swaps occur based on request so they'd end up with some in-person visits and some phone calls. This creates a possibility that the method of the follow-up may be associated with, or even cause, different outcomes.

Besides, the protocol lists a variety of tests used to confirm infection. The nasal swab is preferred but saliva and respiratory samples are also used. These tests have different accuracies and characteristics so the mix of test types may affect the count of confirmed cases. For example, people living in high-risk areas are more likely to submit saliva samples by mail versus nasal swabs from in-person visits.

Uncertainty Around the Case Rate

The design assumes a 0.75% baseline case rate within six months but the researchers acknowledge uncertainty around this number - that's why they end the enrollment based on the number of cases rather than the number of participants. What if this assumption is wrong?

In particular, this base rate seems fairly low. Those who hope for "herd immunity" might be shocked to learn that the virus is expected to infect less than 1 percent of a community over six months. At that sluggish rate, when will we reach 70 percent or so prevalence?

All sample sizing requires guessing at the base rate, and sometimes we can get it wrong. If the base rate is somewhat higher than 0.75%, then the trial will get infections faster than expected, and thus end enrollment earlier than expected. The real case rate among those participants can be, say, 2%.

Here's the rub. If 2 percent had been used as the base rate in the design of the clinical trial, statisticians would have stipulated more than 53 infections to trigger the first interim analysis. This means, if the observed case rate is higher than expected, the first interim analysis suffers from insufficient data. Smaller-than-expected sample size makes it more challenging to beat the significance threshold.

***

The Moderna protocol is reasonable: there are some unconventional elements, but in an emergency, we'd have to accept some accommodations. It's still hard to imagine how any of these trials will get to a strong result by end of October or November.

News about the vaccine trials is coming fast. Just this weekend, the Washington Post reported that the FDA may tighten the vaccine approval requirements a little to pacify critics and build trust. I have another post up discussing these modifications.

 

 

 

 

 

A story about poop, and the scientific method

These headlines flooded the media last week. I'm listing the first five results selected by my search engine.

University of Arizona is screening for COVID-19 via dorms poop (New York Post)

University of Arizona says it prevented COVID-19 outbreak by testing students feces (Daily Mail)

Poop test stop COVID-19 outbreak at University of Arizona (Science)

Poop becomes secret weapon to detect COVID-19 cases in dorms, says University of Arizona (National Post)

Researchers at the University of Arizona say they stopped a coronavirus outbreak before it spread by testing students poop (MSN)

***

The sequence of events that reportedly happened is as follows:

1. As part of the re-opening program at University of Arizona, the school tests wastewater from 20 on-campus dorms for presence of the coronavirus.
   
   
2. Last week, one sample came back positive, and was traced to a specific dorm (Likins Hall).

3. Researchers notified school administrators and convinced them to test every resident of that dorm.
   
   
4. Of the 311 samples collected from residents, researchers found 2 cases, implying a positivity rate of 0.6%.

5. The two students, both asymptomatic, were isolated.

6. Researchers concluded that they prevented an outbreak in that dorm.

This is good news for managing campus re-opening. Testing sewage water should be in the toolkit, used in conjunction with other tools. But establishing the cause-effect relationship is not as simple as the Arizona researchers made it out to be.

This story is also a case study of how it is easy to make a scientific claim but it is hard to prove it.

***

The university tested about 10,000 people, as they checked into campus housing about 10 days ago (other returnees and off-campus students are tested only if they request it). They found 46 cases. That's a positivity rate of 0.5%. On average, if we select 311 people randomly, we should expect 1.4 positive results, so it is normal to find one or two cases if that is the prevalence.

Recall that they discovered two cases in 311 tests (0.6% rate) in Likins Hall. The scientific question is whether that positivity rate is abnormal. If they had tested samples from another dorm - one for which the wastewater testing turned out negative, and found no positive cases, that would strengthen the causal claim. Did wastewater testing accurately detect the presence of virus in a specific building? The answer is yes, conditional on the key assumption that the prevalence of virus in dorms that passed sewage testing is zero.

Since the residents of the other dorms were not tested, we don't really know the answer. That's why it is an assumption of the analysis. There are reasons to believe that the prevalence is low but not zero: (a) the residents started checking in on August 21 and the wastewater testing program was announced on August 25 (link) so it is possible that residents have gotten infected in the interim (b) it's likely that at least a few residents have not yet completed arrival testing, and they may be infectious (c) up to two guests per resident are allowed in the dorms, and testing of off-campus students is voluntary (d) vendors (including movers) can bring virus into the building.

***

Now, consider the tracing of sewage water to Likins Hall. How accurately can this determination be made? This depends on where the sewage water is collected. In a notice to on-campus students, the university made this disclosure: "This [wastewater] testing does not identify individuals or even floors of the dorm or building that might have infections." This statement directly contradicts the published story in which pinpointing the building was crucial in the chain of events.

I also wonder how they deal with any mixing of sewage, or possible environmental contamination.

***

The consultant hired to supervise campus re-opening told a reporter: "if we had missed it, if we had waited until they became symptomatic and they stayed in that dorm for days, or a week, or the whole incubation period, how many other people would have been infected?" This interpretation assumes that the virus found in the sewage water came from the two students who subsequently tested positive, and were still infectious.

Wastewater testing is beneficial only if it detects active infections. So establishing timing is essential. It is possible that the infected sewage came from past infections but not from the two active cases - there is still a lot that is unknown about virus shedding in feces, only a fraction of infected people do it. I'm not saying this is what happened but the coincidence of (a) virus found in sewage testing and (b) cases found in building does not prove that the infected sewage came from those specific cases.

***

There are three types of university responses to Covid-19. The most risk-averse group has moved all instruction online. The second group is attempting a return to the classroom but will require comprehensive testing. The third group re-opens but is looking for shortcuts to avoid having to test everyone. Sewage testing is of interest to this final group of schools.

The Arizona case study provides some hope that testing wastewater may reduce the amount of individual testing necessary to monitor and control infections on campus. Conceptually, this is related to group testing (explained here), replacing the usual nasal swab tests with sewage tests in the first stage.

More work needs to be done to establish the cause--effect. For example, controls should be included in the study, testing a dorm where the sewage came back negative. Another reason for controls is to measure false negatives. A deeper understanding of the "schedule" of virus shedding during the infection period is necessary. Measures to rule out contamination help reduce false positives. As with other tests, the scientists should publish the accuracy characteristics of the wastewater test.

Answers on excess deaths, test accuracy and quick testing

The recent posts have sparked some nice discussion here and elsewhere. This post collects additional comments I've since made.

Excess deaths caused by the lockdown

There was a verbal scuffle when reporter Jonathan Swan interviewed President Trump a few weeks ago about how to measure deaths due to Covid-19. Should we look at deaths per case (mortality rate) or deaths per million? In two recent posts (1 and 2), I discussed the visualization and modeling of "excess deaths," which is a better way of measuring the pandemic's impact than both of those popular metrics.

The starting point of an excess death analysis is neutral and apolitical. Based on historical patterns, we know that X people tend to die in the weeks from March to the present. Instead of X, the actual deaths (by any cause) have spiked in most U.S. states by 50% to 700% of the normal level in the worst weeks of the pandemic.

These "excess deaths" are not explained by any of the known causes. A majority of these deaths can be explained by deaths attributed to Covid-19, a novel cause of death. Most scientists believe that the unexplained excess deaths are probably under-counts of Covid-19 deaths. Should convincing data emerge, other causes of deaths could be accepted.

Some readers asked whether "excess deaths" may be caused by the lockdown rather than the virus. The lockdown does not kill anyone directly. What they mean is the side effects of the lockdown, such as more suicides from depression due to lockdown, or more cancer deaths because the patients were turned away from hospitals.

This is a logical argument but let me explain why this argument seems like a dead-end:

• The biggest challenge is the scale of the gap. A 50-percent to 700-percent gap is not easy to reconcile. Cancer is 20 percent of all U.S. deaths in 2017 (source). Let's say half of those occur between March and July. Then, if 2020 is similar to 2017, we should expect 300,000 deaths during those months. If we want cancer to explain 100,000 excess deaths, we're saying the lockdown caused cancer deaths to rise by 33 percent. That's a huge jump for a well-known disease that requires investigation. And we still have another 100,000 excess deaths to explain.
  
  
• Other than rhetoric and anecdotes, I haven't come across any serious attempt to quantify these lockdown side-effects. If some cancer patients have died because they are turned away from hospitals, then they should start investigating people who died at home from late-stage cancer. How many such deaths can we confirm? It's plausible that such deaths exist but I doubt this will make a dent in explaining the 200,000 excess deaths.
  
  
• Most cancer patients who died at home because they couldn't schedule surgery at the hospital during the pandemic were seriously ill, and would have died eventually of cancer. The side effect of the lockdown is accelerated deaths. In the world of "excess deaths," an accelerated death contributes to one positive excess death now and one negative excess (avoided) death later. After lockdown is called off, we should experience a reversal of these excess deaths. Will we find hundreds and thousands fewer deaths from all non-Covid19 causes in the aftermath? I don't think so.

 

False positives in the NFL

News broke that many recent positive test results on NFL players turned out to be false positives. This has been described as a "testing debacle."

First thought: false positives are the price to pay for being able to trust negative results. Notice that the NFL is not re-testing negative results; the NFL retests positive results. In order to afford such a luxury, the negative results must be rock solid. If the test comes back negative, the person is more or less certain to not be infected. This is how a good coronavirus test should behave. A false-negative mistake is an infectious person who spreads the virus unknowingly. For a diagnostic test to have fewer false negatives, it will have more false positives.

Second thought: testing errors, when interpreted in a broad sense, can be caused by operational mishaps. From what has been disclosed so far, it appears that the NFL contracts a single private testing company to perform all coronavirus testing, and that business operates labs in multiple states. All the false positives came from the New Jersey lab. This might be traced to say a contamination of equipment at the lab, just for argument [PS. That's the official reason given by the lab now.]. If true, the inaccuracy has nothing to do with the chemistry; the error came from lab operations.

All accuracy metrics published by test developers assume zero operational mishaps, 100 percent complicance, no gaming, etc. I brought this important point up in Chapter 4 of Numbers Rule Your World (link) - an entire chapter devoted to the complex issue of testing accuracy.

This is why when discussing group testing in this post, I singled out operational mishaps that could happen because group testing adds complexity.

 

Wishful thinking on treatments

The U.S. President has endorsed plasma therapy as the latest best therapy for Covid-19. The FDA has given it emergency use authorization. We should be clear that this is temporary, pending the results of proper scientific testing, with randomized controlled trials that prove plasma works.

I understand the media love the feel-good stories about people who insist that plasma saved their lives. But these survivors don't know. Because if plasma indeed saved their lives, then had they not been given the blood transfusion, they would have died. But no one knows that since they got the plasma!

It is entirely possible that plasma worked for some specific cases. The point of science is to establish that it would make a difference for enough patients to matter. [PS. The FDA Commissioner has already sort of back-tracked.]

Elsewhere, on college campuses and in professional sports, the testing labs are turning to quick saliva tests to scale up testing. This is of course a good thing. There are several such tests developed (I think I've read about those made by Yale, University of Illinois, Rutgers, etc.). The developers of these tests make many magical claims (see here). The quick test is supposed to be just as accurate as the nasal swab PCR test; much faster than that test; can pick up even lower amounts of virus; can provide a numeric scale of the amount of virus, not just its presence or absence; etc.

I find these claims hard to believe, except the efficency claim. Saliva should contain less materials mixed with unrelated fluids. It's probably less accurate. But the benefit of scale is obvious. Since it's used primarily for "surveillance testing", let's hope it doesn't generate too many false negatives, who become silent carriers.

A testing mess across the pond

Controversy has engulfed the A-Level examinations in the U.K. The A-Levels are university entrance level exams of subjects (math, English literature, history, economics, computer science, etc.). Universities conditionally admit students with minimum grade requirements for relevant subjects. The A-Level results are often the most important admissions criterion, by far. Their closest American cousins are the SAT Subject Tests, although in the U.S., these tests are not widely required, and no college conditions admissions on test scores.

The coronavirus pandemic threw a wench into the exams system. Students could not sit for these exams in 2020 thus the authorities concocted a statistical scoring model to assign grades. The model predicts what grade the student would have scored had s/he sat for the exam.

Roughly speaking, each grade has three components: a baseline score that applies to every student, a school factor that applies to all students of the same school, and an individual factor that applies to each student within the same school. The school factor is based on performance of past cohorts of students in the A-Levels. The individual factor is based on the student's prior performance in other public examinations, and on a teacher evaluation.

As part of this exercise, teachers are asked to grade their own students, which have been described by students as their "predicted grades". On results day, a lot of students and their teachers were disappointed by the assigned grades. They found that the model's predicted grades are lower than their teachers' predicted grades. There was a huge outcry about so-called downgrades. The statistical modeling is attacked as "utterly unfair and unfathomable", hurtful to young minds, inconsistent, disappointing, lacking transparency, etc. (See this BBC Live Feed for reactions.)

***

The term "downgrade" interests me a lot. We have two predictions of an unknowable quantity, how the student would have performed had exams not been cancelled. These predictions sometimes don't match.

From the students' perspective, the conditional offers from universities set targets for A-Level grades. Their teachers told them what grades to expect, i.e. the predicted grades. The official grades they receive are evaluated relative to these teacher assessments. If the official grade is lower, it's a "downgrade" and if it's higher, it's an "upgrade".

I believe the exam administrators are shocked by this reaction. They probably regard the teacher assessment as one input in a complex process that generates official grades. They probably see the model's assigned grade as the "truth". It's the teacher assessment that may be over- or under-estimates.

This situation reminds me of my recent post on how the business community interprets metrics about the economy. They take their expectations - potentially generated by models - as the reference level. Official statistics on unemployment, inflation, etc. are evaluated relative to these expecations, rather than the other way round.

***

The media has by and large sided with students and their teachers. In this post, I provide the other side of the story, the challenges faced by the statistical modelers.

There are limited capacity at U.K. universities, and particularly at the more competitive schools. Admissions officers set A-Level grade requirements, the conditional offers, based on historical distribution of grades, and their historical yield rates. If the exams were not cancelled, the administrators are reasonably confident that the grade distribution in 2020 would have been similar to those of recent years. 

Not surprisingly, the distribution of grade predictions by teachers exhibits grade inflation. The exams authority, Ofqual, published the following chart:

The yellow line represents the proportion of A and A+ grades for the last four years. The orange dot labelled CAGs represents teacher assessments for 2020. Ignoring the 2020 yellow dot for the moment, you see that teachers in aggregate assigned 12.5 percent more A and A+ grades than the 2019 results. As a ratio, this is 50 percent more top grades than the year before!

The so-called downgrading describes the gap between the orange dot (37.7%) and yellow dot (27.6%) in 2020. The teacher assessments were fed to the statistical model which brought the proportion of top grades down to roughly the historical level. In fact, the proportion of top grades issued by the model is 1.4 to 2.4% higher than in recent years.

What this means is the attack on the model is rather misguided. I'm sure there are problems with the model, as with any model, but no model can escape the constraint that the proportion of As and A+s should be about 25 percent. That means all models will issue a lot of "downgrades". They differ by which students are downgraded.

Whoever gets downgraded has a grievance. We haven't heard complaints from those who got upgraded (about 2 percent of grades), and probably won't. Such asymmetric complaints tend to make models worse. Deficiencies are remedied along one dimension but not the other, which introduces a "correction bias" to the model.

I cover this issue in Chapter 2 of Numbers Rule Your World (link) in the context of credit scoring models. When the credit bureaus allow us to check the data, we will challenge anything that might reflect negatively on the credit report, but if the bureau makes a mistake that benefits us, for example, if the report failed to record a prior delinquency event, most of us will not volunteer this information (that will certainly lower our credit score).

***

Another factor that no model can escape is the differential quality of schools. Students of some schools have historically done better at A-Levels and other public exams. It is a bit unfair that a student's grade is partially determined by the quality of the school s/he attends.

In fact, standardized testing solves this problem, except the pandemic wiped that away in 2020. To a large extent, all models assume the future is like the past. Existing inequities and biases get propagated.

 

If you check your pulse daily, one day it will flatline

When reviewing the clinical trial design for the Moderna vaccine, I mentioned the statisical fallacy known as testing to significance.

Every pharma developing a vaccine or drug must prove a statistically significant (and clinically relevant) outcome. The difference between the treatment and placebo arms must be large enough to overcome background noise coming from sampling, measurement errors, etc. Any statistical result is attached a margin of error, typically 5 percent. For a clinical trial, there is a five percent chance that a statistical significant difference is detected even if the vaccine may be useless.

Even when the vaccine is useless, five percent of the time, the clinical trial yields a benefit large enough to be "significant". Because of this, if we keep analyzing the difference between treatment and placebo day after day after day, eventually, there will be a day when the difference meets the significance threshold - even if the treatment is useless.

If we keep checking the differences beyond that day, we will discover that the significance does not last. That's not the problem. The fallacy arises if the trial is declared a success at first sight of significance.

***

I now use a simulation to demonstrate this statistical fallacy.

For the simulation, I generate 10,000 alternate worlds. In each world, I track 5000 people who get the vaccine and 5000 other people who receive the placebo. I then count the cumulative infections over time in each arm. I assume the vaccine is useless, so that infections grow at a normal rate within the community, regardless of whether the participant got the vaccine or the placebo.

The following line chart shows a single "sample path", the growth curves of infections in one of those 10,000 scenarios. (The blue line represents the vaccine group, the green line, placebo. Fewer cases are better.)

The rates of infection are just these counts divided by 5000, so the curves of rates look exactly the same.

As per the trial protocol, I follow both groups for up to 760 days (2 years after the second injection).

In the next chart, I add a third line (in yellow) to display the difference between the placebo and the treatment infections. Positive differences imply fewer infections on the vaccine arm than the placebo (vaccine may be beneficial); negative differences mean the placebo arm has fewer infections (vaccine may be harmful).

The raw differences don't matter as much as their statistical significance. When we say a difference is statistically significant, we are confident that the difference is driven by the vaccine, as opposed to background noise.

I perform significance testing on the infection rates over the course of the trial, at Day 60, 90, 120, ..., up to Day 760. Around Day 90, the gap between the placebo and vaccine arms is large enough to meet the statistical significance threshold.

If I look again on Day 240, the difference is not significant. The gap does not hold - this is in line with the assumption that the vaccine is useless. The fluctuations of the yellow line (differences in cases) are driven by random variability. (Important note: this phenomenon has nothing to do with a vaccine losing effectiveness over time. In my simulation, the vaccine is assumed useless.)

What if I test to significance? This means I end the research after finding the Day-90 difference to be significant, and conclude that the vaccine is beneficial. This is equivalent to erasing the rest of the line the first moment the line hits the significance threshold.

Such a conclusion is spurious as the simulation stipulates that the vaccine is no better than the placebo. Besides, if I continue to track the data, the Day-90 difference does not hold up.

***

Recall that I've created 10,000 alternate worlds.

The next chart shows 20 lines: each line is the difference between placebo and vaccine in one of 20 alternate worlds (selected randomly from the 10,000). I am showing you the background noise. By Day 750, in most scenarios, the observed differences hover around zero but in a few cases, the gaps are around 100 cases in one or the other direction.

The next chart shows what happens if I stop the analysis at first sight of significance. In over 25 percent of these scenarios, a winner is determined before the end of two years even though by assumption, the vaccine and the placebo groups should respond similarly. In fact, in two scenarios (10 percent), the significance threshold is crossed by Day 90.

The basic point is that if I follow each line far enough, eventually I will find a day on which the vaccine shows significant benefit relative to the placebo. If the trial continue indefinitely, those non-significant lines will eventually reach significance!

Here are 50 scenarios in which the vaccine can be declared the winner by Day 120. The thick part of the lines is observed; the thin part of the lines will not see the light of day if I declare mission accomplished by Day 120.

I remind you once more that the simulation assumes the vaccine is useless. Any finding of statistical significance is an error. This includes the lines at the top of the chart in which significance appears to hold through Day 750.

***

Let's move from the simulated fake world to the real world. In any single clinical trial, all we observe is one of these sample paths. We don't have the luxury of comparing the one observed path to the background noise. Over the course of the trial, the path reveals itself; we also cannot predict where it is heading - as demonstrated by the simulation, any path can turn at any moment.

In order to prove the success of a vaccine, it must demonstrate a statistically significant reduction in infections. The crucial question is by what time. The wrong answer is by the time we see a significant difference and before we run out of resources or patience.

This answer is wrong because it essentially guarantees success, bringing along a high likelihood of a spurious result. In the simulation, I force the vaccine to be useless (no signal, all noise). In real life, the vaccine might be slightly useful (weak signal, plus noise). This is even harder because a few of the significant readings may in fact be real.

An antidote to the statistical fallacy is patience. This is in short supply during this pandemic. The political and societal expectations are exactly opposite - stressing "warp speed". The lurking danger is confirmation bias.The pharmas track the trial results over time, and at first sight of significance, they face tremendous pressure to declare mission accomplished.

That's why I feel uncomfortable that the design documentation for the Moderna trial does not specify analysis time points. It gives an "up to" two-year time frame, which everyone knows is not real. They are not saying it would take more than two years to get a scientific result. In fact, they have told the media that we might get an answer within a few months!

***
This situation is not limited to clinical trials. I've encountered this debate throughout my career in marketing analytics. Let me give you some more color on the nature of the debate.

In one of the charts above, I show 50 scenarios in which the vaccine can be declared a winner by Day 120. The following chart displays a different set of 50 scenarios: these paths lead to the placebo reaching significance by Day 120.

If the one observed sample path comes from this chart, then by Day 120, we should stop the trial early, and declare the vaccine a failure. (As before, any such finding is spurious because I set up the simulation to assume no difference between the placebo and vaccine groups.)

No, you don't think the vaccine effort will be aborted. Neither do I.

The decision-makers will pummel the data analysts with endless questions to make sure they did not make a mistake. They can easily come up with pages of reasons why we need more data. Shutting down the study is like closing a losing position of a stock investment; you have to immediately write off the billions invested. It feels better to keep the position in the hope that the stock price or the response curves would reverse course.

Now, the exact reverse happens if the initial analysis shows the vaccine to be winning. The data analysts urge patience, wanting to more data before drawing a conclusion. The decision-makers demand swift action, arguing they can't afford to wait.

In one case, the statisticans are criticized for being too "academic" and cautious; in the mirror scenario, they are blamed for not being "certain" and using imperfect data.

Is there a way out of this? One solution is called "pre-registration". All parties have these discussions before the clinical trial even begins. Lay out the posisble outcomes. Gain consensus on what to do under each outcome. Once the data start appearing, it is virtually impossible for any side to remain neutral.

 

P.S. [8/4/2020] Based on some side conversations I've been having with a few readers, let me clarify a couple of things. I'm not suggesting that Moderna is planning to test to significance. I expect that they have protocol to mitigate this issue. I was expecting to find breadcrumbs in the document submitted to ClinicalTrials.gov but it contains no information about how the data would be analyzed. Experts say the regulators should have received a more detailed protocol but that has not been made public.

Statisticians have developed solutions to deal with this issue. Pre-specifying stages of analysis is important, and the threshold for statistical significance is adjusted to account for each early view of the data.

Edited sentence per convesation with AR in the comments.