Big Data, Plainly Spoken (aka Numbers Rule Your World)

Bad covid data: feature or bug in Texas?

Let me show you how bad the state of Covid-19 data is in the U.S.

Recall my post from two weeks ago about Covid-19 deaths in Texas starting to catch up with the earlier rise in reported cases. When cases started to spike in Texas, talk turned to the declining death rate, which is misleading since we should have been looking at cohort-adjusted death rates from the start. The media conspired to spread this misinformation, ignoring epidemiologists - when given a choice between reporting a misleading statistic today and reporting a more valid number in two weeks, the media cannot help but quench the immediate thirst.

In putting together the prior analysis, I already noticed the sad state of the data. On that day (July 27), Texas just changed how it reports deaths to rely on cause of death on death certificates. This created a single-day death toll of 675, well above the daily average, which presumably corrected for under-counting from previous days. With this correction, the state claimed that it will henceforth provide "more accurate" death counts.

What the state official did not explain is that the change in reporting also ensures a reporting delay for all deaths of at least one week. Let's think through this in the context of rising case rates and "declining" death rates. I make the nonsensical assumption that if patients die from Covid-19, they die immediately, on the day they are reported as new cases. Under the new reporting procedure, these cases will be reported today but the deaths will be reported when death certificates have been issued and can be inspected, which is at least 7-10 days in Texas. Thus, this group of patients (who already died) inflates the number of cases today while suppressing today's death rate since their deaths are not yet reported. This time lag is due to reporting procedures, and has a similar effect as the time lag between infection and death.

The data source, Covid Tracking Project, rated Texas "A" for "data quality" so presumably the state of the data elsewhere is worse.

***

I intended to update the analysis for this post last week but my effort was met with more data mischief. On August 2 (Sunday), Texas did not report any data, blaming a "systems upgrade" and promised that "data for Sunday will be posted with Monday’s data update." So I waited another week before publishing this update.

Just a reminder, two weeks ago, I presented the following chart:

The thin gray line is the growth curve for reported cases in Texas, smoothed using a seven-day window, from mid May through late July. The thick gray line is this growth curve shifted forward in time by three weeks. The red line is the growth curve of deaths in Texas, smoothed using a seven-day window. Except for the single-day anomaly (July 27), it appears that deaths roughly lagged cases by three weeks.

The next chart extends the time-line to August 1, the day before the data blackout:

This chart is disturbing. The growth of deaths in Texas has started to markedly outpace the growth of cases (time-shifted). Using the same time lag of 3 weeks, we should have expected deaths to climb 7 times between mid May to August 1. Instead, deaths from Covid-19 in Texas jumped over 9 times during this period.

This observation opens up a large can of worms. Back in July, the optimistic doctors and people were attributing the "declining" death rate not just to time delay but also to better treatments, younger people getting infected, virus becoming less lethal, etc. The data now suggest that treatments became less effective, or the virus got more lethal, etc. I'm not making such an argument, just pointing out why it is risky to draw premature conclusions from little bits of statistics.

***

Apparently, someone called time-out on these concerning numbers. On August 2, the state took a day off. When the data started flowing again, the death counts fell back to trend, as seen below:

For any data analyst, the alarm bells are deafening.

There are a number of suspicious issues here. Firstly, the official reason for the August 2 black-out was a system upgrade, and that the release for the next day would contain two days' worth of data. It doesn't appear that Monday's data contained two days' worth. The number of new cases released on August 3 was ~11,500. That is supposed to include cases for both August 2 and 3. Between July 27 and August 5, excluding Aug 2-3, daily cases were above 8,000.

Secondly, it should be simple to report deaths daily now that Texas is using death certificates to confirm deaths. I don't see how a system upgrade destroys the ability to know the dates of death. I wonder if they backtracked to the old way of reporting deaths after realizing the new procedure reveals a higher death toll.

***

Here's the problem facing data analysts in the Covid-19 era. The data are what the officials want them to be. They are constantly changing definitions, accounting rules, timing of releases, etc. It is impossible to make sense of the mess unless you maintain the database. Meanwhile, this data catastrophe is severely hampering public health decisions. The question is whether it is a feature or a bug.

 

 

Tautology in data arguments

One of the less discussed fallacies in statistical arguments is tautology. By this, I mean saying the same thing twice. Or, a conclusion that is true by assumption.

A recent example is the following popular argument:

(A) Our economy has suffered greatly because of anti-pandemic measures such as lockdowns, as evidenced by layoffs and GDP decline.

(B) Meanwhile, the deaths due to Covid-19 are insignificant - relative to the lost jobs. (*)

(C) Therefore, the anti-pandemic measures were useless and misguided.

The conclusion is that public health measures were useless.

Every statistical data argument has assumptions. What is a key assumption behind the jump from (B) to (C)? It is that the deaths due to Covid-19 as reported would be the same without the anti-pandemic measures that were imposed. Said differently, the assumption is that anti-pandemic measures are useless.

If one assumes anti-pandemic measures are useless, one will conclude that anti-pandemic measures are useless. So the argument amounts to saying the same thing twice. It is tautological.

Imagine if the opposite assumption were made, that anti-pandemic measures are effective at suppressing the death toll. Then, one will conclude that those measures are useful.

It turns out it's very easy to fall into the trap of tautologies in making statistical (data) arguments. That's because not everything is measured or measurable. Conclusions are an amalgam of data and assumptions. It's not always clear even to the analyst what assumptions have been made.

(*) Some article I came across this morning said there were 330 layoffs for every Covid-19 death in the U.S. This layoff-to-Covid-death metric is apples-to-oranges. All layoffs are counted as if there would not have been a single layoff in the absence of the pandemic (untrue) while only deaths that are confirmed with a positive SAR-CoV-2 diagnosis are counted in the denominator (under-counted).

***

Data science is not immune to tautologies. Here's one:

(A) We run an e-commerce website, and have developed an algorithm to recommend products to our customers.

(B) After the recommendation engine was launched, browsing (page views) of the top recommended products increased significantly relative to past trends.

(C) Therefore, our recommendation engine made the right recommendations.

The conclusion is that the engine recommended the right set of products. What is a key assumption behind the jump from (B) to (C)? It is that page views of other products would not increase similarly if they were recommended to the customers. Said differently, the assumption is that the engine recommended the right products to these customers.

If one assumes the recommendation engine works, one will conclude that the recommendation engine works. The argument amounts to saying the same thing twice. It is tautological.

Alternatively, one can assume that the engine is useless. Whatever is recommended to customers will get top views. In that case, one concludes that the recommendation engine is useless.

***

Have you come across examples of tautologies? Comment below!

 

 

If you check your pulse daily, one day it will flatline

When reviewing the clinical trial design for the Moderna vaccine, I mentioned the statisical fallacy known as testing to significance.

Every pharma developing a vaccine or drug must prove a statistically significant (and clinically relevant) outcome. The difference between the treatment and placebo arms must be large enough to overcome background noise coming from sampling, measurement errors, etc. Any statistical result is attached a margin of error, typically 5 percent. For a clinical trial, there is a five percent chance that a statistical significant difference is detected even if the vaccine may be useless.

Even when the vaccine is useless, five percent of the time, the clinical trial yields a benefit large enough to be "significant". Because of this, if we keep analyzing the difference between treatment and placebo day after day after day, eventually, there will be a day when the difference meets the significance threshold - even if the treatment is useless.

If we keep checking the differences beyond that day, we will discover that the significance does not last. That's not the problem. The fallacy arises if the trial is declared a success at first sight of significance.

***

I now use a simulation to demonstrate this statistical fallacy.

For the simulation, I generate 10,000 alternate worlds. In each world, I track 5000 people who get the vaccine and 5000 other people who receive the placebo. I then count the cumulative infections over time in each arm. I assume the vaccine is useless, so that infections grow at a normal rate within the community, regardless of whether the participant got the vaccine or the placebo.

The following line chart shows a single "sample path", the growth curves of infections in one of those 10,000 scenarios. (The blue line represents the vaccine group, the green line, placebo. Fewer cases are better.)

The rates of infection are just these counts divided by 5000, so the curves of rates look exactly the same.

As per the trial protocol, I follow both groups for up to 760 days (2 years after the second injection).

In the next chart, I add a third line (in yellow) to display the difference between the placebo and the treatment infections. Positive differences imply fewer infections on the vaccine arm than the placebo (vaccine may be beneficial); negative differences mean the placebo arm has fewer infections (vaccine may be harmful).

The raw differences don't matter as much as their statistical significance. When we say a difference is statistically significant, we are confident that the difference is driven by the vaccine, as opposed to background noise.

I perform significance testing on the infection rates over the course of the trial, at Day 60, 90, 120, ..., up to Day 760. Around Day 90, the gap between the placebo and vaccine arms is large enough to meet the statistical significance threshold.

If I look again on Day 240, the difference is not significant. The gap does not hold - this is in line with the assumption that the vaccine is useless. The fluctuations of the yellow line (differences in cases) are driven by random variability. (Important note: this phenomenon has nothing to do with a vaccine losing effectiveness over time. In my simulation, the vaccine is assumed useless.)

What if I test to significance? This means I end the research after finding the Day-90 difference to be significant, and conclude that the vaccine is beneficial. This is equivalent to erasing the rest of the line the first moment the line hits the significance threshold.

Such a conclusion is spurious as the simulation stipulates that the vaccine is no better than the placebo. Besides, if I continue to track the data, the Day-90 difference does not hold up.

***

Recall that I've created 10,000 alternate worlds.

The next chart shows 20 lines: each line is the difference between placebo and vaccine in one of 20 alternate worlds (selected randomly from the 10,000). I am showing you the background noise. By Day 750, in most scenarios, the observed differences hover around zero but in a few cases, the gaps are around 100 cases in one or the other direction.

The next chart shows what happens if I stop the analysis at first sight of significance. In over 25 percent of these scenarios, a winner is determined before the end of two years even though by assumption, the vaccine and the placebo groups should respond similarly. In fact, in two scenarios (10 percent), the significance threshold is crossed by Day 90.

The basic point is that if I follow each line far enough, eventually I will find a day on which the vaccine shows significant benefit relative to the placebo. If the trial continue indefinitely, those non-significant lines will eventually reach significance!

Here are 50 scenarios in which the vaccine can be declared the winner by Day 120. The thick part of the lines is observed; the thin part of the lines will not see the light of day if I declare mission accomplished by Day 120.

I remind you once more that the simulation assumes the vaccine is useless. Any finding of statistical significance is an error. This includes the lines at the top of the chart in which significance appears to hold through Day 750.

***

Let's move from the simulated fake world to the real world. In any single clinical trial, all we observe is one of these sample paths. We don't have the luxury of comparing the one observed path to the background noise. Over the course of the trial, the path reveals itself; we also cannot predict where it is heading - as demonstrated by the simulation, any path can turn at any moment.

In order to prove the success of a vaccine, it must demonstrate a statistically significant reduction in infections. The crucial question is by what time. The wrong answer is by the time we see a significant difference and before we run out of resources or patience.

This answer is wrong because it essentially guarantees success, bringing along a high likelihood of a spurious result. In the simulation, I force the vaccine to be useless (no signal, all noise). In real life, the vaccine might be slightly useful (weak signal, plus noise). This is even harder because a few of the significant readings may in fact be real.

An antidote to the statistical fallacy is patience. This is in short supply during this pandemic. The political and societal expectations are exactly opposite - stressing "warp speed". The lurking danger is confirmation bias.The pharmas track the trial results over time, and at first sight of significance, they face tremendous pressure to declare mission accomplished.

That's why I feel uncomfortable that the design documentation for the Moderna trial does not specify analysis time points. It gives an "up to" two-year time frame, which everyone knows is not real. They are not saying it would take more than two years to get a scientific result. In fact, they have told the media that we might get an answer within a few months!

***
This situation is not limited to clinical trials. I've encountered this debate throughout my career in marketing analytics. Let me give you some more color on the nature of the debate.

In one of the charts above, I show 50 scenarios in which the vaccine can be declared a winner by Day 120. The following chart displays a different set of 50 scenarios: these paths lead to the placebo reaching significance by Day 120.

If the one observed sample path comes from this chart, then by Day 120, we should stop the trial early, and declare the vaccine a failure. (As before, any such finding is spurious because I set up the simulation to assume no difference between the placebo and vaccine groups.)

No, you don't think the vaccine effort will be aborted. Neither do I.

The decision-makers will pummel the data analysts with endless questions to make sure they did not make a mistake. They can easily come up with pages of reasons why we need more data. Shutting down the study is like closing a losing position of a stock investment; you have to immediately write off the billions invested. It feels better to keep the position in the hope that the stock price or the response curves would reverse course.

Now, the exact reverse happens if the initial analysis shows the vaccine to be winning. The data analysts urge patience, wanting to more data before drawing a conclusion. The decision-makers demand swift action, arguing they can't afford to wait.

In one case, the statisticans are criticized for being too "academic" and cautious; in the mirror scenario, they are blamed for not being "certain" and using imperfect data.

Is there a way out of this? One solution is called "pre-registration". All parties have these discussions before the clinical trial even begins. Lay out the posisble outcomes. Gain consensus on what to do under each outcome. Once the data start appearing, it is virtually impossible for any side to remain neutral.

 

P.S. [8/4/2020] Based on some side conversations I've been having with a few readers, let me clarify a couple of things. I'm not suggesting that Moderna is planning to test to significance. I expect that they have protocol to mitigate this issue. I was expecting to find breadcrumbs in the document submitted to ClinicalTrials.gov but it contains no information about how the data would be analyzed. Experts say the regulators should have received a more detailed protocol but that has not been made public.

Statisticians have developed solutions to deal with this issue. Pre-specifying stages of analysis is important, and the threshold for statistical significance is adjusted to account for each early view of the data.

Edited sentence per convesation with AR in the comments.

 

We're told almost nothing about the Moderna vaccine trial

Crisis opens the door to wishful thinking. On this door may hang the tag "Operation Warp Speed," the code name for the US government's effort to fast-track a vaccine for the novel coronavirus.

News arrived last week that Moderna is to begin the Phase 3 human trial of its candidate vaccine. I am curious how we are streamlining this process, so I pulled up the document that lays out the design of this clinical trial. (link)

A vaccine trial presents many unique challenges. We cannot ethically inject someone with the coronavirus, especially those who are assigned the placebo at random. Instead, we have to round up people who live or work in high-risk environments, such as hospital staff or communities with a lot of virus. (This presents a different ethical issue, as poorer communities with a high density of minorities have been hit hard by Covid-19.)

Half the trial participants are given the vaccine shot while the other half receive a placebo. It's crucial that no one knows who has gotten the vaccine, and that the assignment to vaccine or placebo is determined by a coin flip. After administering the shots - really two shots a month apart, as Moderna learned that one shot was not enough, the scientists must wait to see what proportions of each treatment arm eventually get infected with the novel coronavirus.

***

The general design is simple enough but the operational challenges are substantial. I'll focus on three issues.

Measuring the outcome

How do we know if and when someone gets infected? Like any clinical trial, there needs to be follow up. Testing a drug such as remdesivir on patients is much easier since the participants are staying at the hospitals. In a vaccine trial, you rely on the participants roaming around the high-risk communities. Each participant must be monitored until s/he gets infected (or till end of trial, whichever occurs first). How frequently are the participants tested for infection? What test is used? How are test results reported to the trial managers?

There's more. The two doses of the vaccine are administered one month apart. How will they deal with people who tested positive after the first dose before receiving the second dose? (In my view, these people should be counted as infected.)

Timing the analysis

When will the data be analysed? This is extremely tricky requiring a lot of guesswork. How long does it take for the vaccine to do its job? We don't even have a solid handle on how many contacts people have each day with infected people, how likely it is to get infected upon contacting an infectious person, how long it takes for an infection to become detected, etc. We also need to have a firm grasp of how much virus there is in these high-risk neighborhoods. One element of any clinical trial design is a specific analysis plan. For example, in Gilead's remdesivir trial, the scientists analyzed the mortality rate at day 15, meaning each patient is tracked for 15 days after they received treatment. There may be multiple "endpoints"; another analysis was planned after 30 days for remdesivir.

Controlling for behavioral changes

Someone might have opted out of attending a birthday party if they weren't participating in the vaccine trial but after receiving the shots, they might decide to risk it. Of course, people will be told not to adjust their everyday life; for a trial lasting many months, that's a big ask. To measure such behavioral change due to trial participation, a third group can be included in the clinical trial, which will not receive an injection at all.

***

Those were three questions I was hoping to find answers to from the clinical trial design documentation. So, what did I learn?

Measuring the outcome

Surprisingly, there is nothing on the follow-up plan in the design document. We don't know what diagnostic test will be used, who will be testing the participants, how frequently are they tested, etc. This is a worrying omisison. Also omitted is how they intend to analyze people who test positive between the two doses.

Timing the analysis

The analysis plan is also largely absent. One hint is that the key metric of the infection rate has an observation window described as "Time Frame: Day 29 (second dose) up to Day 759 (2 years after second dose)". This implies that each trial participant is monitored for up to two years.

The phrase "up to" will prove pivotal. It means Moderna will not declare failure until two years have passed but they may announce mission accomplished much earlier. In fact, the leader of Operation Warp Speed predicted on CNN last night that they could have a working vaccine with over 90% effectiveness by the end of the year.

Such a statement almost defies belief. Enrollment in the trial starts for practical purposes in August. For someone who joins the trial on August 1, s/he gets the second dose on August 29, if the scientists monitor her or him for 60 days, this patient can have an outcome by November 1. No trial can enroll the entire cohort in one day. Assume it takes 30 days to find enough participants. Those who start the trial on August 31 will be ready for analysis on Nov 30. Even if the data analysts work quickly, it will be around Christmas when the first analysis can be reported.

If a vaccine has to be approved and commercially available by year's end, then they are banking on the vaccine showing a benefit within one month of the second dose. Is it reasonable or wishful thinking?

Controlling for behavioral changes

Nothing in the design document addresses this issue. It's highly unlikely they will set up a third test group as that will use up some of the trial participants, lengthening the time they'd need to report results.

***

Not specifying the time points of the analysis is a deeply worrying omission in the trial design. As I explained above, this might reflect a lack of confidence in our understanding of the virus. However, such an omission opens the door to a statistical fallacy known as testing to significance.

The following analysis procedure is the statistician's worst nightmare: every 30 days for up to two years, the Moderna analyst computes the infection rates for the vaccine and the placebo, and tests the difference for statistical significance; if the difference is not significant, the analyst waits another month and repeats the process; if the difference is significant, the analyst declares victory, ending the trial after approval from the review board.

Such a procedure sounds reasonable but it is the source of a lot of spurious statistical findings.

I'm preparing another post to explain this statistical fallacy. What you need to know is that if one waits around long enough, anything can happen. In the context of a clinical trial, if one checks for a statistically significant difference between the vaccine and placebo arms every 30 days for years, at some point, such a difference will appear - even if the vaccine is useless.

In order to prevent such spurious results, statisticians recommend nailing down an analysis plan prior to running an experiment. This is precisely why clinical trials are "pre-registered" - why Moderna is required to submit the trial design prior to the start of this clinical trial. The analysis plan should not be influenced by peeking at the interim results. This possibility is present in the Moderna vaccine trial since the design document is missing some crucial details.