Big Data, Plainly Spoken (aka Numbers Rule Your World)

Dr. Backlog jumps the pond

The UK has joined California, Texas, etc. in the list of organizations that couldn't keep their testing data straight. It just admitted that over the last week or so, it under-counted Covid-19 cases by 30 percent.

(Graphic from BBC.)

The official reason is an Excel limitation, which is a brilliant stroke of PR (I have previously praised the PR firm behind the UK's coronavirus response. See here.) This message is intended for those who do not use Excel, which, I guess, is the majority of the UK population. Among those who have been exposed to Excel, there exists a set of vocal dissenters, people who hyperventilate at the mention of Microsoft's software, and they will help propagate the message.

Anyone who uses Excel a lot will not fall for this.

Unlike California and Texas, where the officials remain mum about the details of their supposed faux pas, the UK government explained to the Daily Mail what this Excel error is. We are to believe that there is a master spreadsheet recording all cases, that spreadsheet ran out of space, nobody noticed it until a backlog of 16,000 cases has stacked up, and they fixed the issue by splitting the data onto multiple spreadsheets.

One word: implausible.

***

The limit on the number of rows on a single Excel spreadsheet is above 1 million. The cumulative number of cases in the UK as of October 2 was fewer than 500,000. Assuming that they collect data for each individual case, so that there is one row of data per case, the spreadsheet should be at least 50% empty.

I'm not convinced that's the workflow. It would seem more likely that there is one line per reporting unit, not one line per individual case. If that is so, then the spreadsheet is even smaller.

Let's play along and assume somehow someone created a process filled with duplicates and errors or other horrors such that the spreadsheet manages to overrun 1 million rows.

We now have to believe that the Excel spreadsheet is maintained by a machine without a single human opening it. If a human is transferring the data onto this master spreadsheet, as I expect to be the scenario, how is it that this live person is not alert to the space outage? One simply cannot add any more rows to the bottom of the sheet!

If a machine is programmed to add rows to this spreadsheet, we now have to believe that neither Excel nor the program returns any error messages or warnings when the spreadsheet runs out of room!

Let's keep playing along. Against all odds, the spreadsheet is somehow completely full and nothing more can be added. Neither humans nor machines notice anything odd. The new data simply fell into a black hole.

Now, how do we explain that the case counts are increasing each day for a week before they noticed the missing cases?

From September 25 to October 2, the period in question, the UK reported on average 4,368 cases before admitting the mistake. The revised data bumped the daily average to 6,348 cases, which means the public was given a number 30 percent below the real tally.

According to the Daily Mail, "some 16,000 confirmed infections had to be added to the daily totals running back more than a week." If we buy the Excel story, what should have happened is that one day, the spreadsheet ran out of space, and from that point on, the case count plateaued with no more additions.

What should have happened is suddenly no more cases are recorded, and the error is immediately obvious.

P.S. Lets' keep playing. So they keep one gigantic spreadsheet not just of positive cases but of all test results from which they extract the cases. As of October 2, the U.K. said they have "processed" 22.7 million PCR tests. Remember the limit of a single spreadsheet is 1 million, which has been passed long time ago.

***

Dr. Backlog sure chose a curious time to jump the pond. The undercounting occurs right as cases were spiking in the U.K., France, Spain, and all over Europe, and governments had to make the difficult decision of whether or not to re-impose restrictions to ward off a second wave.

Yes I will be the first to admit the data world is messy, and there can be innocent explanations for data discrepancies. However, any experienced data analyst realizes one cannot lazily cling to the most convenient explanation; all possible explanations must be investigated, including politically incorrect ones.

A testing mess across the pond

Controversy has engulfed the A-Level examinations in the U.K. The A-Levels are university entrance level exams of subjects (math, English literature, history, economics, computer science, etc.). Universities conditionally admit students with minimum grade requirements for relevant subjects. The A-Level results are often the most important admissions criterion, by far. Their closest American cousins are the SAT Subject Tests, although in the U.S., these tests are not widely required, and no college conditions admissions on test scores.

The coronavirus pandemic threw a wench into the exams system. Students could not sit for these exams in 2020 thus the authorities concocted a statistical scoring model to assign grades. The model predicts what grade the student would have scored had s/he sat for the exam.

Roughly speaking, each grade has three components: a baseline score that applies to every student, a school factor that applies to all students of the same school, and an individual factor that applies to each student within the same school. The school factor is based on performance of past cohorts of students in the A-Levels. The individual factor is based on the student's prior performance in other public examinations, and on a teacher evaluation.

As part of this exercise, teachers are asked to grade their own students, which have been described by students as their "predicted grades". On results day, a lot of students and their teachers were disappointed by the assigned grades. They found that the model's predicted grades are lower than their teachers' predicted grades. There was a huge outcry about so-called downgrades. The statistical modeling is attacked as "utterly unfair and unfathomable", hurtful to young minds, inconsistent, disappointing, lacking transparency, etc. (See this BBC Live Feed for reactions.)

***

The term "downgrade" interests me a lot. We have two predictions of an unknowable quantity, how the student would have performed had exams not been cancelled. These predictions sometimes don't match.

From the students' perspective, the conditional offers from universities set targets for A-Level grades. Their teachers told them what grades to expect, i.e. the predicted grades. The official grades they receive are evaluated relative to these teacher assessments. If the official grade is lower, it's a "downgrade" and if it's higher, it's an "upgrade".

I believe the exam administrators are shocked by this reaction. They probably regard the teacher assessment as one input in a complex process that generates official grades. They probably see the model's assigned grade as the "truth". It's the teacher assessment that may be over- or under-estimates.

This situation reminds me of my recent post on how the business community interprets metrics about the economy. They take their expectations - potentially generated by models - as the reference level. Official statistics on unemployment, inflation, etc. are evaluated relative to these expecations, rather than the other way round.

***

The media has by and large sided with students and their teachers. In this post, I provide the other side of the story, the challenges faced by the statistical modelers.

There are limited capacity at U.K. universities, and particularly at the more competitive schools. Admissions officers set A-Level grade requirements, the conditional offers, based on historical distribution of grades, and their historical yield rates. If the exams were not cancelled, the administrators are reasonably confident that the grade distribution in 2020 would have been similar to those of recent years. 

Not surprisingly, the distribution of grade predictions by teachers exhibits grade inflation. The exams authority, Ofqual, published the following chart:

The yellow line represents the proportion of A and A+ grades for the last four years. The orange dot labelled CAGs represents teacher assessments for 2020. Ignoring the 2020 yellow dot for the moment, you see that teachers in aggregate assigned 12.5 percent more A and A+ grades than the 2019 results. As a ratio, this is 50 percent more top grades than the year before!

The so-called downgrading describes the gap between the orange dot (37.7%) and yellow dot (27.6%) in 2020. The teacher assessments were fed to the statistical model which brought the proportion of top grades down to roughly the historical level. In fact, the proportion of top grades issued by the model is 1.4 to 2.4% higher than in recent years.

What this means is the attack on the model is rather misguided. I'm sure there are problems with the model, as with any model, but no model can escape the constraint that the proportion of As and A+s should be about 25 percent. That means all models will issue a lot of "downgrades". They differ by which students are downgraded.

Whoever gets downgraded has a grievance. We haven't heard complaints from those who got upgraded (about 2 percent of grades), and probably won't. Such asymmetric complaints tend to make models worse. Deficiencies are remedied along one dimension but not the other, which introduces a "correction bias" to the model.

I cover this issue in Chapter 2 of Numbers Rule Your World (link) in the context of credit scoring models. When the credit bureaus allow us to check the data, we will challenge anything that might reflect negatively on the credit report, but if the bureau makes a mistake that benefits us, for example, if the report failed to record a prior delinquency event, most of us will not volunteer this information (that will certainly lower our credit score).

***

Another factor that no model can escape is the differential quality of schools. Students of some schools have historically done better at A-Levels and other public exams. It is a bit unfair that a student's grade is partially determined by the quality of the school s/he attends.

In fact, standardized testing solves this problem, except the pandemic wiped that away in 2020. To a large extent, all models assume the future is like the past. Existing inequities and biases get propagated.

 

Excess deaths, reporting delays, statistical adjustments, misleading headline

The New York Times published a very nice presentation on "excess deaths" in the United States from March through July. (link)

The following chart shows how public health policies utterly failed in many Northeastern states in the April and May.

In Connecticut, for example, in the middle of April, the number of deaths exceeded expected deaths by 2.6 times. In Massachusetts, around the same time, actual deaths exceeded expectation by 2.3 times.

A lot of work went into generating these charts - starting with meticulous data collection from each state, and as much as possible, reconciling how metrics are defined differently across states. Then, there is a bit of statistical modeling.

Modeling is required because "excess deaths" is - like most statistical quantities - not directly observable. Excess deaths is the difference between actual deaths (from all causes) in the presence of the Covid-19 pandemic and expected deaths (also from all causes) in its absence. The former is directly measurable but the latter is not.

To establish how many people could have died during a specific week in Connecticut if the novel coronavirus did not exist, statisticians look at deaths during the same week in that state in past years. Over a number of years, one should find some statistical consistency in the death rate. This illustrates the power of averaging.

If 2020 were a normal year, the chart should look something like this:

The above line is for Hawaii, which, because of its remote location, has not been affected much thus far by Covid-19. What you see is the actual weekly deaths hovering around the baseline (corresponding to the historical average), sometimes rising above, and sometimes dipping below.

The line for Connecticut, though, looks nothing like that for Hawaii. The most important feature is that the entire line sits above the baseline, meaning that there have been excess deaths for every week since March.

***

Ready for even more modeling?

For most states, the time line reaches the last week of July, which may surprise you. The first part of excess deaths - the observable part - counts all causes, obviating the need to verify coronavirus test results. The measure relies on death certificates. But it may take weeks to months to receive data on death certificates so how does NYT have estimates of excess deaths up to the week prior?

Does NYT have a crystal ball? In a sense, yes. It uses a statistical model to "top up" the incomplete deaths data in the most recent weeks. It turns out that actual deaths are not completely observable always. There is a reporting delay which means that the data for more recent weeks are incomplete. The data age like fine wine - the older the data, the more complete.

The NYT utilized a similar strategy as the model for projecting expected deaths - assuming historical patterns hold. The CDC keeps track of the amount of reporting delay. It might know that typically, 20 percent of the data showed up with one weekrepo's delay, 50 percent with two weeks' delay, and so on.

There is a crucial difference betweeen these two statistical models, though. The first model projects expected deaths in the absence of the novel coronavirus. The second model tops up the actual deaths reported at the time of analysis, essentially projecting the number of deaths already occurred but not yet reported. These actual deaths are in the presence of the coronavirus. This is the crucial difference. This second model has a further assumption - that reporting delays have not been affected by the pandemic.

The NYT team disclosed this assumption clearly in the text. This is what they mean by the following sentences:

Even with this adjustment, it's possible there could be an underestimate of the complete death toll if increased mortality is causing states to lag more than they have in the past or if states have changed their reporting systems.

It's quite likely that the volume of deaths, the budget pressure, and political interference have changed the pattern of the reporting delay.

The reporter then argued that the adjusted estimates are still better than the unadjusted ones. One can't refute that argument. Of course, this discussion assumes that the chart must show a time line leading to the present. The designer can choose to show only those weeks with virtually complete data.

Or, we can bring in a third model. This one looks at the early weeks of the pandemic, for which we believe almost all of the death-certificate data have been recorded. We can then estimate the error of the second model that serves to top up the actual deaths.

***

NYT's excess-death analysis has found over 200,000 more deaths (from all causes) in the U.S. between March and July than expected based on historical patterns. Given the U.S. counted about 140,000 deaths due to Covid-19 during this period, there have been about 60,000 unexplained deaths so far.

I find the NYT headline, "The true coronavirus toll in the US has already surpassed 200,000," alarmist. The analysis did not support this conclusion - unless they make the further assumption that the only factor causing excess deaths during those months is Covid-19. That is plausible but not fact-based. A more correct headline should be "Deaths due to Covid-19 may have been undercounted by up to 40 percent." That's also shocking and sobering.

Is the U.S. speedboating? The merging of business and politics

In response to the note about the time lag between infection and deaths, Mark P. pointed me to an old blog post of his that features another example from the business world.

Mark's example comes from money lending. In that world, a key metric is the "charge-off rate". This is defined as the amount of new charge-off (i.e. bad loans) divided by the total amount of outstanding loans. Both numbers are computed as of the current date. What's the problem? Loans don't typically go bad right away. In fact, as Mark outlined, the industry won't even write off loans until it is already, say, 90 days past due.

And yet, investors evaluate lending businesses based on this "charge-off rate". How is this being gamed? Mark explains:

Let's say I book an account that goes delinquent after one year. That was a bad deal for the bank – – it lost money due to that decision – – but for one year, having that account actually lowered the bank's charge-off rate. Eventually, of course, this will catch up with the bank, but the reckoning can be delayed if the bank continues to book these bad accounts at an increasing rate.

I'd add that (a) the decision-maker has earned all his/her bonuses, and may not even be around for the reckoning, (b) the reckoning can be very far off so long as the bank has a pipeline of these risky creditors to lend money to, and (c) in the brave new world of massive corporate bailouts, the reckoning might show up as free money, more bonuses, and a stock price spike.

I'd further add that this practice - apparently having a nickname of "speed boating" - (a) is legal and (b) violates no accounting rules.

Of course, the whole mess can be avoided if the investors focus on cohort-adjusted charge-off rates. Computing the adjustment does require better data management.

***

Business executives are extremely good at crunching numbers. They may not know statistical theories, but they figure out what metrics investors are concerned about, and how to dress those numbers up. The U.S. President is a business exeuctive. He wasn't joking and he is not wrong when he said Covid-19 cases go up with more testing. (Don't forget deaths go up with more testing, too. And it's not how many you test but who you test that is the key to understanding those numbers.)

Let's talk about data backlogs

Yesterday, I featured some weird things I saw in Texas's Covid-19 data. A host of issues are officially recognized and flagged by sites like Covid Tracking Project. Reasons for bad data include software upgrades and data backlogs. News also came out that California "found" hundreds of thousands of unreported cases (link). This was blamed on a server outage necessitating code revision that stayed longer than it's welcome resulting in a data backlog. (I'm not joking; this came straight from the official explanation.)

If you haven't already, find a friend who has worked with databases, and ask if s/he believes these stories. Or, just keep reading...

 

Data Backlogs

If your job is to maintain a data pipeline that merges data from many sources, you will at a minimum create an audit report that tells you how much new data have been processed from each data source every day. If that is too much effort, you will make a report that tells you whether each data source has sent their data (yes/no) at the agreed-upon time each day. There is no database administrator who haven't made such a report. If someone hasn't, rest assured that these audit reports appear in the inbox of the CEO, the day after the first data processing error shows up, which probably occurs during the first week on the job.

It's very common, especially in modern, loosely managed distributed databases, that some minute or hour of data went unprocessed. Those are sometimes hard to detect from daily or weekly statistics in the audit reports.

It is simply inconceivable that hundreds of thousands of missing records have escaped attention. I laughed uncontrollably reading this delicious sentence: "the state hadn't realized they weren't receiving data from one of the largest commercial labs for five days." Oh, and the White House press corps didn't realize the President hasn't tweeted for five days. Missing multiple days worth of data goes beyond incompetence.

California is a repeat offender. I previously wrote about another such situation that happened on April 22, when on a single day, the state published 10 times the usual amount of data - to add insult to injury, this batch of data had one-tenth the usual number of positives.

 

Software Upgrades

Texas announced a data blackout on August 2, and then released two days' worth of data on August 3. A scheduled software upgrade was the official reason. But a software upgrade does not cause two days of data to get mixed together.

Data typically come with many dates. Two key dates are the date of the event (e.g. date of death, or date of diagnostic testing), and the date of the report. The reporting date is generally different and later than the date of the event. The systems at hospitals and clinics - the source of testing data - apply a time stamp to mark when the data are reported to the public health agency. The agency adds a time stamp recording when it uploads the data to its databases.

A software upgrade at the Texas public health agency isn't going to erase dates collected at hospitals and clinics. There is no reason why they can't save the data files for August 2 separately from those for August 3. Most likely, the hospitals and clinics probably have those files saved just like that, with the date of reporting as part of the file names. There is no logical reason why the two days of data must be aggregated.

Why couldn't the Texas public health agency divert the August 2 data to a temporary hard drive? Why couldn't it ask hospitals and clinics to send two files? Why couldn't it parse the dates in the data files to separate out the two days? Any of these solutions would have just delayed the release of the data but not scrambled two days of data. This is not rocket science. This behavior goes beyond incompetence.

***

Since the states blame these problems on server outages and software upgrades, they should name the vendors of these technologies. Which database needed an upgrade and why? What server broke down? If the upgrade was "scheduled", why wasn't there a contingency plan for storing the data temporarily in a different place? What measures are taken by the new technologies or vendors to prevent these same problems from happening?

Bad covid data: feature or bug in Texas?

Let me show you how bad the state of Covid-19 data is in the U.S.

Recall my post from two weeks ago about Covid-19 deaths in Texas starting to catch up with the earlier rise in reported cases. When cases started to spike in Texas, talk turned to the declining death rate, which is misleading since we should have been looking at cohort-adjusted death rates from the start. The media conspired to spread this misinformation, ignoring epidemiologists - when given a choice between reporting a misleading statistic today and reporting a more valid number in two weeks, the media cannot help but quench the immediate thirst.

In putting together the prior analysis, I already noticed the sad state of the data. On that day (July 27), Texas just changed how it reports deaths to rely on cause of death on death certificates. This created a single-day death toll of 675, well above the daily average, which presumably corrected for under-counting from previous days. With this correction, the state claimed that it will henceforth provide "more accurate" death counts.

What the state official did not explain is that the change in reporting also ensures a reporting delay for all deaths of at least one week. Let's think through this in the context of rising case rates and "declining" death rates. I make the nonsensical assumption that if patients die from Covid-19, they die immediately, on the day they are reported as new cases. Under the new reporting procedure, these cases will be reported today but the deaths will be reported when death certificates have been issued and can be inspected, which is at least 7-10 days in Texas. Thus, this group of patients (who already died) inflates the number of cases today while suppressing today's death rate since their deaths are not yet reported. This time lag is due to reporting procedures, and has a similar effect as the time lag between infection and death.

The data source, Covid Tracking Project, rated Texas "A" for "data quality" so presumably the state of the data elsewhere is worse.

***

I intended to update the analysis for this post last week but my effort was met with more data mischief. On August 2 (Sunday), Texas did not report any data, blaming a "systems upgrade" and promised that "data for Sunday will be posted with Monday’s data update." So I waited another week before publishing this update.

Just a reminder, two weeks ago, I presented the following chart:

The thin gray line is the growth curve for reported cases in Texas, smoothed using a seven-day window, from mid May through late July. The thick gray line is this growth curve shifted forward in time by three weeks. The red line is the growth curve of deaths in Texas, smoothed using a seven-day window. Except for the single-day anomaly (July 27), it appears that deaths roughly lagged cases by three weeks.

The next chart extends the time-line to August 1, the day before the data blackout:

This chart is disturbing. The growth of deaths in Texas has started to markedly outpace the growth of cases (time-shifted). Using the same time lag of 3 weeks, we should have expected deaths to climb 7 times between mid May to August 1. Instead, deaths from Covid-19 in Texas jumped over 9 times during this period.

This observation opens up a large can of worms. Back in July, the optimistic doctors and people were attributing the "declining" death rate not just to time delay but also to better treatments, younger people getting infected, virus becoming less lethal, etc. The data now suggest that treatments became less effective, or the virus got more lethal, etc. I'm not making such an argument, just pointing out why it is risky to draw premature conclusions from little bits of statistics.

***

Apparently, someone called time-out on these concerning numbers. On August 2, the state took a day off. When the data started flowing again, the death counts fell back to trend, as seen below:

For any data analyst, the alarm bells are deafening.

There are a number of suspicious issues here. Firstly, the official reason for the August 2 black-out was a system upgrade, and that the release for the next day would contain two days' worth of data. It doesn't appear that Monday's data contained two days' worth. The number of new cases released on August 3 was ~11,500. That is supposed to include cases for both August 2 and 3. Between July 27 and August 5, excluding Aug 2-3, daily cases were above 8,000.

Secondly, it should be simple to report deaths daily now that Texas is using death certificates to confirm deaths. I don't see how a system upgrade destroys the ability to know the dates of death. I wonder if they backtracked to the old way of reporting deaths after realizing the new procedure reveals a higher death toll.

***

Here's the problem facing data analysts in the Covid-19 era. The data are what the officials want them to be. They are constantly changing definitions, accounting rules, timing of releases, etc. It is impossible to make sense of the mess unless you maintain the database. Meanwhile, this data catastrophe is severely hampering public health decisions. The question is whether it is a feature or a bug.

 

 

We're told almost nothing about the Moderna vaccine trial

Crisis opens the door to wishful thinking. On this door may hang the tag "Operation Warp Speed," the code name for the US government's effort to fast-track a vaccine for the novel coronavirus.

News arrived last week that Moderna is to begin the Phase 3 human trial of its candidate vaccine. I am curious how we are streamlining this process, so I pulled up the document that lays out the design of this clinical trial. (link)

A vaccine trial presents many unique challenges. We cannot ethically inject someone with the coronavirus, especially those who are assigned the placebo at random. Instead, we have to round up people who live or work in high-risk environments, such as hospital staff or communities with a lot of virus. (This presents a different ethical issue, as poorer communities with a high density of minorities have been hit hard by Covid-19.)

Half the trial participants are given the vaccine shot while the other half receive a placebo. It's crucial that no one knows who has gotten the vaccine, and that the assignment to vaccine or placebo is determined by a coin flip. After administering the shots - really two shots a month apart, as Moderna learned that one shot was not enough, the scientists must wait to see what proportions of each treatment arm eventually get infected with the novel coronavirus.

***

The general design is simple enough but the operational challenges are substantial. I'll focus on three issues.

Measuring the outcome

How do we know if and when someone gets infected? Like any clinical trial, there needs to be follow up. Testing a drug such as remdesivir on patients is much easier since the participants are staying at the hospitals. In a vaccine trial, you rely on the participants roaming around the high-risk communities. Each participant must be monitored until s/he gets infected (or till end of trial, whichever occurs first). How frequently are the participants tested for infection? What test is used? How are test results reported to the trial managers?

There's more. The two doses of the vaccine are administered one month apart. How will they deal with people who tested positive after the first dose before receiving the second dose? (In my view, these people should be counted as infected.)

Timing the analysis

When will the data be analysed? This is extremely tricky requiring a lot of guesswork. How long does it take for the vaccine to do its job? We don't even have a solid handle on how many contacts people have each day with infected people, how likely it is to get infected upon contacting an infectious person, how long it takes for an infection to become detected, etc. We also need to have a firm grasp of how much virus there is in these high-risk neighborhoods. One element of any clinical trial design is a specific analysis plan. For example, in Gilead's remdesivir trial, the scientists analyzed the mortality rate at day 15, meaning each patient is tracked for 15 days after they received treatment. There may be multiple "endpoints"; another analysis was planned after 30 days for remdesivir.

Controlling for behavioral changes

Someone might have opted out of attending a birthday party if they weren't participating in the vaccine trial but after receiving the shots, they might decide to risk it. Of course, people will be told not to adjust their everyday life; for a trial lasting many months, that's a big ask. To measure such behavioral change due to trial participation, a third group can be included in the clinical trial, which will not receive an injection at all.

***

Those were three questions I was hoping to find answers to from the clinical trial design documentation. So, what did I learn?

Measuring the outcome

Surprisingly, there is nothing on the follow-up plan in the design document. We don't know what diagnostic test will be used, who will be testing the participants, how frequently are they tested, etc. This is a worrying omisison. Also omitted is how they intend to analyze people who test positive between the two doses.

Timing the analysis

The analysis plan is also largely absent. One hint is that the key metric of the infection rate has an observation window described as "Time Frame: Day 29 (second dose) up to Day 759 (2 years after second dose)". This implies that each trial participant is monitored for up to two years.

The phrase "up to" will prove pivotal. It means Moderna will not declare failure until two years have passed but they may announce mission accomplished much earlier. In fact, the leader of Operation Warp Speed predicted on CNN last night that they could have a working vaccine with over 90% effectiveness by the end of the year.

Such a statement almost defies belief. Enrollment in the trial starts for practical purposes in August. For someone who joins the trial on August 1, s/he gets the second dose on August 29, if the scientists monitor her or him for 60 days, this patient can have an outcome by November 1. No trial can enroll the entire cohort in one day. Assume it takes 30 days to find enough participants. Those who start the trial on August 31 will be ready for analysis on Nov 30. Even if the data analysts work quickly, it will be around Christmas when the first analysis can be reported.

If a vaccine has to be approved and commercially available by year's end, then they are banking on the vaccine showing a benefit within one month of the second dose. Is it reasonable or wishful thinking?

Controlling for behavioral changes

Nothing in the design document addresses this issue. It's highly unlikely they will set up a third test group as that will use up some of the trial participants, lengthening the time they'd need to report results.

***

Not specifying the time points of the analysis is a deeply worrying omission in the trial design. As I explained above, this might reflect a lack of confidence in our understanding of the virus. However, such an omission opens the door to a statistical fallacy known as testing to significance.

The following analysis procedure is the statistician's worst nightmare: every 30 days for up to two years, the Moderna analyst computes the infection rates for the vaccine and the placebo, and tests the difference for statistical significance; if the difference is not significant, the analyst waits another month and repeats the process; if the difference is significant, the analyst declares victory, ending the trial after approval from the review board.

Such a procedure sounds reasonable but it is the source of a lot of spurious statistical findings.

I'm preparing another post to explain this statistical fallacy. What you need to know is that if one waits around long enough, anything can happen. In the context of a clinical trial, if one checks for a statistically significant difference between the vaccine and placebo arms every 30 days for years, at some point, such a difference will appear - even if the vaccine is useless.

In order to prevent such spurious results, statisticians recommend nailing down an analysis plan prior to running an experiment. This is precisely why clinical trials are "pre-registered" - why Moderna is required to submit the trial design prior to the start of this clinical trial. The analysis plan should not be influenced by peeking at the interim results. This possibility is present in the Moderna vaccine trial since the design document is missing some crucial details.

 

 

 

 

It's high time to call out those who got it right

During this pandemic, epidemiologists have been portrayed as clueless idiots who "got everything wrong". That makes sense only if one ignores the evidence. I can go so far as to say that epidemiologists have gotten everything right - they have not predicted every detail but they have nailed the big picture. (They told us about re-opening too soon, second waves, viruses not chased away by summer heat, etc. etc.)

Remember just a month ago, when cases started to surge in Texas in June, many people looked at the glass half full - they said deaths weren't increasing even as cases exploded. This was the picture they were looking at:

I wrote a blog about this statistical artifact at the end of June. Deaths always lag cases. I said you've got to have patience. Stop interpreting daily tracking charts, and wait a few weeks.

It's more than a month later. What does the chart look like?

By early July, deaths from Covid-19 started to rise rapidly in Texas. In fact, the slope of the growth of deaths in July is roughly the same as the slope of the growth in cases in mid-June.

To see what happened more clearly, I shift the gray line (cases) to the right by three weeks. You can see that deaths and (shifted) cases have changed in lock step.

This chart reveals a couple of insights.

First, the time lag is sufficient to explain everything. There have been speculation about other contributing factors, such as better treatments, and younger patients but those factors are not necessary to explain the trends in cases and deaths. It's important for any insights to be validated by the experience on the ground. Hospitals should know how long an average patient who eventually dies from Covid-19 stays hospitalized.

Second, notice the weird spike in deaths on July 27 (the spike has already been tamed by a 7-day moving average); that's due to a change in how deaths are counted. Here are some notes from the Covid Tracking Project about all the recent changes to how they count things in Texas:

The recent switch of requiring deaths to be confirmed by death certificates, for example, introduces a delay in reporting. It may take a week or two for a death to appear in the data. It would take quite a bit of effort to go back and switch the entire series of data to reflect this new definition. Meanwhile, it's impossible to trust the dates of deaths.

Every note listed above should concern the data analyst. That's the key reason I cannot have confidence in the 3-week time-lag estimate described above. It must come with a margin of error, probably of at least 1 week, to account for data deficiencies, known and unknown.

It's very sad to see that the Covid Tracking Project team has rated Texas's "data quality" as grade A. I can't even imagine how bad the data quality is to earn a B or C grade!

 

 

The power of groupthink

In a congressional hearing this week, a leader from the CDC (or was it NIH, I couldn’t recall) mentioned “group testing” as a new tactic to be rolled out. Unfortunately, this may be another case of too little, too late for the United States. But the concept of group testing is rooted in statistics, so let’s talk about why it is an attractive idea.

Imagine we need 20 cc of blood to conduct a diagnostic test. Test kits are short in supply, so we must make some magic. Normally, we use one test kit per person tested. Is it possible to test N people using fewer than N kits?

Statistics enter the picture. We’ll consider 10 people. Take 2 cc of blood from each sample and mix them thoroughly. Use one test kit on the pooled sample. If this pooled sample tests negative for the virus, we conclude none of the 10 are infected, and so save nine test kits. On the other hand, if the pooled sample tests positive, we have to next test each individual, so in this case, we use 11 kits for 10 people.

There is a chance we may use fewer than 10 test kits. But it’s not guaranteed. Whether we will end up with savings depends on the proportion of pooled samples that test negative. A key calculation is the probability that when forming groups of 10 people, all ten are not infected.

Suppose 10 percent of test-takers have the virus. Then, in an independent pool of 10 people selected at random, the chance that no one is infected is (0.9)^10 = 35%. Thus, we expect 35% of these pools of size 10 to test negative, resulting in a saving of nine kits each. For the other 65% of pools, we run one extra test relative to the base case of 10 tests for 10 individuals. On average, we save 2.5 tests for every 10 people. The saving rate is 25%, a quarter fewer test kits. Not bad!

***

For the U.S., is this development coming too late? The group testing strategy confers savings only if the virus is not too widespread. If there are too many infected individuals, then most pools will have at least one infected person, and instead of saving kits, we may burn up even more kits than normal. Let’s repeat the previous calculation but assume 20% of the test-takers have the virus. In this case, about 10% of the pools have no virus while 90% comprise at least one infected sample. On average, we break even – the expected number of tests using group testing of size 10 is ten tests, the same as one test per person. All for naught.

For those interested, you can analyze different pool sizes.

***

Breakeven is not neutral though. If I were the decision-maker, I’d need the group testing strategy to show a clear, positive benefit. That’s because the group testing method has several non-obvious costs.

First is the cost of administration. One test per person is simple to execute. Group testing requires splitting samples, storing samples until the pooled test result is available, matching samples, etc. Much better book-keeping and tracking are required.

A second cost comes from potential errors due to complexity. Samples can get contaminated, mixed up, or lost.

Third is its impact on test-takers. More blood or sputum must be collected in anticipation that some of the samples will be tested twice. Also, it will take longer for the results to be confirmed.

The fourth cost is more false negatives. Imagine a pooled sample with exactly one infected individual. By taking one-tenth of each sample and mixing them together, we dilute the viral load of the infected individual 10 times. It is possible that the pooled sample – and thus all 10 individuals - will be declared negative. If the infected person were tested alone, the viral load would have been high enough to return a positive but the pooling causes a false negative. Such false negatives may make kit savings a pyrrhic victory.

***

Significance magazine has a recent article about group testing.

How to act like a data scientist 11: do not assume random

Credit: ep_jhu/Flickr

 

You might find it curious, that someone who pointed out how testing drives the case numbers from my very first Covid-19 post (here), and who called for broad, non-targeted testing in this Wired column also from early in the pandemic (also here), has been silent on testing statistics. Specifically, I have not - until now - commented on the media's two favorite testing metrics: the number of tests per capita, and the "positivity" ratio, i.e. the proportion of tests that came back positive.

The reason? I quote from my post earlier in the week:

In the short history of the Covid pandemic, people started with case statistics. Then, they claimed that death statistics are less manipulated than case statistics, when they learned about testing. Then, they claimed that testing statistics are less manipulated, until they realized governments determined who got tested. Some governments counted tests shipped, not test results. Then, they claimed hospitalization statistics are less manipulated, until they learned that hospitals sent sick patients to nursing homes.

Commentators citing and interpreting these statistics behave as if they aren't manipulated. The President already contradicted them on that count. Manipulation is both intentional and unintentional. (I use unintentional intentionally to include intended unintentional.)

In my Wired column, I pointed out that one huge concern with non-random testing is it produces biased data that prevent us from understanding the pandemic. What makes this worse is when analysts speak about the statistics as if they came from randomized testing.

The positivity rate going up only tells you that of the people who were recently tested, more people have tested positive. If people who got tested were a random selection of the entire population, then yes, it indicates that the coronavirus would have spread more widely. A higher proportion of the population would have been infected.

This is not a given when the sample of people tested is non-random. Another reason why positivity rate goes up is more infected people are getting tested than before. (This is a broader statement than saying more people are getting infected.) Randomized testing almost never happens on its own - it must be organized. If the sample were truly random, you'd expect that someone who is not infected is equally likely to present themselves for testing as someone who is infected. Are you kidding?

Greater spread is one of several reasons for positivity rate to go up. Unless the analyst has examined the other reasons and found them lacking, then the conclusion is based on assuming greater spread, and assuming the other reasons do not apply. It is a tautology.

So, anyone saying positivity rate going up implies Covid-19 is spreading is doing some or all of: (a) assuming that Covid-19 is spreading, (b) assuming that the tested population is randomly selected, (c) assuming that the tested population has always been randomly selected [demonstrably wrong], (d) assuming that uninfected people are comprising a growing proportion of those being tested, (e) assuming testing data are comprehensive and accurate, (f) assuming no one is manipulating the testing data.

***

Regarding manipulation, we have indirect evidence, such as the U.S. President musing about slowing down testing. We also know that the UK government counted "tests shipped" rather than "tests results returned". This is like automakers counting "sell through" rather than "sell out". Sell through tallies include cars sitting in lots of dealers, and is part and parcel of the "channel stuffing" strategy to inflate short-term revenues. Sell out happens when drivers take ownership of the cars. Recently, the UK government stopped publishing testing numbers. In the U.S., I previously reported that California's testing numbers were fishy. I'm suspicious of testing data in the U.S. given the variety of public and private entities conducting tests (while being appreciative of those who put in effort to gather such data.)

The fact that few governments have implemented random testing, even to a small subpopulation, makes it clear that politics is interfering with the public health imperative. The side effect is suboptimal policies misled by low-quality data.

Recall what I said here: don't blame low-quality data for misguided policies when politicians have the wherewithal to improve the data quality.

***

For data scientists, this is yet another instance of garbage in garbage out. In an age in which the provenance of data is frequently unclear, it's worth our time to investigate and interrogate the data - and never assume random when we simply don't know.