Big Data, Plainly Spoken (aka Numbers Rule Your World)

Masks and vaccines

United Nations Covid-19 Response

Wasn't planning on blogging today but the news cycle won't leave me alone :)

CDC Director Dr. Redfield made a bit of a stir yesterday when he told Congress - and I paraphrase - that masks may offer individuals greater protection than vaccines.

This is a very thoughtful statement which rewards some further comments.

***

The way medicine is practiced, we are often misled to over-estimate the impact of drugs and vaccines. When the doctor says take pill X to treat condition Y, we think if we take pill X, condition Y will be cured. In reality, for most conditions and meds, if we take pill X, there is a chance C that pill X will cure condition Y. C is not 100 percent, and is usually way below 100 percent.

Flu vaccines are said to be 50 percent effective or less. What does that mean? Perhaps, that half the people who've gotten flu shots will be protected while the other half will still catch the flu. This isn't how experts use the word "effectiveness" but let me stay with this intuitive definition for now. (I'll turn to the technical definition in the last part of the post.)

This situation reminds me of the famous put-down of TV advertising: half the budget is wasted but we don't know which half.

The trouble of course is that if you halve the TV spending, you will still waste half the money while also losing half the benefit. Same with the vaccines. If only half the people get vaccinated, a quarter of the people will be protected.

The FDA has announced that a SARS-Cov-2 vaccine with 50% effectiveness is acceptable. Staying with the intuitive definition, this means getting the vaccine shot is like buying a lottery ticket with a 50 percent chance of winning. This is not news though. Taking any pill X or undergoing any surgery  has always been a lottery with C percent chance of winning. The value of C depends on the treatment.

One of the reasons why some vaccinated people will still catch the flu is biological variation. Different people respond to the vaccine differently.

***

What about wearing a mask? The mask works as a physical barrier and its function is not affected by anything biological. The same mask worn properly confers the same degree of protection on anyone. Any decent mask blocks some fraction of virus from entering one's respiratory system. Imagine an infectious person coughing directly at me. The mask reduces the amount of viral projectile even if it is not eradicated.

In other words, wearing a mask has a high C. Even though I am a skeptic of many things, mask wearing is not one of them.

Whether the vaccine will work for me or you depends on biology. I have no control over that. Either my body responds to the vaccine or it doesn't. The mask, on the other hand, works on everyone who wears one. That's what the CDC Director is saying, and he's right.

***

Now that I've gotten your attention, let's switch to a proper interpretation of vaccine effectiveness. We have a coronavirus vaccine that demonstrates 50% effectiveness in a clinical trial. One possible scenario is it reduces the infection rate from 1% to 0.5%. (For reference, in the Moderna trial, they assume the baseline infection rate with no vaccine is 0.75% over a six-month period.) That's a 50% drop, and statistically significant.

For every 1000 patients receiving the vaccine shot, five patients will benefit from it as they would have caught Covid-19 without the vaccine. Another five will get sick despite being vaccinated. So C is 5/1000 = 0.5 percent for the vaccine. Getting vaccinated is like buying a lottery ticket with 5 percent chance of winning.

In medical jardon, we say the number needed to treat is 1000/5 = 200. For one person to benefit from vaccination, we must administer 200 vaccine shots. Similar to TV advertising, we can't know who would benefit ahead of time. Statistics tell us that on average, one out of 200 will.

Notice that C isn't 50 percent but only 0.5 percent. It can't be 50 percent because most people won't catch Covid-19 even without the vaccine. For every 1000 people vaccinated, 995 of them won't get sick. Of those 995, 990 would have stayed healthy regardless of vaccination. The 50% effective vaccine really only benefits 0.5% of those who took the shot. (Note again that I'm using the assumption from the Moderna clinical trial design.)

But this is not the full story. The practical effectiveness of the vaccine depends also on what proportion of the population takes the shots. Let's pretend that half the population get vaccinated. Then the 0.5% benefit of vaccination is halved. The infection rate in the population is reduced from 1% to 0.75% since the infection rate for the unvaccinated will remain at 1%.

Herd immunity now complicates this picture. If enough people get the vaccine, typically 60% or higher, the amount of virus circulating in the population is suppressed enough that the infection rate for the unvaccinated will dip below 1% so collective action creates a positive externality.

 

 

 

Confusion over case trends and positivity ratios, an aftertaste of targeted testing

We are now experiencing the aftertaste of targeted (aka triage) testing, a policy that has unsavory consequences that were predictable (link). Targeted testing is the idea that only people with severe symptoms or at high risk of infection should get tested for the novel coronavirus. Compared to a broad-based testing plan (whether comprehensive or randomized), targeting economises the number of test kits. The basis of any targeting strategy is the skill to predict who is infected. The group that is targeted should have a higher infection rate than the group that is not targeted. It's important to bear in mind that targeting not only suppresses how many people are tested but also affects who gets tested.

One way in which targeted testing hurts us is that it ruins our datasets from which we obtain insights about the pandemic. This post is prompted by Ken's comment on my previous post on casual presumptions of randomness:

The number of daily new cases in some countries doubled as fast as every 2 days. This didn't mean a greater rate of infections, it was simply a result of catching up all the non-tested cases.

The connection between infections, tests, and cases is complicated, and the media shift back and forth between the rate of cases (positive test findings) and the positivity ratio (proportion of tests that came back positive). I spent some time explicating these relationships, and hope that this post will succeed in helping you gain some intuition around those concepts.

We start by decomposing the positive test findings. In the flow chart below, starting with the dark gray box on the left side, we examine a fixed population of people. At any given time, the people are divided into three "compartments," the infectious people who currently have the virus, the susceptibles who have not been infected, and the recovered (which by convention, includes those who died from Covid-19).

Also by convention, anyone can be infected only once, and we thus ignore recovered people from further consideration. From the dark gray box emerge two paths that terminate at positive tests: the upper path consists of infectious people while the lower path, susceptibles. Some portion of the population are infectious; within that, some proportion have been tested for Covid-19, and within that, some proportion have tested positive. This last tally is one part of the confirmed cases. The other part of the confirmed cases comes from susceptibles, represented by the lower path.

The number of confirmed cases is the number of positive tests. This is a key metric on all Covid-19 dashboards. The other important statistic is the positivity ratio. This is the number of positive tests (represented by the dark red box on the right) divided by the number of tests conducted (the sum of the two yellow boxes).

The next diagram describes the drivers of the two dashboard statistics:

• the weights of infectious or susceptible compartments in the population (excluding recovered)
• the probability of getting tested among the infectious or susceptible group
• the weights of infectious or susceptible people within the tested subpopulation
• the positivity ratio for the infectious or susceptible people who got tested (call these the group-level positivity ratios to distinguish from the overall positivity ratio)

The overall positivity ratio is affected by three signals. First up is the rate of infection in the population, reflected in the relative weights of the I and S compartments. Second, the testing regime imposes a selection effect on these weights, producing the second set of weights. The notorious triage or targeted testing protocol restricts testing to patients showing severe symptoms or at highest risk - these disproportionately arise from the infectious group, and so the second set of weights are more skewed toward infectious people, compared to the first set of weights. Thirdly, test accuracy affects the number of positive test results, given any tested subpopulation.

Test accuracy is typically measured by sensitivity and specificity. On the upper path, the positivity ratio of the infectious-tested group is just the sensitivity, which we assume to be 90 percent. On the lower path, the positivity ratio for the susceptibles-tested is the rate of false positives (as this group is not infected), which is 1 minus the specificity, which we assume to be 5 percent. (For the sake of this post, we assume that the performance of the diagnostic tests has not changed during the pandemic and thus the group-level positivity ratios are fixed. This simplifies the following examples.)

 

What's special about 5% positivity ratio?

The media keep reminding us that the CDC wants the positivity ratio to fall below 5%. What is the significance of the 5-percent threshold? I can explain using the flow chart.

We hope for the epidemic to dissipate. When this happens, the infectious compartment dwindles to zero while the susceptibles stabilize. This means there will be no more flow through the upper path. All of the positive test results will come from the lower path (and are false positives).

The overall positivity ratio is a weighted average of the group-level positivity levels, the weights being the proportions of those tested who are infectious or susceptible. In this extreme case, the weight is entirely skewed towards the susceptible-tested, for which the positivity ratio is 5 percent. Thus, the overall positivity ratio will be 5 percent.

In other words, when the positivity ratio dips to 5 percent or below, new infections should be dying out.

 

What's the relationship between rate of cases and rate of infections?

I will now use this setup to investigate Ken's comment. He was reacting to the naive interpretation of case growth as infection growth, popularized by the media. Such a conclusion is invalidated because of targeted testing.

Note that infection growth cannot be directly observed. By contrast, case growth is reported on every dashboard. In Ken's scenario, the number of cases (i.e. positive test results) are growing rapidly.

Start from the dark red box on the right of the flow chart, and work backwards. The jump in positive cases may come from either of the two paths.

First, consider the lower path representing susceptibles. During the course of an epidemic, more people get infected, which reduces the number of susceptibles in the population, which in turn decreases the number of susceptibles testing positive. So if the cases from this lower path have to go up, then it must be that susceptibles are suddenly more likely to get tested.

One such scenario is if politicians expand targeted testing to those with mild or no symptoms. This is Ken's hypothesis.

Such a policy shift also affects the upper path. In fact, infectious people are more likely to experience mild symptoms. Even if the rate of infection is stable, the number of positives tests arising from the upper path grows. If infections are also accelerating, both factors contribute to case growth.

Ken argues that the entire case growth can be attributed to the expanded testing capacity while the media generally interprets the case growth as infection growth, ignoring possible shifts in testing policy.

The trouble is that both factors are at play, and we don't know the relative sizes of their contributions. This confounding is brought to you by targeted testing! (I can't resist: it's free and comes with a hidden cost.) This example demonstrates the aftertaste of targeted testing. It messes up the data so we can't interpret them properly.

It doesn't appear that Ken's scenario is happening in the U.S. as the federal government has continued its misguided targeted testing policy. Some politicians even grumble that we are conducting "too many" tests. (This can only happen if we were conducting too few tests, and thus under-reporting cases early on.)

 

How does targeted testing affect the positivity ratio?

In Ken's scenario, he assumes that the increase in testing accounts for the entire jump in cases. So, let's stipulate that the rate of infections is stable, and look at how the testing policy affects the positivity ratio.

The overall positivity ratio is the weighted average of group-level positivity ratios, so the key is to understand what happens to the second set of weights, which describes the mix of infectious and susceptibles in the tested subpopulation.

Under a randomized protocol, the second set of weights just represents the underlying mix of infectious and susceptibles since both groups have equal probability of getting tested. Under a targeted protocol, the second set of weights is biased toward infectious who are more likely to experience symptoms. Thus, early on, the positivity rate is over-stated. If targeting restriction is relaxed, the bias toward infectious is reduced. Thus, the relative weight of susceptibles-tested is pushed up, and so the positivity rate is driven down.

Since the rate of infections is assumed stable, the shift in positivity rate tells us only about the testing policy. We've just discovered how politics enter the data.

***

Targeted testing introduces bias into the data, which pollutes the link between the rate of cases and the rate of infections. Remember: we use the rate of cases to infer the rate of infections because in the real world, no one knows the rate of infections.

Targeted testing is the aftertaste that you can't seem to get rid of. It polluted the data when first implemented. Then, every subsequent change in policy, and the gradual behavioral adaptation to these changes, add to the confusion. The effect on number of tests is observable but the effect on the types of people taking the test is not.

This tragedy on the national level is now being replicated on individual school campuses. Many schools are doing targeted testing only on students who self-report symptoms. Their dashboards are reporting confirmed cases, the trend of which is being interpreted as the trend of infection. What is the point of data-driven decision-making when the data are known to be bad?

 

 

 

 

 

 

 

 

 

I feel sad for those who had to roll out the methodology change announced by BLS for seasonal adjustment

Imagine running a McDonald's franchise. We need lots and lots of potatoes. Over a year, we order 2400 cases, an average of 200 per month. Each December, more people dine out and our sales jump. We order 300 cases, which is 50 percent more than an average month. (For simplicity, let's say February sales dip to 100 cases which gets us back to the monthly average.)

If we run through 300 cases of potatoes in December, that's a normal year. If we end up consuming 330 cases, that's 10% better than normal. 360 cases, that's a special year, 20% above normal. Conversely, if our diners only devour 250 cases in December, that's bad. It's bad despite the 25% month-on-month growth from 200 in November.

We should not directly compare month-to-month data. We want to break up the data into interpretable parts: current December sales = normal December sales x growth factor = normal monthly sales x seasonal factor for December x growth factor. Normal monthly sales is 200. December sales are typically 50% above normal monthly sales. This gets us to 300 cases. Since we ordered 330 cases this year, the implied growth factor is 1.10. We used 10% more potatoes than the typical December.

This calculation is known as a seasonal adjustment. The seasonally-adjusted December sales is 330/1.50 = 220 cases. This number is 10% above the normal monthly rate of 200.

***

Seasonal adjustment returns to the front pages of the business press last week when the Bureau of Labor Statistics made a consequential change to the methodology used to compute the seasonally-adjusted initial jobless claims.

The following chart shows the strong seasonal pattern of new jobless claims for the last decade. They peak in January every year. (2020 will be added later.)

A jump in January does not mean that the employment market has deteriorated. To know whether the job market was better or worse, we must first take out the seasonal factor.

The following chart shows the seasonal factors used for seasonal adjustment from 2010-2020. The line for 2020 is shown in orange.

The number shown on the vertical axis is our seasonal factor. For the first week of the year, the seasonal factor is 1.60, meaning that typically the U.S. experiences a 60% surge in initial jobless claims in the first week of the year compared to the average week.

The orange line for 2020 hugs the other lines until the most recent week. That's when BLS suddenly switched the methodology for seasonal adjustment. Instead of adjusting the claims number up by roughly 25 perent, the new methodology adjusts up by only about 5 percent.

***

The methodology change is from a multiplicative model to an additive model. Instead of measuring the seasonal factor as a percentage of average claims, the new methodology estimates the seasonal factor as an absolute number of claims.

The chart below shows the number of claims added or substracted due to seasonal adjustments using the multiplicative model. For the first week of the year, the adjustment is 1/1.60 = 0.63, meaning the adjusted claims number is 63% of the unadjusted number. The expunged 37% represent about 123,000 jobless claims, shown below.

The first part of the orange curve up to the present strays away from the gray lines because the same percentage of claims translates to a larger number of claims when the level of claims surged during the pandemic. (Recall the previous chart, which shows that the seasonal adjustments used in 2020 fall right in line with previous years when viewed as a percentage of claims.)

BLS published the scheduled adjustments for the rest of 2020 using the additive model. These are shown as the dashed line below:

The new metholodogy takes the absolute numbers of adjusted claims from past years and use those numbers to add or subtract. The reasoning behind this methodology switch is muddled. By August, the level of initial jobless claims in the U.S. is still five times higher than the typical amount in the last years. Why should the seasonal adjustments not scale with the level of claims?

Given the run-up to the November elections, an obvious explanation is a political decision. Notice that between September and November, the typical seasonal adjustments add to the raw counts (see the first chart to confirm); that's because initial jobless claims are usually lower than average in those months. By switching to the additive model, the size of these up-adjustments is smaller than required by the multiplicative model.

***

Back to our McDonald's restaurant. Imagine that our business is doing very well, and we expanded to occupy the next-door storefront, doubling our sales. We know from the past that December potato orders are typically 50 percent above an average month. That's the multiplicative model. Our new average monthly order has doubled to 400 cases. December continues to be the busiest month of the year, and we should expect orders to expand to 600 cases, using the 1.50 seasonal factor as before.

The BLS is throwing away the multiplicative model. In the additive model, they'd say in December, we order 100 more cases than the historically monthly average (200). So, our estimate for December after expansion is 200+100 = 300 cases. The absurdity of this is clear. Three hundred cases is below the new average order of 400 cases! The problem is that the additive model does not scale the adjustment with the level of purchases.

I feel sad that someone at BLS has to justify the methodology switch.

 

What to make of the vaccine pledge

Investors have schedules: they want to make a return of at least x percent within y years. Science does not follow schedules. That's why vaccine manufacturers got into trouble when the U.S. government keeps seting deadlines for when a coronavirus vaccine will come to market. The anticipated arrival date of late October defies belief. It implies that the U.S. government is poised to approve a vaccine that has been observed for basically one month. (Anyone who joined the Moderna trial at the beginning is just getting the second dose right now.)

This week, we learned that the top pharmas signed a "pledge" to assure the public (link). "We believe this pledge will help ensure public confidence in the rigorous scientific and regulatory process by which COVID-19 vaccines are evaluated and may ultimately be approved."

But those are empty words as they have not committed to any specific action. For example, the information submitted to ClinicalTrials.gov by Moderna, predicted as one of the first to cross the finish line, is as sparse today as it was in late July when I first looked at it (link). The document continues to omit the most crucial pieces of information needed to judge its scientific rigor.

First, the submitted documentation does not explain how the scientists will establish whether someone is infected. It appears that they are relying on self-reported symptoms, rather than diagnostic testing, to determine infection. I say this based on what I read at Moderna's recruiting website (link), noting that this is a marketing website, not a scientific website. If true, that's not what one would describe as rigorous science.

Second, it does not state when the scientists plan on conducting analyses. If they analyze the data after one month, the best they can say is that the vaccine provides immunity for one month (and only for some proportion of people, as is the case with any medicine).

For this pledge to have any value, the pharma CEOs must commit to releasing more details of the clinical trials. What I listed above are not fancy requirements; they are rudimentary and fundamental to the practice of pre-registering clinical trials. It is shocking that those documents omit the information; it is insulting after signing a pledge to use rigorous science.

***

See this previous post to learn about the dangers of not pre-registering the analysis plan, and selecting some short observation period based on initial outcomes.

PS. The Oxford/AstraZeneca trial is catching attention as they have paused due to a severe side effect. I must say that the design document they submitted to ClinicalTrials.gov addresses the two questions listed above. They plan on doing an analysis after 6 months, and they will confirm infections using a PCR diagnostic test (although it sounds like they will count asymptomatic infections as not infections - the devil is always in the details).