Open questions of the pandemic, part 2

Today, I bring up two other open questions, as we enter the third year of the Covid-19 pandemic. The previous set of three questions was featured in this blog.

 

4. How many lives did the vaccines save?

How high is the vaccine effectiveness? Not as high as claimed by the pharmaceuticals is the safe response here. But what I really want to discuss is the persistent misinterpretation of vaccine effectiveness (VE) by public officials and the media.

Let's start with an analogy to a direct marketing analytics question on which I spent a good chunk of my career: what is the effectiveness of online marketing spend?

A business spends $x buying Google (or Facebook) ads. During the month in which these ads are running, the business generates sales of $y. What is the benefit of the $x ad buy? The marketing manager maintains it's $y. Now, if the business had not bought those ads, how much sales would it have gotten that month? Would it have been $0?

In the same manner that the marketing manager (or Google or Facebook accounts team) claims credit for the entire $y sales, many people are making arguments about vaccine effectiveness that is entirely exaggerated.

This problem started the first day that VE became a household term in late 2020/early 2021. A VE of 90% does not mean 90% of those vaccinated will be protected while 10% are unprotected despite being vaccinated. The VE is not a proportion but a ratio of two case rates. VE of 90% means the case rate of those vaccinated is 10% (one-tenth) of the case rate of those receiving placebo during the clinical trials. (See also this post and this post.)

Interpreting VE as a proportion makes the laughable assumption that everyone would have been infected if they did not get vaccinated. In reality, only 7% of the placebo participants in the Pfizer vaccine trial were counted as infected 6 months after Dose 1 (even lower if we discount cases occurring before 7 days after the second shot, as the investigators did). Look at the Pfizer case curve again and pay attention to the values of the y-axis.

From the chart, you see that 93% of the participants in Pfizer's trial who were given placebo shots did not catch Covid-19. If the vaccine were completely ineffective, we expect 93% of the vaccinated would also not catch Covid-19. The placebo arm's case rate was 7%, and the vaccine arm's case rate was something like 0.7%. Out of 1,000 people, 70 is expected to get sick by Day 180 if given placebo while only 7 would get sick if given the vaccine. The vaccine was proven to protect the 63 people out of 1,000, but out of 993 vaccinated people who did not get Covid-19, 930 would not have gotten it even if they took the placebo shots. The point of having a placebo group in a clinical trial is to differentiate between the 930 and the 63.

The claim that all 993 are protected by the vaccine does not align with a 90% VE. It actually suggests a quasi-perfect VE exceeding 99%! In order for all 993 to be protected by the vaccine, the case rate on the placebo arm must be 100%: anyone who gets the placebo catches Covid-19 by Day 180. The ratio of case rates is now 0.7% vs 100%, and the VE is 1 - (0.007/1) = 99.3%.

The situation gets more extreme with hospitalizations or deaths. Fewer than 1% of the placebo participants in the Pfizer trials had severe Covid-19 or died. In fact, those trials were underpowered to make any statistically reliable statements about hospitalizations or deaths. The vast majority of the people who got mild cases of Covid-19 would not have gotten a severe case or have died - even if they were given placebo shots instead of the vaccine.

But... they keep telling us that almost everyone in the hospitals is unvaccinated. What doesn't follow is that anyone who is vaccinated but not hospitalized has been protected by the vaccine. Very few of these people would have been hospitalized had they been given placebo shots.

The pertinent concept here is Numbers Needed to Treat (NNT). This is well studied for many classes of medicines. NNT of 30 means you have to treat 30 people for 1 to benefit. In other words, 29 out of 30 who took the same treatment are not expected to benefit, i.e. their outcomes would not be statistically different had they not taken the medicine. Taking any drug is like a buying a lottery ticket; we are buying a chance of the drug changing the course of the disease. The NNT is one way to describe the chance of benefiting from the drug. If NNT is high, most people taking the treatment do not benefit from it. (Of course, someone who chooses not to take the treatment is guaranteed not to benefit from it.)

I previously computed the NNT for Covid-19 vaccines here.

 

5. How credible are the studies that are pouring out torrentially?

The White House recently tweeted the following chart to show what is known as the "pandemic of the unvaccinated":

Pointedly, no data has been released. The information on the above chart cannot be interpreted unless the CDC comes out to answer the following questions:

• What does "fully vaccinated" and "unvaccinated" mean?
• Does this chart include all Covid-19-associated hospitalizations? If yes, are "partially vaccinated" or "two doses not yet 14 days" counted as "unvaccinated"? If no, we need to see the line for the disappeared.
• Have hospital admission policies changed over this period of time? In particular, are vaccinated and unvaccinated people treated differently?
• What does "age-adjusted" actually mean? Age adjustment requires an underlying assumption of the age distributions. It's either imposing the same age distribution on both vaccinated and unvaccinated segments, or it's imposing one segment's age distribution on the other segment. If the former, what is the reference age distribution being used for this calculation? I explain this issue in the Prologue of Numbersense (link) as it relates to resolving the Simpson's paradox.

The reason full disclosure is needed is because the numbers don't gel.

First look at the only two numbers that are found on the above chart. They say that the rate of hospitalizations was 67.8 among unvaccinated persons and 3.9 among fully vaccinated during the week ending November 27, 2021. In the following chart, I mark that date out (on the far right), plus two other dates just for illustration.

How many total hospitalizations are counted for the week ending November 27? For this, I grabbed data from OurWorldInData.org. First, the correct answer should be about 44K.

(I let the right side of the chart stretch to last week so you can see the spike in hospitalizations happening right now in the U.S. See my previous post for some context.)

OurWorldinData.org also gives me the number of fully vaccinated Americans at those three times.

I also pulled the number of unvaccinated Americans (zero doses) from the site. So now, I can multiply the hospitalization rate (per 100K) by the vaccination group (in millions, or tens of 100Ks). For example, for November 27, the calculation is 67.8*1000 + 3.9*1970 = 75,600 hospitalizations.

Yikes! The total number of hospitalizations represented on the White House chart for that week is 70% higher than the OurWorldinData.org number.

What about the other two dates? Oddly, the discrepancy swung the other way. Those numbers on the White House chart are 20-30% lower than the respective OurWorldinData.org numbers.

This is just another example of what's wrong with "science" in the Covid-19 era. The right way to present this information is to supply a spreadsheet showing all the numbers shown on the chart, plus a formula that links the aggregate hospitalization numbers to the disaggregated.

One reason why the numbers don't gel is "age adjustment". Other reasons may be exclusions and other quirks of the analytical methodology, none of which have been disclosed. But age adjustment is not some magic dust. The adjustment is on top of observed data - it's not complicated to show unadjusted and adjusted numbers side by side. Using adjustments makes it even more important to reveal the underlying methods.

Readers may notice that I haven't commented on many of these recent studies. It's because they share this same problem. The disclosed information does not align with observed data, which suggests the use of exclusions, adjustments and other tools of the trade. In some cases, the disclosure is so poor that you can't even check if the numbers align! These analysts are daring readers to trust but not verify. If the analysts don't want to show their work, do they have something to hide?

***

I didn't expect to hit this post length so soon. There are other questions arising from the White House chart. Come back for Part 3.

Comments

[Dan]

Any thoughts on this analysis of COVID vaccine benefit?
https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2787935

[Kaiser]

Dan: A quick glance of that paper seems like it says nothing about vaccine effectiveness. VE is an input to the simulation. They then create a no-vaccination scenario, so all they have done is to reflect the presumed VE. "Vaccine efficacies against infection, symptomatic disease and severe disease after each dose and for each variant were derived from published estimates."

Also, on my quick look, I did not see how they configured the no-vaccination scenario. How would they know the counterfactual rates of cases, hospitalizations, and deaths were there no vaccine?

[Antonio]

Here in Italy the third dose is mandatory for all people because it has been said that VE drops to 50% within the fourth month.
Do you know how VE is computed?
That is, VE at fourth month is calculated comparing vaccinated-in-4-months people to non vaccinated people or just-vaccinated people? I ask because the former and the latter alternative yield two completely different interpretations of the situation...

[Kaiser]

Antonio: The answer is "trust us". I was looking in the much ballyhooed UK report, and found that they just printed the final VE number; there is not any data to be found to learn how they got there. They use a "test negative" design which means that they are not comparing ratio of case rates but ratio of vaccinations. Long story: will write about that design in the future. You can't even do a simple check like I did above because the Supplementary File that provides data behind each chart skipped over all the charts showing VE. Interpret that however you wish.