The other day, I posted some notes about the JAMA paper featuring the interim analysis of the Sinopharm vaccine trial, a Covid-19 vaccine developed in China using the tried-and-true inactivated virus technology. The trials were conducted in UAE and Bahrain and the follow-up period for the median participant has reached 3 months since the second dose, which is a longer follow-up period than the earlier trials.

One takeaway from Part 1 and also my previous reviews of vaccine trial results is that vaccine efficacy (VE) numbers are not easily comparable. Trials were conducted in different countries at different times; definitions of cases were not standardized; case-counting windows varied. The duration of follow-up also matters because we expect the effect of any vaccine to wane over time.

***

In Part 2 of my review, I explore eFigure 1 in Supplement 3 of the JAMA report. This extremely important diagram traces how "cases" are qualified to be counted in the VE metric. All vaccine trials - not just Sinopharm - count only **adjudicated**, lab-confirmed, symptomatic Covid-19 cases.

The final VE metric in the Sinopharm trial included only 142 cases across 2 treatment arms and 1 placebo arm. e eFigure1 (partly shown on the right) revealed that there were 962 suspected Covid-19 cases (with symptoms) at the start of the process. That number was cut by 50-60% to 420 based on PCR testing negative.

The adjudicators then used their judgement to remove another 30-50% of the symptomatic, PCR-positive cases. We have 255 cases standing at this point.

Then the case counting window is applied, and another 113 cases vanished from the count. That leaves 142 cases that ultimately entered the VE calculation, resulting in the headline number of 70-80%.

For the sake of argument, if we got rid of the adjudication step, then the VE would have been reported as 37-56%. Because this calculation includes cases from first dose, compare it to 50-66% that was mentioned in the previous post. Thus, adjudication increased VE by roughly 10%.

Adjudication is not a unique feature of the Chinese trial. All vaccine trials employed adjudicators. The other pharmas have been far less forthcoming about what happened during this process. Pfizer, for example, disclosed to the FDA that they had 3,410 suspected cases (link) but I have not found any details on how that number got winnowed down to 170 cases actually counted in the headline 95% VE number.

***

If you have the stomach for it, I am about to open another can of worms. This happens to be a very material issue common in many settings. When comparing two numbers, one can compute a ratio (sometimes called an index) or a difference. Ratios are very tricky to deal with, and easy to misinterpret. The VE is a ratio of case rates (think of it as a relative risk index).

In the Chinese trial, the headline VE was reported as 73-78%, based on 142 adjudicated cases of Covid-19 across the three arms. In one of the sidebar analyses, the scientists added "asymptomatic" cases, defined as cases found by PCR tests for which the participants did not report experiencing qualifying symptoms.

The following table shows the data disclosed in the report. You can see that the vaccines are clearly not as effective at reducing "asymptomatic" cases, as the differences between 16, 10, and 21 are not statistically significant.

In order to simplify the calculations, I will work with the following idealized example. I'm assuming 10,000 people in two arms. The vaccine arm counts 35 cases of which 20 are symptomatic. The placebo arm has 115 cases of which 100 are symptomatic. We can compute the vaccine's share of total cases to be 23% so that the corresponding VE is 70%. (In this calculation, I include asymptomatic cases.)

An alternative efficacy metric is to compute the difference in the case rates. This means 1.15% (115/10000) for placebo minus 0.35% (35/10000) for the vaccine, resulting in a difference of 0.80%. This number has a straightforward interpretation: for every 10,000 people vaccinated, 80 fewer get sick from Covid-19 relative to placebo.

What happens if we ignore asymptomatic cases from both arms as the pharmas did? The difference in case rates remain at 80 per 10,000 because we remove 15 from each side. This makes sense intuitively since the vaccine is assumed to be ineffective at reducing asymptomatic cases (15 vs 15). Therefore, whether we count symptomatic cases or all cases, the vaccine reduces the number of cases by 80 per 10,000 relative to placebo.

As a relative risk ratio, the VE actually increases from 70% to 80% after removing asymptomatic cases. Note though that the 70% reduction is off the total case rate while the 80% reduction is off the symptomatic case rate, which is a smaller number. This points to the challenge of interpreting VE as a ratio. All the different VE ratios based on different inclusion and exclusion criteria have different base rates and so they cannot be compared to each other! Eighty perent is not always better than 70 percent: 70 percent of a big number can be more impactful than 80 percent of a small number.

You can now see why the game of the vaccine trials is to restrict the VE calculation to regions where the vaccine performs well. (I'm using "regions" in an abstract sense which includes the geographical sense.) Such restrictions lead to a higher ratio reduction but the result is only relevant to those regions, usually where the base rate is lower. In other words, it's a conditional number, as I explained in the last post. It's a higher reduction off an often unintuitive, restricted base rate. The case-counting window is the same game as asymptomatics but with a different name.

***

Another frequently ignored issue in these vaccine trials is the weak blinding. The Chinese trial is actually double-blind whereas the Pfizer, Moderna, etc. are single blind (very commonly mis-reported as double-blind because medical reporters just assume all clinical trials are double-blind).

Blinding is a protective measure that prevents investigators or participants from behaving differently - conscious or not - because they know which arm individuals are in. When the placebo is saline solution, it's actually very hard to blind participants because those getting vaccines are much more likely to experience a variety of symptoms, such as numbness, headaches, mild fevers and so on.

As a result, a few designs use non-saline placebos. The Chinese trial used an adjuvant in the placebo. An adjuvant is an additive in the vaccine that helps it work better. It's unclear to me whether this placebo is preferred to saline. It depends on the chance that the adjuvant causes mild side effects similar to the vaccine (which is a good thing in terms of masking treatment status) and the chance that the adjuvant by itself could be beneficial (which makes the base rate higher and raises the bar for the vaccine).

The only other exception I know of was one of the Astrazeneca trials, in which the placebo was a different unrelated vaccine. Participants might be less able to guess which arm they belong to in that trial.

***

Finally, none of these trials is really serious about measuring asymptomatic cases. The only exception is again in one of the Astrazeneca trials: they actually asked participants to send in nasal swabs every week of the trial. In the typical vaccine trial, they only do PCR testing (a) if symptoms are self-reported and (b) at infrequent intervals, including before each shot, and then once a month or longer. With infrequent surveillance, only a fraction of the asymptomatic infections can be found.

## Recent Comments