Tantalizing unreliable new data on half doses and single doses

A reader sent me an interim analysis of a new Phase 2 trial of the Moderna vaccine in which they tested whether the dosage could be cut by half. An economist pointed to this result as a big win, as it could double the population of vaccinations while retaining two doses, and the same supply of vaccines.

This paper deserves more coverage.

The trial had three arms, one given the treatment of two "full" doses - the same as in trial that resulted in Moderna's emergency use authorization, one given two "half" doses, and the third arm given placebo (saline) shots.

The key result is shown in the following graph from the paper:

Each arm was sub-divided into two age groups (18-54, and 55+). As with the Phase 3 trial, Moderna relied on infrequent follow-up visits and self-reported symptoms. The chart showed the three follow-up moments, once at "Day 29" (i.e. during the appointment to receive the second dose), once at "Day 43," roughly 2 weeks after the second dose, and once at "Day 57", about 1 month after the second dose.

The chart uses a log2 scale (you can expect a Junkcharts post on this tactic soon). The metric for this Phase 2 trial is not cases but antibody levels. The analysis utilizes two controls. The placebo places the "zero" level while the "Conv" level measures the average amount of antibodies found in the blood of those who are known to have been infected with SARS-CoV-2 ("convalescent"). A viable vaccine should induce production of antibodies well above the level obtained from natural infections.

The twitter users used this paper to push the case for halving the dosage for the Moderna vaccine. They are comparing the gray columns to the red columns, declaring them "similar enough". The evidence is not as strong as it seems. The log2 scale squishes the large values while spreading out the small values. If you look at the numbers on those columns, the red columns are definitely taller than the gray bars. Take the 3rd column for 18-54, half dose, and the 3rd column for 18-54, full dose. The values are 189 and 239, the latter being 26% higher than the former.

Nevertheless, the level of antibodies from natural infections is only 48 so both those values may be sufficient to fight the virus. (Note, as the paper's authors admit, no one at this stage knows the minimum level of required antibodies. The connection between antibody levels and vaccine efficacy has not been established.)

***

The paper has another significant finding, which is relevant to the hot-again debate on whether a single dose is sufficient. Recall that Moderna administers the second dose around Day 28. The chart above shows a huge jump in amount of antibodies from Day 29 to Day 43. That appears to be the effect of the booster shot (possibly with an assist from the first dose). Pointedly, this study also did not include a single-dose arm.

Further, the level of antibodies on Day 29 is way below the "Conv" level. Don't be fooled by the log2 scale. The 55+ age group had antibody levels of 18, less than 40% of the "Conv" level. That spiked to well above the "Conv" level after the second shot.

The authors also compared the last two columns of each experimental group. This shows the change in antibody levels in the 3rd and 4th weeks after the second shot. I'm not sure how this finding of a shorter observation period of an indirect metric could possibly alter the finding of the Phase 3 trial which measured confirmed cases for a median follow-up of almost two months after the second shot. We already know that the vaccine's effect holds for roughly two months.

***

Unfortunately, the level of scientific discourse in the pandemic era has sunk like an egg in water. The following is still true about a study like this:

• It is a Phase 2 study
  
  
• This is an interim (early) analysis, meaning the study population has been followed up to one month after the second shot
  
  
• There were only 100 people in each of those groups, and a total of 600 people in the entire experiment. This is 50x smaller than the 30,000 people in the Phase 3 trial.
  
  
• The sample size is so small that the researchers did not perform statistical analysis - which is a way of saying we can't generalize this result to the larger population. (That's why I wrote about the "law of small numbers" last week.)
  
  
• Another obstacle to generalizing this result is that when you only have 600 people, you have to narrow the criteria for enrollment. The sampling error is a function of sample size, and the inherent variability of the participants. If the participants have to look more alike, they are going to look more dissimilar as a group compared to the general population.
  
  
• Lastly, I will let the authors of the paper do the talking. They described the following limitations of their experiment: "The study population was not designed at the time to be representative of those at risk for SARS-CoV-2 infection or COVID-19. ... the present study was not designed for statistical comparison of superiority or equivalence between doses of [the Moderna vaccine] ... Although antibody responses were generated ... that, on average, exceeded those of convalescent sera, at the present time, a serological correlate of protection against SARS-CoV-2 remains to be determined."

These descriptions of limitations always give me a chuckle, just like reading the "risk factors" section of business prospectuses, after which you wonder how any investors could possibly open their wallets.

After concluding that the half doses and the full doses both produced good antibody responses, the researchers say their study wasn't intended to draw such a conclusion (neither of superiority nor of equivalence). In addition, the connection between level of antibodies and propensity to get infected has not been established. What the study does show clearly is that the level of antibodies spiked about 3 times after the second shots.

 

P.S. [2/23/2021] Post about the graphical design of the chart above is now up on Junk Charts. Find it here.

Comments

[Bryan F]

Hi Kaiser
Noticed a few common things in VAX trials reports.
No geogogphic or occupational data provided.
Infections appear to grew in a very smooth manner.
Testing data unclear.
Only one control for 'guessing' placebo
Low seroprevalence and +ves in screenings.

Any insights?

[Kaiser]

BF: The list of issues is very long, as you pointed out. This shines a light on an industry that has never been all that transparent. The press for time during the pandemic has further lowered the bar. (For example, I have pages and pages of questions about the Remdesivir study, and I never ended up putting a post up on that because each time I tried, the list grew longer!)
For vaccine studies, I think the most significant issues are (a) infrequent testing to proactively identify cases (b) lack of transparency around "adjudications" of cases (c) lack of information on ITT (Intent to treat) analyses (d) failure to realize that an interim analysis coupled with sequential enrollment by subgroups present a different analysis context from a full analysis (e) lack of updates after the initial analysis (f) science by press releases (g) glaring lack of discussion among scientists around major hiccups such as the low dose "accident" of Astrazeneca-Oxford vaccine, and the Moderna dataset being way below the required two-month follow-up period (h) the definition of a case in terms of the symptoms list and testing regime (i) the definition of the case-counting window, which I've written a lot about (j) ignoring all confidence intervals (most of which are laughably wide) (k) the silence from the investigators surrounding post-hoc analyses published by associates or foes ....
All of these must be seen in the context of the culture of this science. The situation is similar to the on-going battle Andrew Gelman and associates are waging over the psychology research establishment - many if not all of the above practices appear to be accepted, while outsiders like myself are raising concerns about what we perceive to be non-ignorable issues.

[S Kefteridis]

I am tempetd to speculate that some of this disjointedness is due to a change / rush to efficacy from safety. Aside from the issues about doing science you have raised, the question then is what are the consequences direct and indirect of potential innaccurate or massaged outputs?

Kudos BTW for your excellent posts!