Do better

The Covid-19 pandemic unleashed a torrent of medical research, but the quality of the presentation is frequently unimpressive.

In this post, I cover an example typical of the problems I've encountered when reading Covid-19 vaccine studies. In the "final analysis" of the J&J vaccine (published here in a NEJM peer-reviewed paper), the authors made the following unambiguous claim:

The Kaplan-Meier cumulative incidence curves for moderate to severe-critical Covid-19 separated after 14 days (Fig. 2A)

What should readers expect to find in Figure 2A? Let's guess: evidence that the cumulative incidence curves (that is, of vaccine and placebo groups) separated at 14 days after the J&J shot was administered. The claim was precise and unambiguous: the separation occurred on the 14th day, not 13th or 10th or 15th or 21st.

Here is all of Figure 2A:

In a flash, the chart might be seen as confirming the finding. But...

... since we are sophisticated consumers of data graphics, we should notice

1. the chart is much shorter than it is wide: the cumulative incidence rate is only about 0.5% in the first 14 days, while the vertical axis extends to 12.5% (the median participant in the trial has only reached 120 days at the time of the analysis)
2. the dots have a diameter of 20 points, which is equivalent to 0.6% in the vertical scale: these dots are so big that one can fit only four such dots inside each vertical tick interval
3. it appears that 21 dots are squeezed between tick marks on the time axis, causing massive overlapping
4. the blue dots are printed on top of the red dots

As shown below, in the first 14 days, the chart crams 14 dots each of which has a diameter equivalent to 0.6%, while the entire 14 values span a total range of 0.5%. It is simply impossible to tell when the two curves "separate". It is highly unlikely that separation occurred on the 14th day as asserted.

The density of these dots robs their presence of any meaning. There is in fact a line through the centers of those dots and if the chart presents only those two lines, we'd be able to confirm whether or not separation occurred on the 14th day.

***

Found right below, Figure 2B directly contradicts the claim.

Figure 2B plots the estimate of vaccine efficacy (VE), which is a function of the relative cumulative incidence rates at any given time t. As I explained in this post, 1-VE is essentially equivalent to the height of the cumulative value of the vaccine group divided by the height of the cumulative value of the placebo group.

For example, on Day 15, VE is shown at about 50%. That means 1-VE is 50%, which means the height of the placebo group's value on Day 15 is roughly double that of the vaccine group on the same day. If on Day 14, the blue dot was at 0.5% as eyeballed above, then the red dot would have been at around 0.25%. Thus, the curves have already diverged by day 14, and did not just "separate after 14 days".

Figure 2B also shows VE around day 10 to be 30%. Thus, 1-VE = 70% meaning the vaccine group's cumulative value by Day 10 is 70% that of the placebo group on that day. So we know a gap already existed by the 10th day.

***

In order to establish Day 14 as the watershed, one would ideally look at cumulative case data on Day 14. Thankfully, raw data are provided underneath Figure 2A. But wait... the researchers chose to show raw data in 21-day increments, making it impossible to directly verify their claim. (Meanwhile, they chose to print huge dots on the chart for each day, obscuring the underlying values.)

Further, for Figure 2B, the researchers did not stick to the 21-day increments. Instead, they provide data in 30-day increments. Notice that no tick values coincide on Figure 2A and Figure 2B until day 210 (7 months after trial start date). Thus, it is challenging to correlate these two diagrams even though Figure 2B presents results from a model presumably built from the raw data that underlie Figure 2A.

Why is day 210 significant? As with other vaccine trials, the FDA allowed the pharmas to vaccinate the placebo group, thus dismantling the trial. In J&J's trial, this started happening on March 10, 2021, about 170 days after the start of the trial. Any data beyond 170 days are essentially worthless. (Ultimately, almost every placebo participant took the vaccine.)

 

 

Comments

[Anonny]

Maybe the "separation" refers to the overlap in the Kaplan-Meier confidence intervals, rather than the point estimate curve. It's still impossible to verify but it might explain the discrepancy.