« Real-world studies: how to interpret them | Main | The confusing pictures in Israel and in the Israel study »


Feed You can follow this conversation by subscribing to the comment feed for this post.

Dave W

Hi Kaiser,

As you say the clarity even if blurred on patient days and follow up is not as blurred as in some studies. Would prefer if the daa were presented in all in a much clearer way. For example some present no breakouts of at risk days by segments or swap between patients and days.

They did not include testing data or occupation in their analysis.

Additionally since the study is retrospective and confined I believe to Mayo datasets the reason for existence in the database might be a concern if there are differences between vaccinated and control. As an example vaccinated might be there because they are vaccinated, control because they go for a test or for some other reason that increased test likelihood in the control cohort.

This "Mayo effect" might also affect propensity natching on comorbitities and the hospitalisation data since Mayo specialises in certain areas.

So if I were looking into study would like to see testing data in sampled areas and hospitalisation rates.


DW: Agree that they can disclose more on the zip-code matching. It would be interesting to know how many zip codes were excluded due to not enough vaccinations. Also, I put here as a footnote that they omitted the follow-up distribution for the unvaccinated group which can be computed based on the matched Day 0. There is even a table that summarizes the distribution but on the unvaccinated column, it was left blank. Nonetheless, I featured the Mayo study first because there are fewer missing pieces than some of the other studies that I'm also looking at.
I suspect that their matching variables are constrained by missing data. The standard PS model uses logistic regression which doesn't handle missing data well so occupation would present challenges.
Your concerns about the Mayo bias are legitimate but remember they are not generalizing the finding outside the analysis set during the analysis period.

Dave W

Agreed. Some concerns in all studies might be addressed by tighter descriptions and more complete presentation. Missing data, that is likely meaningful should, I think, be addressed
narrative at minimum and ideally with a note about degree of spanner in worksness and, or, projectability. The concern Iess about peers (though even with some of these) than the filtering through to" TV presenters".

In the hospitalisation section I think the potential bias deserves more attention on their part, because they have overhangs in 1-10 matching that might impact the admission
outcomes in control. Again could be clarified with data they likely have.


DW: Oh, this prompts another footnote. I have exclusively focused on the main analysis of this study, which is about the VE. There is a second strand of work relating to hospitalizations, which I have no plans to discuss at this point. I may come back to it.


Would be interesting to see the equivalent data from Figure S2(a) plotted against the Pfizer's cumulative case curve. I'm picturing essentially a survival curve starting at 100% and decaying to essentially zero as we get to the right side of the chart.

Would illustrate just how many of the participants are represented at each point along the curve.

The comments to this entry are closed.

Get new posts by email:
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR, Wired.

See my Youtube and Flickr.


  • only in Big Data
Numbers Rule Your World:
Amazon - Barnes&Noble

Amazon - Barnes&Noble

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee

Next Events

Jan: 10 NYPL Data Science Careers Talk, New York, NY

Past Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Principal Analytics Prep

Link to Principal Analytics Prep