Big Data, Plainly Spoken (aka Numbers Rule Your World)

Science, politics or business

I'm surprised the TV pundits have applauded the State Department's advisory against travel to Japan (link). That decision is not science-driven.

Let's look at the data:

Even with the so-called "surge" in cases in Japan, the case rate per million in Japan is half that in the U.S. as of yesterday. Paradoxically, the chance of an American getting sick in Japan is half than that of an American in the U.S.

Sitting in a country where we have allowed coronavirus to spread readily makes us lose perspective. The case rates in most of Asia are a fraction of what we experienced in the U.S. If we isolate Japan, there is indeed a "surge":

Case rate jumped five times in the last month, and may have topped. At this peak, the rate was still half that of the U.S. case rate - at which level our public health "experts" are declaring the pandemic all but over!

According to the "experts", those case rates mean nothing. The only thing that matters is vaccinations. The Japanese have only vaccinated 2% of their population. One wonders if this is about selling vaccines or getting infected?

***

Not long ago, CDC told us that the benefits outweigh the risks when it comes to blood clots and the J&J vaccine. At the time, they estimated that the risk of blood clots after taking the J&J vaccine was 7 per million. Roughly 30-50% of these cases led to death.

[Harlan pointed out below that the data required to make this point is not quite what I had so I've revised this section as below.]

One reason why the rate of vaccination in Japan (and in many other Asian countries) is low is that people have experienced a much lower risk of dying from Covid-19.

Japan's death rate has not breached 0.8 per million in the entire pandemic. In the U.S., we experienced periods of high death rates, up to 10 per million on some days. A total of 13,000 people have died from Covid-19 in Japan since the beginning of the pandemic while almost 600,000 have in the U.S. Over 40,000 people die from accidents in Japan every year (link).

The risk vs benefit trade-off is different for each country, and in fact, it is different for different age groups, ethnicities, genders, etc.

 

Instant Classic for Science Communications Students

When I blogged about Pfizer's adolescent trial (here), I was wondering when and if they will release the data or the scientific report to support those press releases (one by Pfizer, and one by the FDA). A report has just been published in the prestigious New England Journal of Medicine.

I predict this will become a staple of future science communications textbooks.

The 12-15 year old age group presents extreme difficulties for conducting clinical trial research - the case rate is expected to be extremely low, and there is a strong desire to fast track approvals. Those two issues are incompatible. You need a huge sample size to detect low rates of events, let alone differences between low rates of events. From the start, the FDA has allowed a head fake: the primary endpoint of the adolescent trials will be immunogenecity results, i.e. test of antibody levels in the vaccinated. This is no ordinary trial, as we will not be comparing adolescents to adolescents, vaccinated against placebo. We will be comparing adolescents to 16-25 year olds, vaccinated against vaccinated.

So what does the highly respected NEJM say about the 16-25 year olds, the designated "control" group? I searched for every mention of 16-25 year olds in the report. Below are some representative examples of the matching sentences (Scroll to the very end of the post to see the entire list of mentions).

• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.

You can read every word of the above sentences, or every word of the entire report, including every word in the footnotes, and you walk away thinking this adolescent trial included two age groups (12-15 and 16-25). What if this weren't true? What if the number of 16-25 year olds in this adolescent trial is exactly zero?

As I was reading the prestigious NEJM report, I was feeling a sense of dread, which worsened as page after page repudiated what I thought was true: that the 16-25 year olds data came from the previous adult trial. Indeed, one of the key issues I flagged in my previous blog post is that the analysis plan involved comparing one age group in one trial to another age group in a different trial. Did I mislead my readers? Was I so blinded by my own biases that I made up this detail?

It surely was not possible that scientists and peer reviewers and the editors of the prestigious NEJM would allow such a basic but important detail to be omitted. It has got to be my brain fog... am I losing grip of my sanity?

So I checked my notes... The very first line of the Pfizer press release from March 31 stated unmistakably: "In participants aged 12-15 years old, BNT162b2 demonstrated 100% efficacy and robust antibody responses, exceeding those reported in trial of vaccinated 16-25 year old participants in an earlier analysis, and was well tolerated." (their italics)

Next, I checked the FDA press release on May 10. The source of data for the 16-25 year olds has been wiped out. In the safety section, the FDA stated "The side effects in adolescents were consistent with those reported in clinical trial participants 16 years of age and older. " As if the adolescent trial contained participants 16 years of age and older. In the efficacy section, the FDA stated "The immune response to the vaccine in 190 participants, 12 through 15 years of age, was compared to the immune response of 170 participants, 16 through 25 years of age. " Unless you know otherwise, you're going to assume the trial contained participants of both age groups.

I warned you up top that you're witnessing an instant classic in science communications.

It takes discipline and skill to write an entire report that dances around a key detail. A true masterclass in deception. This is an instance of "story time," lulling readers in with clinical trial data but presenting conclusions from analyses that do not enjoy the benefits of an RCT. But "story time" trivializes what has happened here. I hate to use the F word. I'm not thinking the four-letter F word. The five-letter one. F R A U D.

***

Also of interest is the other side of the equation. How does one develop a nose for detecting fraud in data? I have to admit that it would have been very hard for anyone to notice the deception in this NEJM paper - that's why it belongs on the syllabuses of science communications courses.

There were just two tiny cracks. The first was when they disclosed that all 12-to-15-year-old participants came from the U.S., while 16-to-25-year-old participants were globally recruited. This would have been unusual to do in a single trial enrolling 12-25 years old, stratified into two age groups.

The second crack was in the way the waterfall chart was presented.

Notice the two age groups were presented in two separate panels. What we expect to see if the trial actually enrolled both age groups is something like this:

The waterfall starts with the entire set of enrollees and then there is a split by age groups. This chart came from the Phase 2 Moderna study of two half-doses that I discussed in this post.

As I said, these are subtle signs, easily missed.

***

Is it never allowed to pull a control group from a different place? Doing so does not automatically disqualify the analysis but such an analysis does not enjoy the exalted gold-standard status of a randomized controlled trial. Such workarounds must be scientifically and rigorously justified. Work must be presented to convince readers that any biases coming from different times, different places, different ages, different demographics, different anything have been measured and corrected for.

Here, the authors did not present any evidence about possible biases. They didn't even acknowledge the source of the control group data.

***

Listed below is every single mention of the 16-to-25-year-old control group in the Pfizer adolescent study in the NEJM scientific report. Each mention represents an opportunity for the researchers to inform readers that this control group came from a different trial.

• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was an immunogencity objective.
• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The immunogenicity objective was to show noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants.
• Noninferioirty was assessed among participants who had no evidence of previous SARS-CoV-2 infection with the use of the two-sided 95% confidence interval for the geometric mean ratio of SARS-CoV-2 50% neutralizing titers in 12-to-15-year-old participants as compared with 16-to-25-year-old participants 1 month after dose 2.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was assessed on the basis of the geometric mean ratio of SARS-CoV-2 50% neutralizing titers.
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Severe injection-site pain after any BNT162b2 dose was reported in 1.5% of 12-to-15-year-old participants and in 3.4% of 16-to-25-year-old participants.
• After BNT162b2 injection, severe headache and severe fatigue were reported in a lower percentage of 12-to-15-year-old participants than of 16-to-25-year-old participants.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• The use of antipyretic agents was slightly more frequent among BNT162b2 recipients who were 12 to 15 years of age than among those who were 16 to 25 years of age.
• [Footnote of Table 1]: Results are for the reactogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and a subset of participants in the 16-to-25-year-old cohort.
• Among the 16-to-25-year-old BNT162b2 recipients, 11% reported any adverse event and 6% had vaccine-related adverse events. Among BNT162b2 recipients, severe adverse events were reported in 0.6% of those who were 12 to 15 years of age and in 1.7% of those who were 16 to 25 years of age.
• One BNT162b2 recipient in the 16-to-25-year-old cohort discontinued vaccination because of severe vaccine-related injection site pain and headache, both of which were reported 1 day after dose 1 and resolved within 1day.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.
• Appendicitis was reported in 2 participants: 1 BNT162b2 recipient in the 16-to-25-year-old cohort and 1 placebo recipient in the 12-to-15-year-old cohort.
• The immune response to BNT162b2 in 12-to-15-year-old adolescents was noninferior to that observed in 16-to-25-year-old young adults.
• The geometric mean ratio of the BNT162b2 neutralizing geometric mean titer in 12-to-15-year-old participants to that in 16-to-25-year-old participants 1 month after dose 2 was 1.76.
• Among all participants regardless of serologic evidence of previous SARS-CoV-2 infection, the serum-neutralizing geometric mean titer 1 month after BNT162b2 dose 2 was 1283.0 in the 12-to-15-year-old cohort and 730.8 in the 16-to-25-year-old cohort.
• Substantial increases in the 50% neutralizing titer from baseline were observed, with GMFRs from baseline to 1 month after dose 2 of 118.3 among 12-to-15-year-old participants and 71.2 among 16-to-25-year-old participants.
• [Caption of Figure 2]: The results shown are for the reacogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and the subset of participants in the 16-to-25-year-old cohort who had electronic diary data available.
• [Footnote of Table 2]: The geometric mean ratio and two-sidded 95% confidence intervals were calculated by exponentiating the mean difference of the logarithms of the titers (the 12-to-15-year-old cohort minus the 16-to-25-year-old cohort) and the corresponding confidence intervals (based on the Student's t distribution).
• Antipyretic use after both dose was slightly higher in the 12-to-15-year-old cohort than in the 16-to-25-year-old cohort.
• All 12-to-15-year-old participants in this trial were enrolled at U.S. sites, whereas the 16-to-25-year-old participants were recruited globally.

If X causes Y, does it follow that X and Y are correlated?

Andrew points out an error in a new book authored by Dan Kahneman, Cass Sunstein, et. al. The authors said that while correlation does not imply causation, "wherever there is causality, there is correlation." In other words, if X causes Y, then X and Y must be correlated.

I like to say statistics is common sense systematized, but here is an example that shows statistical analysis requires deep thinking.

That statement is true if there is nothing else in the system but X and Y. In any real-world system, there are more than two relevant factors.

Since the summer is heating up much of the U.S., let's talk about room temperature. One should expect room temperature (Y) to be positively correlated with outside temperature (X): as outside heats up, inside temperature rises.

Not so if we introduce an air conditioner (C) to this system. The air conditioner regulates the room temperature (Y) to some pre-set number, say, 70 F (21 C). In other words, the effect of C is to keep Y around 70F no matter what the outside temperature (X) is.

The air conditioner destroys the correlation between X and Y. If we only collect data on outside and inside temperatures, we shall see no correlation! Despite outside temperature clearly affecting room temperature in a linear way, the data will be uncorrelated.

To learn the true relationship between X and Y, we must measure the lurking variable (C). If we have all three data series, then a regression will assign the proper effects of X and C on Y.

***

If you are comfortable with algebra, read Cosma's post here for a formal discussion.

If you're looking for intuition, keeping reading.

If we receive a data set with X and Y (outside and inside temperature), we can plot a scatter plot or do a regression, and we'll discover no correlation between X and Y. Using the erroneous thinking in the new book, we'd conclude that X and Y could not have a cause--effect relationship since they are not correlated.

And you've just learned that is a wrong conclusion. If we also have data on air conditioning (C), and we throw that into the regression, we will discover the correct relationships, that inside temperature is positively correlated with outside temperature, and negatively correlated with air conditioning.

In practice, whether the analyst draws the right conclusion comes down to whether the analyst knows about C, and whether data on C can be obtained. What separates a good analyst from an average one is the "numbersense" of anticipating what missing data might be useful to shed light on the data you already have.

Also, in any practical system, X and Y are not zero-correlated! Rather, the correlation of X and Y is artificially attenuated because of the lurking variable C. The alarm bell is very likely to ring loudly if the analyst sees zero correlation. If the expected correlation exists, though, that's when the analyst may neglect to consider lurking variables.

 

P.S. [5-27-2021. A number of correspondents have brought up the issue of linearity so I'm adding some comments here. The strict statistical definition of "correlation" is linear. This may be different from our folk understanding of the word. The left chart below shows a positive linear correlation between X and Y:

When the correlation is linear, we can fit a straight line through the scatter plot of X and Y. The chart on the right shows a specific example of non-linear correlation. For example X may be the amount of sun and Y is the plant growth.

If the standard correlation formula is applied (such as the one included in Excel), you get zero correlation between X and Y on the right chart. Roughly speaking, the two sides of the chart cancel each other out.

The lesson is that no correlation means no linear relationship but not no relationship. The same advice is frequently dished out for regression analysis. If fit is low for linear regression, it doesn't follow that there is no relationship between X and Y.

So one can say that if X is a cause of Y, and the effect of X on Y is shaped like the inverted U, then X and Y will show zero correlation (as defined by the linear correlation formula). Nonetheless, this particular issue is present whether or not we impose a causal interpretation on the X-Y relationship.

What I said in the last paragraph of the main post still applies. In real life, we rarely see a perfect inverted U, or perfect other shapes that would result in zero correlation.

Science in the Age of Covid-19, Part 3

Here is Part 3 of my talk on research methods used in Covid-19 science. In Part 2, I point out the key differences between randomized clinical trials (RCTs) and real-world studies. In particular, real-world studies are subject to a variety of real-world biases. The quality of such studies hinges on whether the researchers make judicious decisions about how to correct for such biases.

In Part 3, I explain why even RCT results merit careful interpretation. RCTs are not perfect either, and these vaccine studies offer great materials to learn about the unintended shortcomings. The key topics covered are interim analyses, case-counting windows, staged enrollment, week-to-week variability, placebo effects, immortal time bias (again), conditionality, and story time.

Towards the end of these slides, I discuss the trend of "re-analysis" or "post-hoc analysis". This is one of the more worrisome trends in Covid-19 research. Post-hoc data mining is an about-face to prespecifying an analysis plan, and is the mother of spurious results.

Having run well beyond an hour of materials, I then very quickly describe another research method - using models and simulations. I may come back with a Part 4 to give this section justice. In the meantime, you can review the set of posts I wrote last year about the Oxford model (here).

Here are the links to Part 1, Part 2, and Part 3 of the talk.

Know your data 29: shadow databases

Recently, I received a letter in the mail from a company I have never heard of. The letter has some alarming news to disclose. It's also a standard letter that any American has probably received more than once in our lives.

The company apologizes for having lost my personal data in a recent data leak. 

Don't worry, the letter writer continues, because we're paying for two years of subscription to some kind of monitoring program. 

Receiving this letter unnerves me for the wrong reason.

Why is my data found in the database of a company I have never heard of, much less patronized? Do companies now keep "shadow profiles" of everyone, rather than just their customers?

Yes, I'm well aware that industry has always paid for databases of prospective customers - people who are likely to become customers but aren't currently. In the past, these databases contain only contact information and basic demographics but not sensitive data like driver's license numbers or social security numbers. 

The letter came from an auto insurer, who says they leaked my driver's license number (among other items not disclosed). Not only am I not a customer, I don't even own a car. Any decent predictive model should have rated the chance of getting my business to be an outlier event.

The letter explains that someone was able to steal my license number by using their "online quote form and application process". How does that even work?

What's more, a driver's license number is one of those personal data that can only be lost once. It's like a password except we can't change it willy-nilly. The damage of losing this type of data is immeasurable, much more serious than, say, losing a credit card number.

***

Think of your data as having different degrees of mutability.

1. Data like your date of birth, your ethnicity, and social security number are immutable. Once any organization leaks it, your privacy has been permanently violated. Most of the time, they only know that their servers have been breached, and sometimes, who did it but once the data exit the servers, it's impossible to trace and delete all copies of it.
   
   
2. Data that are effectively permanent. A driver's license number belongs to this category. In theory, one can move to a different state, making the stolen number obsolete. For most of us, our residential address is semi-permanent. If you're a homeowner, it's more immutable, until you move. 
   
   
3. Data that can be changed with a price of inconvenience. If our cell phone number or email address gets stolen, we are free to switch it. However, rewriting these data causes inconvenience. We have to let everyone else know about the change. 
   
   
4. Data that can be easily changed. Credit card companies have figured this one out long time ago. The minute your credit card number is lost, they issue you a new one. You may still get penalized later, for example, when your old credit card number bounced for a subscription.
   
   
5. Data that are temporal. What book are you reading? What's your favorite movie?

A sensible data privacy policy ought to take the degree of mutability into account. The harm to people differs by what type of data is leaked. 

The value of immutable data is far higher than data that can be easily changed. This also makes them more desired. One hopes that the penalty for losing such data is sufficiently high for businesses to make real efforts at safeguarding the data.

 

Menu engineering hijinks

Andrew has a great post about menu engineering. 

Designing restaurant menus has become a science, and the referenced BBC article offers up a parade of the peer-reviewed findings from that field. 

Some headline grabbers include weight of menu is associated with upscale restaurants (kind of like weight of business cards), scripted writing is also deemed higher class, longer words are better, first items are ordered more, and words that "mimic mouth movements" related to the food are more popular.

This type of research has two ingredients that raise their viral potential: the researchers use "conventional wisdom" as foil, either legitimizing them through "science" ("I was right!") or casting doubt ("Everyone is wrong!"); the findings make for light, fun headlines.

Nevertheless, Andrew's post concerns the quality of the underlying research. 

***

Since I've been waxing about research methods of Covid-19 studies, let's also talk about the type of methodology typically deployed for food science studies. 

The researchers recruit a bunch of people, usually a few hundred at the most, sometimes college students, and they put them in a "lab experiment" in which they are exposed to (say) menus that are heavier or lighter in weight. Because it's an lab experiment, the participants are randomly assigned to these treatment groups, and then they fill out a survey which measures outcomes. 

This research method is very common in social science academia, but in recent years, people are discovering that they often fail to replicate. (Lots of posts about this scientific crisis on Andrew's blog.)

The lab experiment method has a number of problems. First, the lab scenario is completely contrived. The participants are not actually hungry, ordering food, or consuming them. They are not spending real money.

Second, the participants are not a representative sample of the population to which the research will be generalized. The recruits are not typical diners (and certainly, not customers of upscale restaurants).

Third, saying in a survey that they will pay more for something doesn't mean they will actually pay more real money for it.  

Fourth, such research typically assumes no meaningful variation in human behavior. The sample size is too small to account for such variability. The investigators are looking for a universal behavior - and they include just enough samples to achieve a magical p-value threshold required for scientific publication.

Fifth, almost all such research isolate a single factor. It is a well-known fallacy in design of experiments to add the effects of separate one-factor studies - remember, each experiment assumes all else equal.

***

I left the following comment:

Now, wake me up when they do an actual statistical experiment. They could partner with a restaurant chain, design an experiment involving all of the above factors, print different versions of the menu, and randomly assign them to real diners who make real orders. As a foodie, I’d love to be part of this research team!

 

 

Science by press release should be discouraged

I've been holding off on making comments on a few Covid-19 topics, hoping that some data or scientific reports would be released. Nothing seems forthcoming, though. The FDA/CDC has been busy making many pronouncements but unfortunately, solid scientific evidence is hard to find. Please add links to the comments if you are aware of any data or research reports that address the topics below.

 

Pfizer and Moderna both claimed their vaccines are definitely effective for at least six months.  

The entire basis for this claim appeared to be press releases from the two pharmas (Pfizer, Moderna). The FDA/CDC then parroted what was written. These pronouncements shocked me because I was eager to learn what analytical technique they might use to cure the destruction of the placebo group.

Yes, you heard that right. At this point, the vaccine trials are handicapped. During the  EUA hearings in December/January, both pharmas have argued that it is unethical to withhold a "miraculous" vaccine from people, and therefore, everyone in the placebo group will be offered the vaccine. This deliberate dismantling of the experimental design should have robbed us of any opportunity to assess longer-term efficacy or safety from the clinical trials. (See my prior post that raised this issue on the day Pfizer got approved.)

Allowing the placebo group to get vaccinated presents two problems: the remaining placebo group shrinks in size, making error bars wider; and the self-selected exit from the placebo group destroys the covariate balance that is the hallmark of a randomized controlled trial. Any attempt to salvage such an experiment will be messy.

The Pfizer press release ignores this issue completely. The Moderna press release disclosed this:

"As of April 13, all placebo participants have been offered the Moderna COVID-19 Vaccine and 98% of those have received the vaccine."

This tells us a lot and very little, depending on what question you're asking. The way this sentence is tagged on after presenting the six-month efficacy finding suggests that the analysis was completed before the placebo group was dismantled. However, looking at the timeline, I find that hard to believe. More likely, at least a generous section of the placebo group has been vaccinated by the time they did this analysis.

The 98% number also surprised me, and reveals an interesting tidbit for the very first time. The participants of these clinical trials are vaccine enthusiasts who immediately jumped at the chance of getting vaccinated at their first chance. The clinical trials were ostensibly single-blind; nevertheless, a participant could roughly guess which arm they belonged to based on the degree of side effects. This potential bias is a wonky issue for serious researchers to chew on.

Moderna claimed to have followed the median participant for 6 months while in the Pfizer analysis, only about one quarter of the participants have reached 6 months. These details are important because the subset of participants driving the results in the preliminary analysis is the same group that drives the results of this follow-up analysis. In other words, the subgroup that did not reach 2 months at the two-month analysis did not reach 6 months at the six-month analysis.

Pfizer is approved for adolescents.

I also failed to find any data or scientific reports to support the press releases. There is a Pfizer press release and an FDA press release. If you've been reading this blog during the pandemic, you know that the merits of statistical findings cannot be evaluated based on a press release. It matters a lot how they count cases, what case-counting window they use, what is excluded, etc. etc.

The first issue with the adolescent trial is its small sample size (only about 1200 per arm, 1/16 of the size of the adult trial). In the adult trial, when half the trial population reached two months after the second dose (so roughly 3 months in), the total number of cases across both arms was 325, or 0.7%. If the case rates were similar, we should expect only 18 cases across both arms in the teenagers 3 months in. The real number should even be lower because the case rate for this age group should be smaller than for adults.

It appears that the FDA is primarily relying on an "immunogenicity" outcome, which is a measure of the amount of antibodies found after vaccination. This evaluation breaks the experimental design; it's really no longer a clinical trial. That's because the level of antibodies in the placebo group is expected to be zero. What's more, no one knows what is the minimum level of antibodies sufficient for protection against infection, developing symptoms, severe disease or death!

That's why the level of antibodies of these teenagers is compared - not to the placebo group in the teenager trial - but to 16-25 year olds in the adult trial! Alarm bells are ringing: different ages, different sizes of age range, different time period, different trial, not randomized.

And there's more. They did not measure antibodies in all 1200 vaccinated teenagers. They only had data on 190 people. (This bit of information is omitted in the Pfizer press release but found in the FDA press release. That's exactly why we shouldn't do science by press releases.)

[5/20/21: Added the next paragraph]

In short, the immunogenicity result does not compare teenagers who were randomly assigned to the vaccine group with those in the placebo group. It says that the vaccinated teenagers have at least similar levels of antibodies as the 16-25 age group in the adult trial from last year. Also, only a small subset of trial participants have measured immunogencity outcomes.

Both press releases also boast a 100% efficacy number, without providing the error bar. The two press releases give different numerators and denominators. Let me take the FDA number, which claims a case rate of 16/978 = 1.6% on the placebo arm of the teenager trial. This rate appears way too high. It is double the number I gave before which came from Pfizer's adult trial. Further, the 16 cases count only those happening beyond 7 days after the second dose, while the 325 cases I used above count all cases. Unless there were no cases between first dose and 2D+7 (itself a shocking development), the case rate comparable to the 0.7% would be well above 1.6%. Are they defining cases in the same way as in the adult trial? I don't know since I can't find a trial protocol, a scientific report or any data for the adolescent trial.

***

The theme of my recent talk is that not all research studies are created equal. Even though the adolescent finding and the six-month finding are both analyses of clinical trial data, neither analysis preserves the structure of the experiment, and thus should be classified as observational studies.

The FDA/CDC should commit to releasing the data and scientific reports supporting these decisions. Science by press release should be discouraged.

 

Science in the age of Covid-19, Part 2

In yesterday's post, I set the stage for discussing how science has been practiced during this pandemic.

I first outlined the key challenges for any data science during this period - especially in the early stages of the pandemic. What subsequently happened was not preordained. There was always the possibility that Covid-19 could run out of steam long before killing millions. In the history of pandemics, there hasn't been many that have hit so far.

Next, there has been an absolute avanlanche of research studies coming from all corners. The Medrxiv is averaging 900 new papers a month. No one has the ability to read all of these papers, not even the abstracts. The traditional science by peer review has been replaced with science by press releases and public relations. I can't imagine how long it would take for the peer review process to digest these thousands of preprints.

Unfortunately, coverage of these research studies tend to focus on the headline results, ignoring important scientific issues such as the research method, how data were collected, whether data were biased, etc. I gave an example in which the reporter did not distinguish between (a) a Phase 3 trial with 40,000 participants and a Phase 1 trial with 33, and (b) a result backed by a scientific publication and a result announced by a vaccine manufacturer in a press release. These results were presented together as if they were equally valid.

I then identify six major classes of research methods behind these research studies. The rest of my presentation addresses the strengths and weaknesses of these methods.

***

In Part 2, the focus is on the two most popular data-driven research methods: statistical experiments (aka randomized controlled clinical trials) and observational studies (aka real-world studies).

I address the following questions:

• What are the key elements of a statistical experiments?
• How are vaccine trials like/unlike a statistical experiment?
• Why is the experiment considered the gold standard?
• Does vaccine effectiveness coming out of a real-world study give the same information as the similar outcome from an experiment?
• What is different about real-world studies from experiments?
• How do researchers correct for selection biases in real-world study populations?
• How does immortal time bias and calendar time bias affect real-world studies?

Of course, experiments and real-world studies are both necessary. Once the vaccination campaign started, it is considered not ethical to run experiments so we only have observational data. My conclusion is that these methods should be utilized for different purposes.

 

Download Part 2

 

Download Part 1

 

***

There is a Part 3. Come back next week!

Science in the age of Covid-19, Part 1

A couple of weeks ago, I gave a keynote at a U. of Denver conference, titled "Science in the Age of Covid-19". As readers of the blog know, I've been busy throughout the pandemic, examining how data and statistics are used in the torrent of research studies. This talk summarizes a chunk of those materials.

The talk is quite long. I will release the slides here over three blog posts. Part 1 introduces the topic. I outline the key challenges of conducting data studies during this period of time. I classify many of the key studies by research methods.

 

This sets me up to discuss the strengths and weaknesses of various research methods in Part 2.

The talk is evolving, and I'd be happy to hear any feedback.

 

Download Part 1

 

P.S. [5/13/2021: Part 2 will be posted soon. If the link isn't live, come back by end of day.]

Nothing is perfect especially in statistics

In a previous post, I explain the miracle of randomization for causal inference using a simple business example. I left out some side issues - bad things that could still ruin statistical experiments. These are real issues but edge cases.

As a recap, we pretend that Amazon is running an email test on Christmas eve, hoping that an extra email will lead to incremental holiday sales. One half of the targeted customers receive the email while the other half doesn't.

What randomization gives us is covariate balance, which supports an all-else-equal analysis. Accidents like Yahoo's spam filter unexpectedly suppressing our emails (see previous post) affect both arms of the experiment equally; in this sense, the two arms are still comparable despite the accident, and even if we are unware of its occurrence.

One issue that can arise in that scenario is if a majority of our business comes from customers with Yahoo email addresses. The spam filter did not destroy the covariate balance, and the difference between the test and holdout arms is still unbiased. However, the totality of the test and holdout arms no longer adequately represents our research population.

This situation is directly analogous to the problem I've been pointing out regarding the real-world studies of vaccine effectiveness (see this post). Recall that the matching strategy artificially creates a control group that is balanced with a test group (by construction). However, the totality of both groups no longer represents the entire population: that's because we inevitably fail to find appropriate matches for some people - if the observational dataset is sufficiently biased.

Returning to our A/B testing example, we will still have a good read on customers with non-Yahoo email addresses. We just don't know what the effect of the promotional email has on people with Yahoo email addresses. In the Clalit study, for example, since they excluded everyone living in nursing homes or confined to their homes, and they only matched 15% of the oldest age group, the published vaccine effectiveness doesn't tell us anything about the vaccine's effect on those subgroups.

***

A second problem pertains to the nature of the "accident". We are in serious trouble if the accident preferentially impacts one of the two test groups.

Let's say Amazon plans to track the performance of the promotional emails by routing customers to a special landing page, which eventually leads to the usual checkout process. Now, it turns out that the server that delivers the new landing page suffers intermittent outage during the week of the email promotion. Since this landing page is only seen by the test group (receiving the emails), this accident affects one arm but not the other in the A/B test.

Randomization cannot save us from this type of operational hiccup.

***

Bonus tip: think about how a testing analyst might discover the operational issue. It will never surface if all you're doing is looking at a dashboard that shows you the horserace between the two arms.

It goes down to having the courage to anticipate even unlikely mishaps, designing an auditing process that can identify unexpected problems, and keeping the faith that this work will one day save a wayward test! Ideally, you want to identify the outage during the early phases of the test and correct it.