Big Data, Plainly Spoken (aka Numbers Rule Your World)

A Citizen Petition

A distinguished group of scientists and experts is circulating a Citizen Petition against the premature granting of full approval of the Covid-19 vaccines. You find the full text here.

In the last few weeks, we learned that both Pfizer and Moderna have already lobbied to have their vaccines fully approved. You may recall that their vaccines were granted emergency use authorization, which as the name suggests, is not full approval. A normal vaccine trial takes roughly two years because we need to assess the long-term benefits and risks of these injections. Time passing is the only way we can measure long-term effects. When the EUAs were granted, the FDA loudly proclaimed there would be no "short cuts", that the trials will continue until their designed end dates. It looks like they were just joking.

This fight is about maintaining the standards of science. It is about stopping bad practices from lingering on after this emergency is over. As you have read on this blog over and over, while the scientific results of the trials and real-world studies have been encouraging, the science has been far from perfect - and in this emergency, people have been willing to look the other way on the flaws of these studies.

The EUA status of the vaccines is not stopping anyone who wants to get vaccinated from getting the shots. A while ago, an argument was made that full approval would encourage employers to make vaccinations a requirement for work. This is a red herring that assumes no vaccine hesitancy among top decision-makers at all employers. People who are hesitant will simply find their way to employers who embrace vaccine hesitancy. We should have learned the lesson from the mask-wearing red herring. Some erroneously believed that if vaccinated people are "rewarded" with not having to wear masks, the vaccine hesitant people would suddenly decide to get innoculated. Polls show that the same people who are vaccine-hesitant are the ones who have been ignoring masking rules all along.

Another argument is that people will trust these vaccines more if they were fully approved, rather than authorized for emergency use. The elephant in this room is that a full approval being granted prematurely is not full approval. Those people who make the distinction between EUA and full approval surely can see through this ruse.

Trust is a precious commodity. Once lost, it's hard to earn back. I agree with the petition. Why is there an urgency to grant full approval?

***

To make your voice heard, click here to comment on the petition.

 

[P.S. The authors of the Citizen Petition has a piece in BMJ explaining their reasoning. See here.

Importantly, they said: "publicly raising any element of hesitation about covid-19 vaccines will be seen by some as irresponsible, stoking unfounded fears in the public’s mind and contributing to the “vaccine hesitancy” problem trumpeted every day. But the alternatives—privately raising concerns or simply remaining silent—are arguably more detrimental to public trust in the long run."

]

Adolescent trial showcases pre-registration and the Law of Small Numbers

The scientific report on Pfizer's adolescent trial did not bring closure on the two key issues I flagged in my previous post (link). Here are my notes on this trial. 

The Key Efficacy Finding Does Not Require a Randomized Controlled Trial

The researchers explained that based on the low expected Covid-19 case rates among 2,000 teenagers (split into two groups), they were not going to see enough cases to offer the same kind of analysis as in the adult trial. They have convinced the FDA (or the other way round) to accept immunogenicity findings as a substitute. That is to say, Pfizer will show only that injecting teenagers with the vaccine generates antibodies intended to fight the SARS-CoV-2 virus. 

The sample size issue is real. The infection rate in teenagers is well known to be much lower than in adults. It's hard to find national numbers split by age group, though. I found out that in California, roughly 4% of children below 17 have tested positive for Covid-19 during the whole pandemic while the number was close to 10% for people above 18. These numbers are cumulative for the 12-15 months of the pandemic whereas a typical vaccine trial spans 3-4 months at the time of interim analysis. Detecting a difference in tiny signals (rare events) is challenging and requires a large sample size. Two thousand is like 20 times fewer participants than the adult trial. 

Typically, immunogenicity results advance a treatment to Phase 3. These lab tests prove that the vaccine produces a good amount of antibodies as intended. However, having antibodies does not guarantee the vaccinated people will be protected from infection. Nor do lab results suffice to decide what level of antibodies is sufficient to reduce infection (and transmission). Further, Phase 2 trials involve few people. Thus, a Phase 3 trial is required for FDA approval. Phase 3 trials involve thousands of people, and measure clinically relevant outcomes, which for a vaccine usually means reduction of infection.

With immunogencity as the key efficacy metric, the adolescent trial is in spirit an expanded Phase 2 trial. You don't even need to have a placebo group, nor randomize treatment. Injecting someone with saline will not produce antibodies against the coronavirus. They also measured antibodies in only 200 kids, a 20%-ish subset of those enrolled in the adolescent trial. They could have obtained the same outcome by recruiting 200 kids, giving them two shots, and measuring their antibody levels afterwards. No RCT is necessary.

The average antibody level of the 12-to-15-year-olds is compared to a number they previously obtained from the adult trial, involving those in the 16-to-25-year-old age group. If similar levels of antibodies are found, they presume that the vaccine will work in teenagers.

There are several assumptions required to justify such a claim. First, they assume the vaccine works with the same efficacy for 16-to-25-year-olds in the adult trial as the overall VE. This is not a given since the trial did not contain enough people in that (or any other) age group to produce a reliable estimate of subgroup efficacy. Second, they assume there are no material differences between the two age groups that affect immune response. Third, they assume that a direct link between antibody levels and reduction in case rate has been established in the 16-to-25-year-old age group. If that is true, I haven't seen the evidence.

In the protocol, the scientists were supposed to provide an "immunobridging" analysis, which sounds like a mathematical model that justifies the third assumption. They ultimately did not come through with that model. 

 

The Trial Provides Minimal Data on Safety

While the placebo arm is redundant for the pre-specified efficacy analysis, the placebo participants are useful to establish a background level of side effects, to help the vaccine developer make the case that the vaccine does not cause more adverse incidents than expected.

Nevertheless, the conclusion one can draw from a study of 1,000 vaccinated participants is highly limited. If a deadly side effect has a 1 in 5,000 chance of happening, there is over 80% chance that there will be zero cases of this side effect during this trial. With probably ~15 million Americans aged 12-15, such a side effect would affect 3,000 kids. Fewer than 200 American teens have died from Covid-19.

The trial did not identify any major concerns but what we don't know, we don't know.

 

What about that 100% VE talking point?

On Twitter and cable TV, the talking heads have mostly not presented the trial results as I have above. Below is a typical example:

Notice that they ignored the immunogenicity results and trumpeted a vaccine efficacy (VE) number. In the study, Pfizer reported a "surprise," saying that they found 16 cases of Covid-19 among the placebo group, and zero cases in the vaccine arm. 

In the NEJM paper, on the other hand, the immunogenicity outcome was unmistakably featured as the primary endpoint. This raises a critical scientific issue that is perhaps hard for an outsider to grasp.

All clinical trials are required to submit protocols and analysis plans well ahead of enrollment. This process is called pre-registration (investigators are not supposed to make edits, especially major edits, while the trials are running although this practice has become a casualty of the pandemic). Pre-specifying your analysis prevents data mining for results after the data have materialized. Such data mining frequently results in spurious findings - mistaking one-off phenomenon as generalizable outcome. Spurious findings will not replicate if the trials were to be repeated.

In the adolescent trial, pre-registration made Pfizer think about how to demonstrate efficacy. They feared that using the usual measurement based on symptomatic, PCR-positive Covid-19 cases is likely to saddle the study without a conclusion because they were only willing to run a trial with 2,000 participants. Therefore, they chose the immunogenicity outcome as the primary endpoint.

Now, after the trial finished, the Pfizer press release - and its readers - trumpeted the 100% VE metric as if that was the pre-specified analysis. This behavior usurps the principle of pre-registration. 

So let me now explain why there is a good chance the observed VE is an outlier. If the whole trial were repeated, I'd be surprised to see the same result repeated.

According to the FDA's press release on approving Pfizer's vaccine for teenagers, the case rate in the placebo arm was 16/978 = 1.6%. In Pfizer's adult trial, the corresponding rate was 162/18325 = 0.9%. So, the teenagers were getting sick at a rate 80 percent higher than that of adults. If I assume that the kids were equally likely to get infected as adults at 0.9%, the chance that we would find 16 or more cases in 978 kids is less than 2 percent. If we were to run this trial 100 times, 98 or more of them would observe fewer than 16 cases in the placebo group.

But teenagers have less than half the case rate of adults as mentioned above. If I assume that the case rate for kids is at 0.45%, the chance that we would find 16 or more cases in 978 kids is 0.001%. Don't forget, also, that the case rate only counts adjudicated, lab-confirmed, symptomatic Covid-19 cases occurring at least 7 days after the 2nd dose so the real-life infection rate is even higher than the 1.6%.

This is very strong evidence that what was observed during this trial was an outlier event. 

Nevertheless, this outlier event has been painted as a "pleasant surprise". We didn't think we'd have enough cases to show anything but how could we refuse the unexpected gift of 100% efficacy? The real surprise was not the VE but the inexplicable, highly unlikely incidence of Covid-19 among teenagers enrolled in this specific trial.  

In fact, this is exactly the reason why pre-registration is good practice. If we allowed this type of post-hoc data mining, spurious findings will bloom. 

***

See also my previous rant about the way the paper was written up as if the adolescent and 16-to-25-year-old age group were originally planned as two arms of a trial.

See also my post on the Law of Small Numbers, one of the seminal contributions by Tversky and Kahneman. The Law of Small Numbers is the false belief that the Law of Large Numbers applies to one's small sample.

 

 

 

 

Science, politics or business

I'm surprised the TV pundits have applauded the State Department's advisory against travel to Japan (link). That decision is not science-driven.

Let's look at the data:

Even with the so-called "surge" in cases in Japan, the case rate per million in Japan is half that in the U.S. as of yesterday. Paradoxically, the chance of an American getting sick in Japan is half than that of an American in the U.S.

Sitting in a country where we have allowed coronavirus to spread readily makes us lose perspective. The case rates in most of Asia are a fraction of what we experienced in the U.S. If we isolate Japan, there is indeed a "surge":

Case rate jumped five times in the last month, and may have topped. At this peak, the rate was still half that of the U.S. case rate - at which level our public health "experts" are declaring the pandemic all but over!

According to the "experts", those case rates mean nothing. The only thing that matters is vaccinations. The Japanese have only vaccinated 2% of their population. One wonders if this is about selling vaccines or getting infected?

***

Not long ago, CDC told us that the benefits outweigh the risks when it comes to blood clots and the J&J vaccine. At the time, they estimated that the risk of blood clots after taking the J&J vaccine was 7 per million. Roughly 30-50% of these cases led to death.

[Harlan pointed out below that the data required to make this point is not quite what I had so I've revised this section as below.]

One reason why the rate of vaccination in Japan (and in many other Asian countries) is low is that people have experienced a much lower risk of dying from Covid-19.

Japan's death rate has not breached 0.8 per million in the entire pandemic. In the U.S., we experienced periods of high death rates, up to 10 per million on some days. A total of 13,000 people have died from Covid-19 in Japan since the beginning of the pandemic while almost 600,000 have in the U.S. Over 40,000 people die from accidents in Japan every year (link).

The risk vs benefit trade-off is different for each country, and in fact, it is different for different age groups, ethnicities, genders, etc.

 

Instant Classic for Science Communications Students

When I blogged about Pfizer's adolescent trial (here), I was wondering when and if they will release the data or the scientific report to support those press releases (one by Pfizer, and one by the FDA). A report has just been published in the prestigious New England Journal of Medicine.

I predict this will become a staple of future science communications textbooks.

The 12-15 year old age group presents extreme difficulties for conducting clinical trial research - the case rate is expected to be extremely low, and there is a strong desire to fast track approvals. Those two issues are incompatible. You need a huge sample size to detect low rates of events, let alone differences between low rates of events. From the start, the FDA has allowed a head fake: the primary endpoint of the adolescent trials will be immunogenecity results, i.e. test of antibody levels in the vaccinated. This is no ordinary trial, as we will not be comparing adolescents to adolescents, vaccinated against placebo. We will be comparing adolescents to 16-25 year olds, vaccinated against vaccinated.

So what does the highly respected NEJM say about the 16-25 year olds, the designated "control" group? I searched for every mention of 16-25 year olds in the report. Below are some representative examples of the matching sentences (Scroll to the very end of the post to see the entire list of mentions).

• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.

You can read every word of the above sentences, or every word of the entire report, including every word in the footnotes, and you walk away thinking this adolescent trial included two age groups (12-15 and 16-25). What if this weren't true? What if the number of 16-25 year olds in this adolescent trial is exactly zero?

As I was reading the prestigious NEJM report, I was feeling a sense of dread, which worsened as page after page repudiated what I thought was true: that the 16-25 year olds data came from the previous adult trial. Indeed, one of the key issues I flagged in my previous blog post is that the analysis plan involved comparing one age group in one trial to another age group in a different trial. Did I mislead my readers? Was I so blinded by my own biases that I made up this detail?

It surely was not possible that scientists and peer reviewers and the editors of the prestigious NEJM would allow such a basic but important detail to be omitted. It has got to be my brain fog... am I losing grip of my sanity?

So I checked my notes... The very first line of the Pfizer press release from March 31 stated unmistakably: "In participants aged 12-15 years old, BNT162b2 demonstrated 100% efficacy and robust antibody responses, exceeding those reported in trial of vaccinated 16-25 year old participants in an earlier analysis, and was well tolerated." (their italics)

Next, I checked the FDA press release on May 10. The source of data for the 16-25 year olds has been wiped out. In the safety section, the FDA stated "The side effects in adolescents were consistent with those reported in clinical trial participants 16 years of age and older. " As if the adolescent trial contained participants 16 years of age and older. In the efficacy section, the FDA stated "The immune response to the vaccine in 190 participants, 12 through 15 years of age, was compared to the immune response of 170 participants, 16 through 25 years of age. " Unless you know otherwise, you're going to assume the trial contained participants of both age groups.

I warned you up top that you're witnessing an instant classic in science communications.

It takes discipline and skill to write an entire report that dances around a key detail. A true masterclass in deception. This is an instance of "story time," lulling readers in with clinical trial data but presenting conclusions from analyses that do not enjoy the benefits of an RCT. But "story time" trivializes what has happened here. I hate to use the F word. I'm not thinking the four-letter F word. The five-letter one. F R A U D.

***

Also of interest is the other side of the equation. How does one develop a nose for detecting fraud in data? I have to admit that it would have been very hard for anyone to notice the deception in this NEJM paper - that's why it belongs on the syllabuses of science communications courses.

There were just two tiny cracks. The first was when they disclosed that all 12-to-15-year-old participants came from the U.S., while 16-to-25-year-old participants were globally recruited. This would have been unusual to do in a single trial enrolling 12-25 years old, stratified into two age groups.

The second crack was in the way the waterfall chart was presented.

Notice the two age groups were presented in two separate panels. What we expect to see if the trial actually enrolled both age groups is something like this:

The waterfall starts with the entire set of enrollees and then there is a split by age groups. This chart came from the Phase 2 Moderna study of two half-doses that I discussed in this post.

As I said, these are subtle signs, easily missed.

***

Is it never allowed to pull a control group from a different place? Doing so does not automatically disqualify the analysis but such an analysis does not enjoy the exalted gold-standard status of a randomized controlled trial. Such workarounds must be scientifically and rigorously justified. Work must be presented to convince readers that any biases coming from different times, different places, different ages, different demographics, different anything have been measured and corrected for.

Here, the authors did not present any evidence about possible biases. They didn't even acknowledge the source of the control group data.

***

Listed below is every single mention of the 16-to-25-year-old control group in the Pfizer adolescent study in the NEJM scientific report. Each mention represents an opportunity for the researchers to inform readers that this control group came from a different trial.

• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was an immunogencity objective.
• The geometric mean ratio of SARS-CoV-2 50% neutralizing titers after dose 2 in 12-to-15-year-old participants relative to 16-to-25-year-old participants was 1.76.
• This report presents findings from 12-to-15-year-old participants enrolled in the United States, including descriptive comparisons of safety between participants in that age cohort and those who were 16 to 25 years of age and an evaluation of the noninferiority of immunogencitiy in the 12-to-15-year-old cohort to that in the 16-to-25-year-old cohort. Data were collected through the cutoff date of March 13, 2021.
• The immunogenicity objective was to show noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants.
• Noninferioirty was assessed among participants who had no evidence of previous SARS-CoV-2 infection with the use of the two-sided 95% confidence interval for the geometric mean ratio of SARS-CoV-2 50% neutralizing titers in 12-to-15-year-old participants as compared with 16-to-25-year-old participants 1 month after dose 2.
• The reactogenicity subset included all 12-to-15-year-old participants and a subset of 16-to-25-year-old participants (those who received an e-diary to record reactogenicity events).
• Noninferiority of the immune response to BNT162b2 in 12-to-15-year-old participants as compared with that in 16-to-25-year-old participants was assessed on the basis of the geometric mean ratio of SARS-CoV-2 50% neutralizing titers.
• In the reactogenicity subset, all the 12-to-15-year-old participants were from the United States and the 16-to-25-year-old participants were recruited globally.
• Severe injection-site pain after any BNT162b2 dose was reported in 1.5% of 12-to-15-year-old participants and in 3.4% of 16-to-25-year-old participants.
• After BNT162b2 injection, severe headache and severe fatigue were reported in a lower percentage of 12-to-15-year-old participants than of 16-to-25-year-old participants.
• Fever (oral body temperature, >= 38 C) occurred after dose 2 of BNT162b2 in 20% of 12-to-15-year-old recipients and in 17% of 16-to-25-year-old recipients.
• The use of antipyretic agents was slightly more frequent among BNT162b2 recipients who were 12 to 15 years of age than among those who were 16 to 25 years of age.
• [Footnote of Table 1]: Results are for the reactogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and a subset of participants in the 16-to-25-year-old cohort.
• Among the 16-to-25-year-old BNT162b2 recipients, 11% reported any adverse event and 6% had vaccine-related adverse events. Among BNT162b2 recipients, severe adverse events were reported in 0.6% of those who were 12 to 15 years of age and in 1.7% of those who were 16 to 25 years of age.
• One BNT162b2 recipient in the 16-to-25-year-old cohort discontinued vaccination because of severe vaccine-related injection site pain and headache, both of which were reported 1 day after dose 1 and resolved within 1day.
• Lymphadenopathy was reported in 9 of 1131 BNT162b2 recipents (0.8%) and in 2 of 1129 placebo recipients (0.2%) who were 12 to 15 years of age, as compared with in 1 of 536 BNT162b2 recipents (0.2%) and in no placebo recipients who were 16 to 25 years of age.
• Appendicitis was reported in 2 participants: 1 BNT162b2 recipient in the 16-to-25-year-old cohort and 1 placebo recipient in the 12-to-15-year-old cohort.
• The immune response to BNT162b2 in 12-to-15-year-old adolescents was noninferior to that observed in 16-to-25-year-old young adults.
• The geometric mean ratio of the BNT162b2 neutralizing geometric mean titer in 12-to-15-year-old participants to that in 16-to-25-year-old participants 1 month after dose 2 was 1.76.
• Among all participants regardless of serologic evidence of previous SARS-CoV-2 infection, the serum-neutralizing geometric mean titer 1 month after BNT162b2 dose 2 was 1283.0 in the 12-to-15-year-old cohort and 730.8 in the 16-to-25-year-old cohort.
• Substantial increases in the 50% neutralizing titer from baseline were observed, with GMFRs from baseline to 1 month after dose 2 of 118.3 among 12-to-15-year-old participants and 71.2 among 16-to-25-year-old participants.
• [Caption of Figure 2]: The results shown are for the reacogenicity subset of the safety population, which included all participants in the 12-to-15-year-old cohort and the subset of participants in the 16-to-25-year-old cohort who had electronic diary data available.
• [Footnote of Table 2]: The geometric mean ratio and two-sidded 95% confidence intervals were calculated by exponentiating the mean difference of the logarithms of the titers (the 12-to-15-year-old cohort minus the 16-to-25-year-old cohort) and the corresponding confidence intervals (based on the Student's t distribution).
• Antipyretic use after both dose was slightly higher in the 12-to-15-year-old cohort than in the 16-to-25-year-old cohort.
• All 12-to-15-year-old participants in this trial were enrolled at U.S. sites, whereas the 16-to-25-year-old participants were recruited globally.

Science in the Age of Covid-19, Part 3

Here is Part 3 of my talk on research methods used in Covid-19 science. In Part 2, I point out the key differences between randomized clinical trials (RCTs) and real-world studies. In particular, real-world studies are subject to a variety of real-world biases. The quality of such studies hinges on whether the researchers make judicious decisions about how to correct for such biases.

In Part 3, I explain why even RCT results merit careful interpretation. RCTs are not perfect either, and these vaccine studies offer great materials to learn about the unintended shortcomings. The key topics covered are interim analyses, case-counting windows, staged enrollment, week-to-week variability, placebo effects, immortal time bias (again), conditionality, and story time.

Towards the end of these slides, I discuss the trend of "re-analysis" or "post-hoc analysis". This is one of the more worrisome trends in Covid-19 research. Post-hoc data mining is an about-face to prespecifying an analysis plan, and is the mother of spurious results.

Having run well beyond an hour of materials, I then very quickly describe another research method - using models and simulations. I may come back with a Part 4 to give this section justice. In the meantime, you can review the set of posts I wrote last year about the Oxford model (here).

Here are the links to Part 1, Part 2, and Part 3 of the talk.

U.S. regulators asleep at the wheel

My friend Mark Palko has been on Tesla's case for a while. He just commented on the latest fatal accident involving a Tesla vehicle. Amusingly, at this point in time, we are supposed to believe that a ghost crashed the car killing two passengers - as we're being told neither a human driver nor Tesla's software was directing the car.

Mark rightfully laid the blame on regulators. It's pretty clear that U.S. regulators put business profits above citizens' interests. We learned how Boeing was allowed to inspect themselves. The Europeans, not U.S. regulators, are reigning in the likes of Google and Facebook. Tesla's case is particularly telling.

Observation #1

U.S. regulators are allowing Tesla to lie about its "self-driving" technology. The "Auto-pilot" software does not auto-pilot - in fact, drivers are told to have their hands on the wheel at all times. The "full self-driving" mode is not what researchers in the field mean by full self-driving. Tesla customers are used as test subjects for an immature technology. The regulators should have stepped in to stop the false advertising but they have been absent.

Observation #2

After an accident happens, it appears that regulators rely on Tesla to report on itself. It seems that only Tesla has data on whether Auto-pilot or other features were turned on or off, whether the driver's hands were on the wheels, etc. Imagine the Chauvin trial, and the only video for the incident came from the police's bodycams. How do we know the footage hasn't been edited, and that the best perspectives were submitted?

Observation #3

The entire conceit of requiring the drivers to have their hands on the wheel at all times while using "Auto-pilot" not only makes a mockery of the term "auto-pilot" but also creates a liability-free zone for Tesla. If the driver's hands were determined not to be on wheel (and the only source of this information is Tesla), any accident is blamed on the driver. In past accidents, they even faulted drivers for momentarily not touching the wheel. If the driver had hands on the wheel, the accident is also not caused by Tesla because this accident could not have been prevented.

Observation #4

Some Tesla fans argue that everyone should know "Auto-pilot" is false advertising, and so any dead or injured Tesla customers have themselves to blame since they chose to use the feature. Using this logic, anyone who died from opioids deserves to die since they chose to ingest.

 

 

 

 

In science, truth matters

When I turn on the media and listen to the media hyperventilating about Covid, it's no different from turning on CNBC, listening to Jim Cramer etc. lecturing us on why the stock market went up or down yesterday. If cases went down, it's because of the vaccine. If cases went up, it's the variants. We can write down all the usual suspects: vaccine, lockdowns, variants, holidays, travel, people behaving badly, masks, schools, treatments, parties, dining outdoors, dining indoors, etc. Depending on what happened yesterday, the host picks one from the box. Medical science reporting has descended to the level of stock price watching.

So, I was immediately attracted to Andrew Gelman's recent account of science as "a harbor clotted with sunken vessels," in which he lamented the current state of scientific publications that elevates headline-grabbers, failing to self-correct when such studies subsequently prove to be bogus.

The problem is much broader, and it's about to hit medical research in a huge way, once the pandemic is in the rear-view mirror. The emergency situation has given cover to an ocean of pre-prints and warp-sped refereeing, in which headline-grabbers have floated to the top, without a clear path for future researchers to navigate through this sea of debris. Many - if not most - of these studies will eventually be debunked.

***

In his post, Gelman cited the example of research on the effect of neuropeptide oxytocin (OT) on trust between humans. A seminal paper published in 2005 claimed to have found an association between the two. The original experiment failed to replicate, and by now, it should be clear that the association has not been proven. And yet, Prof. Gelman found that (a) the citations of the original paper are still multiple times larger than those of the replication studies; and (b) the original paper continues to get new citations. He just did a casual Google search so we don't know if those who cited the original research also cited the refutations - I doubt it.

If science advances by cleansing bad ideas as better ones emerge, this situation is not putting it in a good light. This situation is due to a host of reasons. One can be confirmation bias - the researchers are looking for anything that supports an assumption of theirs, so they can move on with the rest of their study. Another reason is professional misconduct, or socialized behavior of a subculture.

For me, the problem is also rooted in the practice of scientific publication. It's too damn hard to find those follow-up studies! The following is a diagram of a scientific paper which links to past papers through the device of "references".

By definition, a reference is a link to the past. So a replication study, or for that matter, any kind of follow-on work, will not be found in the references, nor in the original paper. In this system, it's a chore to discover future papers that reference a given paper. Modern search engines attempt to solve this problem by the device of "citations". From the perspective of the cited paper, a citation is a link to a future paper that lists it in the references.

Unfortunately, the citation device is extremely noisy. Most of the papers that cite the original paper do not evaluate the merits of the paper. Citations are tracked as a means of measuring the impact of a given paper - and this has led to gaming. Citations are given to appease referees, or to elevate colleagues, or any number of reasons.

Buried deep inside a long list of citations may be a few replication or follow-up studies. These are future studies that directly address the validity of the original paper.

Adding to the mess, one expects follow-up studies to be published long after the original publication. Citations that are not follow-up studies are both more numerous and also sooner to arrive than follow-up studies.

Researchers may then rely on meta-analyses or survey studies. If they exist, these studies are easy to locate. But they are published far, far, far into the future relative to the original study. They also require the existence of a prior body of follow-up or replication studies.

The end result is that follow-up studies take a lot of effort to find.

Nevertheless, if there is a will, there is a way. Follow-up studies can be marked or registered. And they could be automatically attached by search engines to the original paper, just like citations, but as a separate tag.

Why should we care? Because in science, truth matters.

***

This timing problem affects other fields as well. I published a letter to the editor once about a New York Times Magazine article. But if you read that original article, you're not going to find my letter. There are no links to the future, only to the past.

 

 

 

 

Blood clots: is there a there

 

A number of mostly European countries have suspended administering the Astrazeneca-Oxford vaccine in order to investigate whether it has led to blood clots. These decisions have aroused angry reactions from the media. Here's an opinion column by Prof. Spiegelhalter that captures the mainstream view.

Many reporters have mis-spoken when they say the countries have banned the vaccine. They are pausing new inoculations so that the health authorities can assess the situation.

Spiegelhalter's argument - which was also that of Astrazeneca - relies on comparing the incidence rate of blood clots among those who have been vaccinated (millions) and the incidence rate of blood clots in the general worldwide population. He estimates that about 100 people in any given week experience blood clots, while so far only 30 cases have been reported post-vaccination (across many weeks).

As with any statistical argument, we should outline the unspoken assumptions under which this argument will prevail.

First, we must assume that the system used to collect information about blood clots is reliable and mostly complete. Spiegelhalter says the "yellow card" system used in the U.K. for reporting side effects has produced far fewer side effects than during the clinical trials. He interprets this as evidence that the vaccine is even safer than advertised. Another possibility is that the side effects are being under-counted. The hard part is to figure out by how much.

Second, we must assume that the people who have been vaccinated so far is a random sample of the entire worldwide population. On second thought, it's not obvious that the rate of blood clots should be the same in every country, or every age group. There may also be seasonality in the condition - the data on the vaccine has come only from the spring season while the general data are averaged over the year.

Third, we must assume that the characteristics of the people who have reported blood clots are similar to the average person who gets blood clots in an average year. It's worth noting there are reports of unusual blood clots in young vaccinated people. The problem may be blood clots in young people, not all people.

Fourth, we must assume that all blood clots are alike. Are there different types of blood clots? Are the reported blood clots similar to the average blood clot? Depending on the answer, we may need an incidence rate for specific types of blood clots (and also for specific age groups).

Fifth, we must assume the outcomes of those who reported blood clots post vaccination are similar to those in any given year. In particular, several deaths have been reported. Is the rate of death among those who experienced blood clots abnormal?

Sixth, I'd like to tally the incidence of blood clots in the unvaccinated populations as well. According to Spiegelhalter's back-of-the-envelope calculation, the incidence of blood clots is actually far fewer than expected (only 30 cases in several weeks vs 100 weekly), and so we should expect the data to reveal that the chance of experiencing blood clots in the last few weeks was much higher among the unvaccinated than among the vaccinated - assuming those weeks were normal weeks. (If this proves true, it raises the prospect that vaccination is protective against blood clots!)

Seventh, we must also assume that there is no one-time anomalous event that is causing a spike in blood clots. Some reports claim that there might be a bad batch of vaccine. It is possible for both of the following to be true: that the Astrazeneca-Oxford vaccine does not cause excessive blood clots, and that a particular batch of the vaccine caused an excess of blood clots among those received those doses.

Eighth, we must assume that the several weeks of data collected about the vaccine are fully representative of ongoing data. There is a precautionary tale here. At any moment in this pandemic so far, if we took the cumulative data and annualized them, we would have produced horrible forecasts of the future.

***

That's a lot of assumptions and the list is not necessarily complete. What we are dealing with is another observational (i.e. real world) study. Without the randomization of randomized clinical trials, it's just really hard to connect an effect to a cause.

Making assumptions is not fatal in statistical analyses - on the contrary, every statistical argument contains some set of assumptions. If I were conducting this investigation, I'd focus on collecting data to check the above assumptions. I believe the mainstream view may prove correct; this would imply most of the above assumptions hold.

It is also true that the current known incidence rate is small, benefits outweigh the side effects, etc. Notice that this is a different line of argument. The statistical argument claims the blood clots are normal and thus there is no problem at all. This second argument accepts the blood clots as abnormal but an acceptable price to pay; it is unnecessary if one agrees with the statistical argument.

***

I don't think it's wrong for European politicians to take a cautious approach. I wish the experts were not all over the media slamming the politicians and citizens about their alleged failure to understand science - because such ferocious rhetoric is drawing unnecessary attention to what should be a routine process. Just say you support a quick investigation, Astrazeneca should cooperate and provide any necessary data, and you expect the vaccinations will resume in short order because this is likely a false alarm.

 

P.S. [3/17/2021] Just listened to the hour-long press conference by the European Medicines Agency outlining the process of the investigation. They clearly state that they don't believe the extent of thie problem would ever outweight the benefits of the vaccine, but they also say the specific circumstances of those cases require examination. They expect to come to a quick conclusion, by Thursday afternoon.

Many of the issues mentioned in this post were brought up, such as specific types of blood clots, batches of vaccines (or manufacturers), and establishing base rates for specific time frames. The committee of experts will be looking into those and other factors. One reporter asked an important question - whether there will be a biological investigation to go along with the statistical analysis.

The scientist representing EMA mentioned the back-of-the-envelope calculation that would indicate that people who were vaccinated were far less likely than normal to have blood clots. I don't think this is their strongest argument; unless the vaccine somehow prevents blood clots, you'd have expected the frequency to be roughly normal - and if it's not, you now have to go down the rabbit hole of explaining the biases in the vaccinated population, which I can assure you opens up a can of worms.

Also worth noting (which I did in a comment below) is that both sides could be right. It may be that the vaccine causes excessive, potentially fatal blood clots of a specific type in a specific subpopulation. That means the vaccine is still safe for almost everyone. This bolsters the cost-benefit argument. The end result may be just a warning about a rare side effect for people with specific conditions.

 

The wish that won't quit

 

It amazes me how the media keep inviting experts that push their gut feelings by cherry-picking only the science that confirms their beliefs. I am talking about the wishful thinking that is single-dose vaccines.

If you believe these experts, the evidence is strong, verging on undeniable, that we should skim on the Pfizer, Moderna and Astrazeneca vaccines and only administer one dose. They have most often cited the calculation by the U.K. government which I discussed here. This same argument was parroted by two Canadians in a more recent push for one dose.

The arguments for sticking with what we know are being buried. Here is an excellent summary of the main points. I covered all of these earlier than most in this blog post.

Since that time, there have been many studies that bolster the case for sticking with two doses. You've heard about these studies except they buried the lead in each case.

***

We recently heard there is a study that showed half doses of Moderna might be enough. Such a study did exist, it's an interim analysis of a small-scale randomized trial - but the big story in that study is that the antibody levels did not exceed that obtained through natural infection until after the second doses. I wrote about that study here.

We recently heard about a study in which the Mayo Clinic found 89% efficacy in their real-world study. Great news. What the media experts did not tell you is that this VE was measured from 36 days after the first dose, thus including the effect of the second dose. If measured from first dose, the VE is 60%. Therefore, real-world evidence shows that the VE of a single shot is nowhere near the 90% claimed by these experts.

We recently heard of a study involving Israel's largest healthcare provider, verifying that the Pfizer and Moderna vaccines have real-world efficacy similar to those achieved in the clinical trials. Great. But did the media experts tell you what else they found? The vaccine effectivness at 14 days after the first dose is in the 40-60% range depending on how they defined cases. It was not until after the second shot where effectiveness edged towards 90%. Therefore, real-world evidence shows that the VE of a single shot is nowhere near the 90% claimed by the experts.

***

What I wrote above are directly lifted from those scientific papers. It does not require any interpretation or analysis. I'm just relaying what the researchers said in those papers. The media and their experts are filtering out what don't fit their worldview, and you're worse off for it.

Of course, I have taken notes while reading those studies so in a future post, I'll provide an evaluation of these real-world studies.

 

 

 

How to rise above the cacophony of sometimes phony data analyses

Data analyses are flooding the airwaves. They paint a confusing, contradictory picture of all aspects of the pandemic. Sadly, most of the work are hastily put together, and low quality. Here are some things to keep in mind when you look at these studies.

What happened in an experiment will not happen again

If a vaccine trial returns a 90% vaccine efficacy, we can say for sure the vaccine’s efficacy is precisely 90% for the 30,000 or so participants in that specific vaccine trial. We are not empowered to conclude that the VE for billions of people who will eventually get the shots will be 90% - no matter how many epidemiologists repeat this falsehood on TV.

That’s because the trial involved a small sample of people, and there is a margin of error around the 90% number. If the range estimate is 70% to 98%, then we say “with 95% confidence” that the VE for the general population is above 70%. That’s still an impressive number, and it has the advantage of respecting scientific principles.

Imagine repeating the same clinical trial with different choices of 30,000 participants repeatedly. The margin of error says 95% of the series of VE estimates from these trials will fall inside the 70% to 98% range. The chance of any of these trials repeating the exact 90% VE is practically zero.

Credit: Jeremy Schultz/Flickr

 

The randomized clinical trial (RCT) is the gold standard for establishing cause--effect. The lack of randomization in “real-world” studies opens a can of worms.

In a vaccine trial, we compare the case rates of those who are vaccinated to those who aren’t, which are the primary ingredients of the vaccine efficacy formula. We are currently being served weekly new studies that also compare vaccinated people to unvaccinated people. The media are telling us that these new studies are better because (a) they are more recent (b) they have much larger sample sizes and (c) they constitute “real-world” evidence. A popular theme is that these studies fill in the gaps left by the vaccine trials.

The noise you’re hearing are the chuckles from incredulous statisticians. Since RCTs are regarded as the gold standard for causal inference, no “real world” evidence should be modifying, and certainly not correcting, findings from a scientific experiment. Doing so is like hiring a C student to tutor an A student, and siding with the C student when they disagree.

There is one critically important property missing from all real-world studies: the randomization of treatment. (See, however, so-called natural experiments.) In an RCT, a coin flip determines who gets the vaccine but in the real world, who gets the shots is anything but random. Most countries have priority lists and specific types of people are getting the vaccines earlier than others. In any “real world” study, the unvaccinated group is different from the vaccinated group, not just by vaccination status but also by many other factors, both known and unknown. Any of those other factors could contribute – majorly or minorly – to the observed difference between the two groups. Randomizing treatment ensures that on average, these other factors will not bias the finding in an RCT, a condition that does not exist in real-world studies.

A partial solution is to define a better control group. We’d take a subset of the unvaccinated people who look like those who are vaccinated. This is an effort to manufacture artificially the randomization condition. This matching process is highly subjective, and we can only match people using measurable and influential factors. There is no law that dictates that every important factor is measurable. As an illustrative example, perhaps people with higher-speed Internet connections are more likely to land vaccination slots. Since medical researchers do not have individual data on people’s Internet speeds, we cannot control for this effect. Even the most obvious adjustments have problems. For example, young people can be excluded because they don’t qualify for vaccination yet but young people who are front-line workers are getting shots.

Another common issue is incurable imbalance: if all care home residents have been vaccinated, we won't find care home residents in the unvaccinated group. The biggest problem with matching studies is lack of transparency. The typical disclosure is vague and confusing – I can’t even figure out what they did. None of these studies publish their data or code.

So, a real-world study that corrects for selection bias is better than one that doesn’t but in no case is an observational study superior to an RCT. Larger sample sizes produce more precise estimates (with lower margins of error) but they do not auto-correct biases. More precise estimates derived from biased data pose even greater risk precisely because they feel more solid when the biases are ignored.

The randomized clinical trial is a victim of abuse.

In reviewing the torrent of studies that have come out of the vaccine trials, I can’t help but notice that analysts are mercilessly abusing the RCT framework. I’m going to echo Andrew Gelman’s criticism of the standards of research in psychology. We’re concerned about the high probability of false-positive findings.

The VE numbers from Pfizer, Moderna, Astrazeneca, etc. are not comparable to each other. That’s because each team measures case rates differently. Pfizer drops all cases prior to 7 days after the second shot; Moderna and Astrazeneca drop cases prior to 14 days after the second shot. Then, you have people reanalyzing the data, who use their own case-counting windows. The U.K. government published a post-hoc analysis of the Pfizer data, counting only cases between 15 and 28 days after the first dose; this is echoed in a Canadian re-analysis of the same data, in which they set the window of 14 days to 20 days. A more recent “real-world” study out of Scotland counted from 28 to 34 days after the first dose. This is no small matter as VE is an improvement metric relative to a baseline, and we can’t keep our heads straight which the baseline applies.

Many scientists justified the choice of when they count cases, on record no less, by pointing to the cumulative case curve, arguing for counting when the vaccine’s curve starts flattening because that’s exactly how long it takes for the vaccine to do its magic. This type of post-hoc thinking is what teachers warn students not to do. These clinical trials were sized to measure the endpoint, and not sized to identify a turning point on a case curve. Imagine repeating this trial many times over. Assuming that the VE after 90 days rises to roughly 90 percent in each trial, it is likely that the turning point on the curve is not precisely at day 14 (or whenever it was in the first trial).

Analysts appear to have looked at their individual datasets, identified the period in which the vaccine showed the best performance against the placebo, defined their VE metric to hone in on that time window, and justified the definition with some sciency blather. This is a recipe for over-estimating the true performance.

The proof is the absence of new studies that go through similar post-hoc thinking to adjust down the findings of the respective RCT. In fact, the re-analyses of the Pfizer data, which moved the estimated VE of the first dose from around 50% to 90% is a nice example of a C student correcting an A student.

To prevent the danger of post-hoc theorizing, scientists agree to pre-register their studies so they have to define how they would count cases ahead of time. Unfortunately, those protocols are written loosely to allow a wide range of options, such as 7 days, 14 days, 21 days, etc. as possible evaluation metrics. This maybe exposes how little investigators know about the behavior of their inventions.

If only the start of the case-counting window were the only “researchers’ degree of freedom”. The researchers also follow their noses about (a) the duration of the case-counting window (b) whether a positive test is required to confirm cases (c) which test is allowed for confirmation and the parameters for running the test (d) the maximum length of time between reporting symptoms and testing positive (e) what symptoms are included and excluded as qualifying (f) how many symptoms are qualifying. As I showed in this post, VE drops fast with just a few more cases on the vaccine arm so any of these decisions matters.

The RCT framework does not protect us against these potential abuses. RCTs are the gold standard, but they aren’t fool-proof.

Story time after RCT

The potential problems outlined in the previous section relate mostly to trial design decisions. The vaccine RCTs are also being suffocated by post-hoc hacking. I already devoted an entire post to this phenomenon of “story time”. Not all results obtained from analyzing data from RCTs have the crucial randomization property. That’s because analysts can destroy the property by cherry-picking the data. We’re lulled into thinking these lesser findings have the same status as a proper RCT finding.

The most infamous example of such analysis is the 90% VE attributed to the so-called low-dose, standard-dose subgroup in the Astrazeneca-Oxford trial (link). This VE is indeed based on comparing case rates of a vaccine arm to that of a placebo arm. In a true RCT, those two arms are statistically the same except for what was placed in the two shots. Nevertheless, the low-dose subgroup happened as an accident, and it subsequently emerged that this subgroup contained only younger participants, have earlier enrollment dates, have longer intervals between doses, etc. And yet, on the front page of the Lancet paper summarizing this trial, the investigators stated “In participants who received two standard doses, vaccine efficacy was 62.1%... and in participants who received a low dose followed by a standard dose, efficacy was 90.0%...” By this point, there has been no disclosure about the “accident,” and anyone reading this sentence is likely to assume that there were two randomized dosage schedules used in this trial.

Likewise, much ink has been spilled on dose intervals. Many analysts have compared subgroups that took their second shots later and earlier. Even though the underlying data were collected in an RCT, dose intervals were not randomized. And yet, these results were published and publicized using methods for analyzing RCTs and not methods for analyzing observational studies. In some cases, the researchers even disclosed that these subgroups differed on numerous important factors, and still they plodded on.

**

In this era, even results coming out of RCTs must be scrutinized. Early analyses, post-hoc analyses, deep-dive analyses, and side analyses typically discard the crucial randomization property, and should be treated as observational studies.

Of course, results from observational ("real-world") studies should be scrutinized even more. In a future post, I’ll outline how to review analyses of observational data.