Big Data, Plainly Spoken (aka Numbers Rule Your World)

When you don't need data

I bet you can't believe I'm going to say I don't need/want data on a blog about intelligent thinking about data.

I'm going to argue that intelligent thinking about data includes recognizing when you don't need data - and by extension, you don't need more data.

This post is motivated by opinions that are circulating about how certain public health measures against infectious diseases such as masks and lockdowns are 100% useless - although if you start thinking about other domains, you can find pertinent examples.

Let's start with lockdowns first as that's more black and white. I don't need any data to believe that lockdowns reduce infections. I'd need data if I want to establish the magnitude of protection but for the direction, I don't need data. Why? Because I have theory.

What theory? Infectious diseases spread by infections, which requires contacts. Turning this around, one can say no infection can occur unless there is contact. During the early part of the pandemic, and especially before vaccines were available, in key hotspots, schools and businesses were closed; people were asked to stay home; employees were allowed to work from home. As a result, the frequency of contacts each of us had with others was drastically reduced, and for some, reduced to near zero. By the theory, the dramatic reduction of contacts resulted in a reduction of infections.

Do I need data to prove that statement? No. [The word "theory" is weird. By theory, I mean an immutable phenomenon, such as incorporating the law of gravity in a physical model. There's probably a better word for this. Structural model?]  

***

Would I like to have data? Sure. But it's not a must-have. If I had data, I hope to answer more questions, like how much reduction? does it affect all demographic segments equally?

Would I not want to have data? There is even an argument for this (although this sentiment won't be universally shared). I suspect strongly that any data that could be made available to me would be almost impossible to interpret - because we cannot run randomized experiments on lockdowns. Thus, it would take a lot of effort to clean the data, and to adjust the data, and none of this can be accomplished without making lots of subjective assumptions, sure to enrage some and delight others.

For example, all such data confound the effects of lockdowns and vaccinations because most places that had lockdowns also pushed vaccines simultaneously. What's more, masking, social distancing, and other measures were also simultaneously put to work. So, if someone had the data, it is likely to confuse rather than illuminate. Let's not forget about enforcement of lockdowns, and compliance to lockdowns. Contacts might not have been reduced if the lockdown was in name only!

So if I had data, it could show infections going up, down or sideways, and the effect of lockdowns would be masked by all the other different factors. More data could be useless.

***

More data, however, could not overturn my theory... otherwise, my theory is just conjecture. If the data happened to show that, after adjusting for all kinds of other factors, lockdowns have zero or negative impact on infections, I'd be much more likely to reject some of the subjective assumptions or the indirect data proxies, rather than to question the theory, which in this case, means I have discovered evidence that infections occur without contacts.

Even if surveillance data show that compliance to lockdowns was nonexistent, the proper way to interpret the data is to say that lockdowns by themselves should reduce infections but its effect is masked by compliance or lack thereof, so that data that confound both factors show that infections did not drop. Embedded in there is still the theory that reduced contacts reduces infections.

The theory - the structural model - is an immutable part of the larger statistical model. We expect data to conform to this structure. We don't use data to modify the theory. (This structural model is not the same thing as a prior in Bayesian models. Bayesian priors are subject to updating on observing data so it does not represent anything immutable.)

***
There are other examples in other domains as well. Here is Andrew's model of golf putting (link). The section called "modeling from first principles" is an example of a statistical model that embeds a (geometric) theory. He also later uses the term "geometry-based" model.

The simple geometry is not enough, and further iterations of the model add other factors but the original geomtry is still in there.

***
This post records my first thoughts on this topic. I'll get to masks next.

Strategies for validating observational studies 1/2

As mentioned before, the team who did the research tying Covid-19 vaccinations to car accidents convinced themselves that they have found a real effect. They attempted a variety of confirmatory analyses. In the prior post, I described how they conducted variants of the basic analysis by analyzing subgroups and by modifying the outcome metrics.

It's instructive to look at several other validation strategies and discuss their strengths and weaknesses.

***

Every observational study delivers the following table, usually called Table 1 because it's the first table in the paper:

Table 1 compares the characteristics of the treatment group and the control group. In the traffic accidents study, the two comparison groups are the vaccinated and the unvaccinated residents of Ontario (after exclusions). For a randomized clinical trial, in which vaccination status would be randomly assigned, the two columns of Table 1 should look statistically the same. For an observational study, in which vaccination status is self-selected (or influenced by biases such as public-health priorities), the differences between the two columns provide hints as to the magnitude and direction of biases.

The first thing to notice is vaccinated residents outnumbered unvaccinated about 5 times. The features are divided into three groups: demographics, prior medical conditions (e.g. diagnoses of cancer in the past year), and something labeled "general". The "general" collection includes clinical contacts, hospital admissions, and emergency room visits in the prior year. These are important variables because prior health status is a likely source of self-selection bias.

The researchers have this to say about Table 1:

The 2 groups spanned a diverse range of demographics, with comparable general health care utilization (Table 1). The largest relative differences were that those who had not received a COVID vaccine were more likely to be younger, living in a rural area, and below the middle socioeconomic quintile. Those who had not received a vaccine also were more likely to have a diagnosis of alcohol misuse or depression and less likely to have a diagnosis of sleep apnea, diabetes, cancer, or dementia. About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

This language is highly typical of how researchers opine on the likely sources of bias, i.e. those characteristics that differ between the comparison groups. Now pay attention to the last sentence: About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

The actual data from Table 1 show 4.1% of vaccinated people had a documented Covid-19 infection in the past year, compared to 3.5% of unvaccinated. That's a relative ratio of 1.17. Given the large sample size (millions of people), a 0.6% difference in infection rates is highly significant! If we randomly selected 4/5 of the population to be vaccinated, the difference in infection rates between them and the unvaccinated would typically be much smaller than 0.6%.

So the claim of no major imbalance on Covid-19 prior infection is incorrect. It is also inconsistent because in the main analysis, based on 95% confidence intervals on relative risk, they declared they have found a significant correlation between vaccination status and traffic accidents. The same method would have found the difference between 3.5% and 4.1% prior infection rate to be highly significant. Indeed, if we apply the same method to all rows of Table 1, quite a few would also be found significantly different.

The prior-year emergency visits metric is of particular interest because the primary outcome being analyzed in this study was "emergency visits for individuals injured in traffic crashes". The unvaccinated group had a much higher prior rate of 26% compared to 20% for vaccinated, a relative ratio of almost 1.3, clearly significant. The analysts did not adjust for any of the variables in the general collection in their logistic regression, perhaps because they thought the two groups have "comparable general health care utilization," which as I just explained, is incorrect.

It's ok - in fact, expected - that the unvaccinated group would differ from the vaccinated group on some metrics. It's the opposite that should raise doubt. If the two columns are effectively equivalent, then the two comparison groups resemble those found in randomized trials. If we believe that, we're claiming that obvious sources of bias due to self-selection and public-health policy have not resulted in observed bias.

The biggest weakness of Table 1 analysis is missing variables. The comparisons are made only on the measured covariates. In the case of Covid-19, the measured covariates are whatever can be found in large government databases, collected for purposes other than conducting a study of traffic accidents and Covid-19 vaccination. The researchers said: "The available databases lacked information on driver skill, functional status, personality traits, traffic infractions, political affiliation, and self-identified ethnicity." Several of these variables are clearly pivotal for our present study.

In addition, Table 1 analysis addresses only biases acting independently. Some biases act in complex ways. When I analyzed prior emergency visits, the observed data appeared to indicate a healthy user bias in the vaccinated group. But this factor interacts with the age bias, which was also present. The vaccinated group was quite markedly older than the unvaccinated group. If older people are more likely to need emergency medical service than younger, then the emergency visit bias is even more pronounced since younger, unvaccinated people have a lower baseline emergency room usage than the older, vaccinated group. If younger people are more likely to need emergency medical service older, then age could explain some of the observed bias in emergency room usage. In other words, these factors should be interpreted simultaneously. In fact, any adjustment in the regression model should contain interaction effects, but this is rarely done.

***

The researchers also used "positive controls" and "negative controls" to convince themselves they have properly removed bias from this observational dataset. The idea is to process the data in the same way but substitute other metrics instead of the traffic accidents outcome. These other metrics are chosen because the researchers are 100% sure of the direction of the effect: for positive controls, they claim they knew ex-ante that the alternate outcomes are definitely affected by Covid-19 vaccination status while for negative controls, they claim they knew ex-ante that the alternate outcomes are definitely independent of Covid-19 vaccination status.

For negative control, they presented an analysis of constipation.

There were 2,985 cases of constipation during the month of August in the study population, and the relative risk was 1.03 meaning the unvaccinated group's risk was 3% above that of the vaccinated. This difference is not significant because the confidence interval of 0.94 to 1.14 crosses the 1.00 reference level. In other words, the data provided no evidence that unvaccinated people differ from vaccinated people in terms of experiencing constipation.

This type of validation analysis has many shortcomings. First, it's a single outcome. We don't know if this was pre-specified. Even if pre-specified, it could still show the desired result by chance since there are hypothetically hundreds or thousands of possible negative control candidates. If it's not pre-specified, it's hard to know if other candidates were tried and failed the test.

Perhaps because of reviewer feedback, the research team added several other negative controls, namely, fall, water craft, appendicitis and conjunctivitis. Perhaps because these were not in the original analyses, they did not discuss these other negative controls in the article.

I excerpted the table of results below.

The last column presents the confidence interval of the relative risk, and the second last column from the right shows the point estimate of the relative risk.

The second problem is inconclusiveness. Notice the point estimates of all five negative controls are tightly clustered between 0.90 and 1.14, and every confidence interval except "falls" shows a statistically insignificant difference between the vaccinated and unvaccinated groups. How should one interpret the result of "falls"? The researchers were silent on that one. When a negative control (constipation) shows insignificance, as expected, they were quick to say this finding validated the study conclusion but when a negative control shows a significant difference, it appeared that they explained it away, probably arguing it's a chance observation.

The third problem is over-confidence. How do they know that vaccination could have zero impact on constipation? I imagine this relies on some kind of structural biological argument, such as that mRNA does not act on certain bodily systems. Remember that statistical analyses describe correlations, not causation. We have to rule out ex-ante not just a direct causal relation between vaccination and constipation but any kind of correlation. That's actually a deep assumption, not as simple as it looks.

***

Positive controls are even harder to grapple with because we have to believe ex-ante that vaccination status will certainly result in some given outcome, in this case, "covid pneumonia".

The use of "covid pneumonia" implies the absolute belief that vaccinations reduce covid pneumonia in the analysis population. A few aborted clinical trials and many observational studies do not constitute incontrovertible evidence. Imagine if the data showed no significant difference in covid pneumonia between the unvaccinated and vaccinated. What would be the conclusion? That the methodology is invalidated? That the assumption of causation was incorrect?

These validation analyses using positive or negative controls have limited utility. They produce a situation in which the only possible conclusion is supporting the study - if these analyses do not support the study, we don't know how to interpret them so we throw them out and find another metric to act as control. (As a practical matter, no one would publish the paper if the authors include "failed" controls. See my prior post on the file drawer effect.)

***

The next tactic used for validation is propensity score matching. I'll discuss that in the next post.

Prior posts about this study: 1, 2, 3

 

 

Too good to be true instinct

Good data analysts have developed an instinct of perking up when results are "too good to be true". This is a must-have attribute that I'm always looking for in a job candidate. It is a protective instinct, a response that protects the analysts from getting fooled by the data. Nassim Taleb called this type of thing "fooled by randomness". The temptation is not randomness itself but the lack thereof.

(Credit: Sonja Alves, Flickr)

Observational studies of the Covid-19 vaccines are a case in point. I have been spooked by the consistency of estimates of vaccine effectiveness. Almost all peer-reviewed studies and most working papers that get mainstream attention contain values of VE that are in the 70-90 percent range. (Since these are relative ratios, a 50-percent multiple improvement is already a huge effect.) There is a consistency across headline results of different studies, and there is also a consistency of values inside individual studies, across segments, using different outcome metrics, different methodologies, etc. etc.

Below is Table 2 from the traffic accidents study that I have been reviewing (link):

The aggregate result (derived from the inadmissible non-adjusted comparison discussed in the first post of this series) is shown in the last row and represents a value of 1.7x. The rows above come from applying the same methodology to subsets of the study population, some of them are mutually exclusive while others are overlapping. Most of the values line up close to or near that 1.7x value. There is only one segment (over 65) that has the opposite direction but barely. What does this consistency tell us?

Shown below is the top part of Table 3. Here, the researchers analyzed many variants of the outcome, applying the same analytical strategy. Whereas Figure 2 above shows any involvement in traffic accidents, Table 3 below breaks down these accidents into subtypes, like involvement as driver versus pedestrian, time of day and so on.

The consistency is once again remarkable. Whereas the aggregate number is shown as 1.7x (top row), all the other numbers ranged from 1.5x to 5.1x. Almost all the variance is on the high side of the aggregate. What does such consistency tell us?

In the Appendix, the authors rolled out even more rows of analysis:

Again, all effects are in the range 1.4x to 1.7x. What does this consistency mean? That's what I want to explore in this post.

***

In the mainstream, such consistency is portrayed as resounding evidence. Each individual study that generates another impressive relative ratio similar to the previous result is regarded as another confirmation. The authors of the traffic accidents study believe that because everywhere they looked, they saw the same consistent number, it must mean that they have found something really important - like Newton's Law of Gravity.

Unlike Newton, they aren't studying the realm of physics. The study's data concern human behavior, and social behavior, and it is actually shocking, rather than comforting, to find a law of nature in this arena. My prior - from decades looking at data about humans - is that (a) aggregate effects are typically small, so that even a 10% improvement is remarkable, and (b) there is substantial variability when looking across different segments, different metrics, etc., which is another way to say that "interaction effects" are important.

Were these researchers, mainstream scientists, and the mass media being fooled by (lack of) randomness?

***

Other than discovering some immutable law of nature, what are other possible explanations for the remarkable consistency?

File Drawer Effect

Let's say there is a medical journal editor who only publishes vaccine studies that show high vaccine efficacy - since anything else has little societal value. The result of this editorial practice is that (almost) every study that sees the light of day will definitely have a VE estimate above some lower bound (say, 70%). In this scenario, the consistency of all the peer-reviewed results may have nothing to do with the vaccines themselves, but can be explained by publication bias that is filtering out studies below that bound. This bias is sometimes called the "file drawer effect," describing researchers who stack away "failed" studies (e.g. p-value > 5%) in their file cabinets since journals would reject them.

A consequence of the file drawer effect is that the average of the published results over-estimates the true vaccine efficacy.

In the context of Covid-19 vaccines, the file drawer effect may be related to the phenomonon of cargo cult science.

 

Cargo Cult Science

Cargo cult science is a term coined by physicist Richard Feynman. Roughly speaking, he warns against the potential of bad science even when scientists follow so-called best practices.

Other than the initial clinical trials conducted in 2020 (which were scientifically aborted upon emergency use authorization when the placebo groups were destroyed), all evidence of Covid-19 vaccine efficacy have come from observational datasets. All observational data sets either lack controls, or if controls existed, treatment status was not randomly assigned, which means that unvaccinated people differ from vaccinated people in multiple dimensions besides vaccination status. Therefore, it is wrong to attribute all observed differences between the two groups to vaccination only.

Let's teleport back to the first part of 2021, weeks after the mass vaccination campaign commenced, when the first observational study appeared in print. For the sake of argument, let's say the first analysis showed a VE of 50%. How do we know whether this estimate is to be believed since the people who got vaccinated were clearly different from those who didn't? The honest answer should be no one knows because there is no ground truth in an observational study.

What circumstantial evidence do we have? The only breadcrumbs are interim analysis results from randomized clinical trials that produced VE of 90%+. We face a gnarly problem because our hypothetical observational study led to a VE estimate of 50%. One of the numbers is not like the other; they cannot both be correct. But which study is wrong?

Not surprisingly, most would give more credence to the RCT results, since RCTs are considered the gold standard of medical research. Therefore, since our observational study came in much lower, we'd go back to the drawing board, and massage the dataset some more... until the estimated VE is closer to 90%, making it more credible. The additional data massaging is justified by best practices - we are simply addressing residual confounding bias.

(It is important to realize that randomized experiments can be analyzed in a straightforward, standardized manner but observational studies rely on statistical adjustments that involve subjective judgment from researchers. Analysts try to guess what biases exist in the data, and device ways of correcting for them. It's an iterative process with limited guardrails.)

To make sure you get what's going on, think about what happens if the first analysis of an observational dataset leads to a VE value of 90%. This result would be immediately accepted. Nevertheless, it's not clear that this research team did a better job adjusting for bias than the previous team who came up with 50%.

Since the RCT result was the only piece of available evidence when the first observational study showed up, it became a filter for all subsequent observational studies. Any observational study that produces a number close to 90% feels comfortable, and any study that shows a number far from 90% feels "wrong". Such an editorial process directly triggers the file drawer effect.

***

I don't know how we landed on this stupid state of affairs. Using an observational study to "confirm" an RCT result - the gold standard - is like asking a C student to grade an A student. After the teacher has already given the A student a top grade, what additional information is gleaned by having a C student grade the A student's exam?

What's happening is even more dubious. The C student is not grading the same exam that the teacher saw - the C student is asked to grade a different exam submitted by the A student. By this, I mean: the observational studies are different from the RCTs in meaningful ways, e.g. timing of vaccinations, incidence of Covid-19, types of people eligible for vaccination, residence of patients, no placebo shots... There is scant justification for assuming that the observational studies should produce VE values in the same ballpark as the RCTs!

When another observational study comes out proclaiming 70-90% VE, it doesn't mean much. It just tells me that the research team has selected a set of adjustments so that the VE estimate lands in the zone that is considered acceptable for publication, i.e. it falls in the vicinity of the RCT result.

 

Lurking Variables

Another way in which we can get fooled by the consistency of results is via unknown or unexplored sources of bias. All observational studies can only control for factors for which there are measurements. I haven't encountered yet an observational study of Covid-19 vaccines in which the researchers collected their own data. The current practice is to pull data from pre-existing, government databases. So the variables available are typically limited to basic demographics (age, gender, etc.) or health-related data (prior health concerns, healthcare usage, etc.). Important factors such as attitudes toward freedom and government authority, risk tolerance and exposure to the virus, are not measured, and not adjusted for.

If key sources of bias are not adjusted for, they could explain why the results are so consistent among the subgroups; each such result contains residual confounding from the missing variables.

The missing variable may be a common cause of both vaccination status and infection outcome, for example, people who are generally more health conscious and healthier pre-Covid. They are both more likely to get vaccinated and less likely to get infected (regardless of vaccination status). Since the government databases have no labels for this type of people, there is no way to correct this bias. It is delusional to think that dumping age and gender variables into a regression equation properly adjusted for such a bias. There are health-conscious people in every agexgender segment! In each such demographic segment, vaccination status is still confounded with health-seeking status.

 

Multiple Comparisons

When so many analyses are performed using so many metrics, the best practice is to adjust for multiple comparisons. The general idea is that if one screens a lot of metrics, a few of them will accidentally show statistical significance, and those results are highly likely to be spurious. In other words, without adjustment, traditional statistical methods underestimate the degree of variability in the outcomes. That is to say, results appear more consistent than they are. Yet, most of these vaccine studies sidestep this issue.

I'd be shocked if elementary medical statistics courses don't cover the problem of multiple comparisons. Practitioners in this discipline seem to think that fishing for significance is fine so long as they pre-specify all their intended analyses. Actually, pre-registration does not solve the problem of multiple comparisons at all, unless the pre-registered statistical analysis explicitly outlines a strategy for dealing with it. (Unfortunately, the randomized trials also pre-registered multiple comparisons without correction.)

 

Poorly Defined Metrics

Another reason for consistent results across many analyses is a poorly chosen scale for one's outcome metric. An analogy to grade inflation in schools is telling. American schools has a grade-point average (GPA) typically defined on a scale of 0-4, 4 being grade A. Due to the grade inflation trend, most of the grades given out are As so in reality, the majority of the scale 0-3.5 is basically empty, with almost all students squeezed into the 3.5-4.0 range. Of course, all students look more or less the same.

I think something like this is at play with the relative ratio scales used to measure vaccines. I think an ineffective vaccine will show a VE substantively higher than 0% so the grading scale is compressed just like in schools.

***

In this post, I have offered five possible reasons why every research study of Covid-19 vaccines seems to offer similar effectiveness estimates, and why there appears to be little or no variability when aspects of the methodology are altered. As a statistician who have analyzed a lot of human data, I find it least likely that the reason for the consistency is that these vaccines are like a law of nature, such an overwhelming force that flattens all human variability.

So what if rule-breakers are unvaccinated

In the last post on the Covid-19 vaccine and traffic accidents study (link), I reviewed the analysis methodology: I found it hard to believe the vaccinated and unvaccinated groups differ only by their vaccination status, even after “adjustments”, and thus I am not convinced by their causal interpretation.

In this post, I examine a few issues.

Right at the start, the research team proclaimed:

The proximate causes of most crashes are human behaviors including speeding, inattention, tailgating, impairment, improper passing, disobeying a signal, failing to yield right-of-way, or other infractions.

This assertion is pivotal to the research conclusion, which they exploited by using involvement in traffic accidents as a proxy for rule-breaking behavior on the road. The researchers speculated that an underlying trait of rule-breaking is the common cause of (a) a higher risk of traffic accidents and (b) more vaccine hesitancy which results in a lower chance of being vaccinated.

The causal diagram – even if true – can have few public policy implications. It explains a plausible self-selection bias in the unvaccinated group (inherent rule-breaking personality). It may suggest that if governments successfully “cure” this bias, then we’d simultaneously observe not just higher vaccination rates but also a reduction in traffic accidents, other traffic violations, other crimes, absenteeism in schools, fare evasion in public transport, tax evasion, etc. But solving general unruliness appears to me a much harder problem than solving vaccine hesitancy!

The claim that most traffic accidents are driver-caused is supposedly supported by a third-party reference but that study tallies only traffic accidents involving fatalities (which comprised fewer than 0.2% of the outcomes in this study), and even so, about half of the documented fatal accidents occurred under “normal” conditions, without any rule-breaking.

The causal proposal is further weakened by linking individuals to traffic accidents even if they were pedestrians or passengers, rather than drivers. These people – who could not have caused crashes through undesirable behavior – accounted for almost 60% of all outcomes.

Imagine a drugged out but unvaccinated driver carrying one passenger in the car ran a red light, and hit and killed a pedestrian. All three individuals are counted as involved in traffic accidents. The study authors would have concluded that the unvaccinated driver would not have crashed the vehicle if s/he had been vaccinated. By contrast, if the pedestrian was unvaccinated, then the study authors would have us believe that s/he would not be dead if vaccinated, regardless of whether the driver was vaccinated!

It’s a mess. It’s easier to avoid such a mistake if one collected the data through shoe leather.

***

One factor that drives vaccination behavior has received scant attention – risk tolerance. It’s well known, and not just among economists or psychologists, that humans have varying levels of risk-seeking or risk-averse behavior. Much of the insurance and gambling/gaming industry is built upon such heterogeneity. We also know that the measured risk of Covid-19 infection or case severity depends substantially on key demographic factors such as age, gender, occupation, and prior comorbidities. So it should be obvious that propensity to want the Covid-19 vaccine should vary with (a) predictable risk of infection and/or case severity (b) individual level of risk aversion and (c) individual’s assessment of probability of exposure, including the interaction between these factors.

The public-health narrative tends to adopt an axiom that everyone is alike, and that may be its biggest failure. As noted in the prior blog, the peer reviewers and the study’s researchers failed, by neglecting the most obvious proxies for the above factors – prior traffic accidents, and prior healthcare usage. They have no excuse for excluding these factors because the former was computed for the follow-up period while the latter was used in an exploratory matching analysis. The research finidng will be seen in a new light if it turns out that the unvaccinated group had been involved in more traffic accidents prior to the study period, for example.

***

All over the paper, risk is given in the form x per million chance of getting into traffic accidents. For example, the risk for unvaccinated was described as 912 per million, and for vaccinated, 530 per million. These are tiny numbers made to look huge. Since there were 1.8 million unvaccinated, the ~900 per million means about 900*1.8 = 1,620 unvaccinated people were involved in accidents, while 500*9.2 = 4,600 vaccinated people were involved in accidents. In what period of time?

Glad you asked :) The study has a follow-up period of just one month (30 days)! Literally, the research only covered the month of August, 2021. They did not look at any other month. (It’s quite likely there would be some reversion to the mean over time.)

912 per million is another way of saying 0.0912%. Roughly 0.1% of unvaccinated people were involved in a traffic accident during August 2021, and roughly 0.05% of vaccinated people were involved in a traffic accident. Only 16% of the population were unvaccinated. Even if we assumed correlation is causation, and if we forcibly vaccinated everyone, the impact of such a policy is 1.8 million * (0.05%) = 900 people. It’s 900 people out of 11 million, and remember, some of these are pedestrians or passengers, not drivers. Most of these accidents have zero fatalities.

This number represents the risk in a single month, not annualized. There is no explanation in the paper as to why such a short follow-up time was applied. Perhaps it has to do with the continuing vaccination campaign, so that the number of unvaccinated – and thus potential beneficiaries – keeps declining. (For those interested, none of the main results in the study censor or remove unvaccinated people who subsequently got vaccinated during or after August 2021.)

***

As with most Covid-19 observational studies, researchers selected a start date for counting outcomes, and didn’t bother to demonstrate that their findings are invariant to the choice of such a date. In this study, the follow-up start date was set as July 31, 2021, so the follow-up period comprises the single month of August 2021.

This is a strange date on which to settle. Take a look at the case curve in Ontario:

At the end of July 2021, cases in Ontario were at a trough, and during August, Covid-19 cases were steadily growing. This entire month missed the next peak of infection, which was toward the end of the year. In each Covid-19 wave, hospitalizations and deaths lag cases. The following curve shows Covid-19 deaths in Ontario:

Thus, many of the deaths associated with cases reported in August showed up after the 30-day follow-up window (in September). We should ignore any commentary on hospitalizations and deaths - specifically, the claim that this study “validate[d] that vaccination is associated with large reductions in subsequent COVID pneumonia.”

The authors pointed to Table 3 to support the COVID pneumonia claim. There, we find a total of 5,358 cases. This number is five times higher than the hospitalization count shown in the above chart for August 2021 (31 days x < 30 per day), implying less than 20% of patients with COVID pneumonia are hospitalized. This contradicts the description by Cleveland Clinic that says "If you’re diagnosed with COVID pneumonia, it’s likely that you’ll be admitted to the hospital." (link).

What else happened during August 2021? The Ontario government started to differentiate testing policies between vaccinated and unvaccinated mid-month (link). In some settings, only unvaccinated people were subject to routine testing. The more tests, the more cases found.

What else also happened during August 2021? According to this press release, the provincial government issued a vaccine mandate for "high risk" settings, made third doses more widely available, and expanded Pfizer vaccine to younger children. These policies create selection bias in the analysis groups.

The timing issue is an endemic problem with all Covid-19 observational studies. Researchers should repeat the same analysis methodology applied to different time periods and show us the results. I have a hunch that many of these results are transient.

 

P.S. [1/11/2023] Added link to the original study

The traffic accidents study exposes the lamentable state of Covid-19 science

tldr; The recent study tying lack of vaccinations to traffic accidents illustrates everything that is wrong with the state of our peer-reviewed scientific process. The study uses data that do not capture any of the important behavioral factors driving one's vaccination status and propensity to traffic accidents, and yet draws conclusions as if the biases in the observational data have been sufficiently cured. The researchers ignores some findings that contradict their headline and which reveal shortfalls of their methodology. These inconvenient results suggest that their regression model did not adequately correct the biases. The model structure does not align with the hypothesized causal structure, which in any case is far too complex to be handled by a basic regression. Worse of all, the study foregrounds an entirely inappropriate analysis, on to which reporters latched. Peer reviewers somehow looked the other way.

Late last year, several readers prodded me to blog about a sensational study urging people to take Covid-19 vaccines in order to prevent traffic accidents (link). Indeed, I had already noticed the breathless headlines, and flagged the study as suspected “science for emergency use” (i.e., not the usual science).

Now that I’ve read the paper, I confirm the initial suspicion. The study is not frivolous, as these researchers made efforts to strengthen the evidence on a finding that is predictably controversial. Nevertheless, they fell too much in love with a headline grabber, and missed some red flags.

As with most Covid-19 observational studies, these researchers gained access to a set of government databases – vaccinations, Covid-19 cases, demographics, traffic accidents, etc., linked them together, selected a “test group” and a “control group”, and performed several statistical analyses, ranging from simple tests of proportions to regressions with simple adjustments. Reflecting the preference during pandemic times, this type of research is conducted purely remotely, in an arms-length manner, without any engagement with the persons or objects under study. It’s quintessential ivory-tower data mining, no use for shoe leather.

For this study, the researchers obtained “N=All” data from Ontario, Canada. They selected about 11 million people to include in their analyses, divided into those who have taken zero vaccine shots (“unvaccinated”) and those who have taken one or more vaccine shots (“vaccinated”). The unvaccinated group comprised only 16% of the analysis population at the time of the analysis.

The researchers chose to highlight their least convincing result, perhaps because it carries the biggest “effect”. This is the claim that “unvaccinated individuals [experienced] a 72% increased relative risk [of traffic crashes] compared with those vaccinated”. This claim orginates from a simple, unadjusted comparison of the rate of traffic accidents between the two groups.

This simple, unadjusted comparison is completely useless! It is valid only under the obviously false assumption that the only difference between the “test group” and the “control group” is their vaccination status. Nevertheless, reporters latched onto it like sticky glue. I like to use the term “XYopia” (i.e. X-Y myopia) to describe the fallacy of pretending that the observed outcome Y (traffic accidents) is affected only by the offered explainer X (vaccinations) and nothing else.

In fact, the observed difference in traffic accident rates could be partially or wholly due to any number of other factors. After all, this type of research study is an exercise in data mining, not a carefully designed clinical trial. In every province and country, the unvaccinated group is expected to vary from the vaccinated group, not just in their vaccination status but due to self-selection and local, discriminatory vaccination policies. Thus, any unadjusted analysis ignoring other factors is pure garbage.

Why did the researchers let this happen? It’s not like they can’t smell the stench from a mile away. This is elementary stuff taught in the first lecture on observational studies. The paper actually included several adjusted analyses. Maybe they will argue that the simple analysis was “pre-registered”, and thus according to the norms of scientific publishing, they are “obligated” to headline it. The protocol is not published so we don't know. If true, this situation exposes the Achille’s heel of using “pre-registration” to preempt bad science – since patently absurd statistical methods could be pre-approved. (Besides, in many other Covid-19 studies, including trials, the protocols were continuously "revised" up to the time of analysis, rendering pre-registration toothless.)

After adjustment, the relative risk increase dropped to about 50% (the adjustment is insufficient, as indicated below).

***

The researchers are well aware that their finding would be controversial, obviously a prime example of correlation, not necessarily causation. This is also where "causation creep" shows up. Just like the majority of such studies, the authors clearly state that "correlation is not causation", but their conclusion - "Together, the findings suggest that unvaccinated adults need to be careful indoors with other people and outside with surrounding traffic" - makes sense only under a causal interpretation of the observed correlation.

Think about the situation faced by all research on observational data. If the finding is merely an interesting correlation, there is no reason to publish it, as the correlation could be spurious. The only studies that could get published are the ones in which the researchers somehow turn that observed correlation into causation. What's strange and wrong in my opinion is the declaration that they have found only a correlation when they have convinced themselves (and peer reviewers) that they have offered sufficient evidence of causality. If that's what they believe, they should own it and say it clearly.

To strengthen their own conviction of causality, the researchers sliced and diced the data some more, and found a similar signal almost everywhere. Instead of ringing alarm bells, such unlikely consistency is interpreted as confirming the aggregate result. But the signal isn’t “everywhere”, only everywhere they looked. If an overlooked factor was driving the aggregate result, it may well drive all the subgroup analyses, with the neglected factor cutting across all the known subgroups.

For example, in Table 2, the analysts replicated the simple unadjusted comparison of rates for a variety of subgroups, such as males, younger or older age, people with cancer, etc. We learn that among those older than 65 years old, those unvaccinated have a 30% lower risk of traffic accidents than those who are vaccinated. (This finding, just like the other findings, assumes all else is equal, which is obviously false.) Applying the same reasoning these authors used to interpret the 50% overall percentage increase in risk, we’d have to advise Ontario residents above 65 years old against getting the Covid-19 vaccine as doing so may make them more vulnerable to traffic accidents! This effect remains even after adjustment.

Next, in Table 3, the researchers performed the simple comparison of vaccinated vs. unvaccinated on subsidiary outcomes. One such outcome was involvement as pedestrian (rather than driver or passenger). Amusingly, being unvaccinated increased the risk of getting hit by vehicles by 40% (p = 0.00). Moreover, not getting the vaccine more than doubled the risk of traffic accidents *at night*.

Any attempt to explain away these inconvenient findings will involve invoking some other factors e.g. maybe unvaccinated people are more likely to have a night life – such theorizing does not rescue the analytical model but merely proves that it failed to account for all relevant factors.

The wayward results were disclosed in the paper – the research team chose to look the other way, focusing on the outcomes that support their headline-grabbing conclusion. But these results demonstrate that the "adjustments" applied have failed to cure the selection biases in the observational data.

***

It comes as little surprise that the simple adjustments do not cure the selection biases. (And when such biases are not cured, they could infect any analysis.) This problem is endemic in the Covid-19 literature. Most studies - including this one - rely on basic regression models that include just main effects of various demographic variables (age, gender, location, etc.). Very few of these studies publish the actual models - almost never do we see any tables of coefficient estimates or goodness of fit statistics. Almost no one discusses the causal structure associated with the structure of their regressions. Nevertheless, they all proceed as if their adjusted models have completely cured any selection biases.

The regression model in the traffic accidents study accounted for "baseline demographic and diagnostic predictors", which are not explicitly listed. Presumably they contain the items in Table 2: age (<40, 40-64, >65), gender, urban/rural, socioeconomic status (high, medium, low), prior alcohol misuse, prior sleep apnea, prior diabetes, prior depression, prior dementia, prior hypertension, prior cancer, prior Covid-19 infection.

Missing from this list is the most obvious one: prior traffic accidents. Another obvious one is prior vaccinations. I'd also include prior traffic violations. Yet another relevant factor is prior hospitalizations or hospital contacts. I'm not sure how peer reviewers missed all these.

This last factor is clearly important - but most of the Covid-19 researchers make a ridiculous decision - that is, they excluded people with recent hospital contacts, which improperly injects a healthy person bias into all such studies. This factor should not be used as an exclusion criterion but should be an adjustment variable.

The study's appendix includes a very telling causal graph:

I'm not going to adjudicate as to whether this diagram properly captures the complex relationships between these factors, nor whether it includes all relevant factors. Just focus on the white nodes - these are the only factors/outcomes that have real-world measurements, and are represented in the regression model. Everything else is not directly observable.

The regression model ignores all the unmeasured quantities so it is inconsistent with the above diagram. And yet, they assert that all biases have been cured. Alright - they didn't really claim all biases have been cured. In fact, they compiled a long list of factors that they knowingly ignored. In describing the "limitations" of the study, they listed the following factors: distrust of government, belief in freedom, misconception of everyday risks, faith in natural protection, antipathy toward regulation, chronic poverty, exposure to misinformation, insufficient resources, other personal beliefs, political identity, negative past experiences, limited health literacy, social networks, amount of driving, vehicle factors (speed, spacing, configuration, location, weather, distance driven), access to care, risk compensation, non-reported crashes, etc.

The researchers' stance is that since they don't have data for these factors, they are left as homework exercises for future research. That regrettably is normal research practice: even though many important factors are ignored, the study is still publishable since the researchers acknowledge the "limitations". It's like an investment banker disclosing all the reasons why a tech startup would tank, while still selling the IPO stock fervently. Here is a sample from Twitter's 2013 IPO that is good for giggles:

• "Our products and services may contain undetected software errors, which could harm our business and operating results."
• "We rely on assumptions and estimates to calculate certain of our key metrics, and real or perceived inaccuracies in such metrics may harm our reputation and negatively affect our business."
• "We have incurred significant operating losses in the past, and we may not be able to achieve or subsequently maintain profitability."
• "If we fail to grow our user base, or if user engagement or ad engagement on our platform decline, our revenue, business and operating results may be harmed."
• "If we fail to effectively manage our growth, our business and operating results could be harmed."

Maybe it's just me - if I believed those risk factors (plus many others), I wouldn't buy this stock. But of course, I wouldn't buy any tech stocks since they all use similar language.

If had read the limitations of the traffic accidents study first, I'd have saved a lot of time by not reading the study.

 

P.S. This post has run too long, and I'll post the second part next week.

 

 

Other numbers from the colonoscopy trial

You think I'm done with the colonoscopy trial - I'm not.

Previously, I discussed the core controversy, which is when investigators learned that the trial outcomes are not what they expect - in fact, colonoscopy effectiveness was far below what the medical profession has believed - they decided to cancel the primary endpoint and tout an "adjusted per protocol" analysis, which was designed with knowledge of the disappointing primary result.

Any per-protocol analysis is a partial lie. It takes a well-designed randomized clinical trial and dissolves the bedrock of random treatment assignment, effectively turning the RCT into an observational study. I call it a lie because people who do PP analysis describe their result as coming from an RCT. The authors of the colonoscopy study are proud that this is the first ever randomized controlled trial for this treatment. This is exactly what happened with all the Covid-19 vaccine trials too: they only analyzed people who took two doses (but if someone gets sick after the first dose, they are ineligible to get the second dose). If the analysis is PP, the study should be classified as an observational study even if the original design is an RCT.

In the last post, I did a deep dive into the two analyses that were presented in the NEJM paper to show why the adjusted PP analysis allowed the research team to print an outcome of 30% improvement on diagnosing colon cancer rather than 18% in an ITT analysis. I find the underlying mechanics rather implausible but as with any observational study, it's anybody's guess as to whether the statistical adjustments helped or hurt.

***

There are a few numbers reported in the study that haven't gotten as much attention in the media.

The most important is the unproven impact on all-cause mortality. Exactly 11% of the invited-to-screen group died within 10 years, and the same proportion died in the usual-care group from any cause. In other words, even if colon cancer was the cause of death on fewer death certificates in the screening group, the same number of death certificates would have been issued in both groups. This is probably because colon cancer kills at an advanced age, and people die from other (related or unrelated) causes if they didn't die from colon cancer. Thus, colonoscopy did nothing to change the life expectancy of the participants.

The researchers did report a large impact on colon cancer specific mortality in their "adjusted PP analysis" (50% compared to 10% in ITT) but they were silent on all-cause mortality. The mortality result was merely stated, and no further details were offered on compliers vs noncompliers, age groups, countries of origin, etc., thus no one can judge whether it makes sense.

Also, note that all rates are cumulative over 10 years of follow-up. Actually, that is also a partial lie. It's a "median" of 10 years of follow-up, meaning that only half of the participants need to have reached year 10. From Figure 1, I can see that about 66% have reached year 10, 85% have reached year 8, etc. The important thing to realize is that the further out in the case curve you go, the smaller the sample size that underlies the numbers (i.e. less reliable).

Another important number is the absolute risk difference. Thirty percent sounds like a large improvement as a relative ratio but it isn't because the baseline risk is very low. Cumulatively over a median 10-year follow-up period, the baseline risk was only about 1.22% (the risk of the usual-care group). Thus, 30% improvement on that is about 0.4%. That means out of 1000 people invited to screen, we can expect about 4 fewer diagnoses of colon cancer over roughly 10 years.

Flip that ratio around and we have what is called Number Needed to Treat (NNT). To prevent 1 colon cancer case, we need to screen 250. Using the ITT result of 18%, the NNT is 455.

A glaring gap is the lack of a cost-benefit analysis. A colonoscopy apparently costs $3000 in the U.S. So, out of 1,000 invitations, perhaps 500 do the screening, which cost $1.5 million, in order to lower diagnoses of colon cancer by 4. Thus, the cost of each avoided colon cancer case (not death, just case) is $375,000.

 

Why no serious researchers conduct "per protocol" analyses

All observational data must be adjusted in order to correct biases. But our science teems with irresponsible statistical adjustments - throwing a bunch of variables into a regression model, and declaring that all known biases have been satisfactorily resolved is all too common.

In the medical literature, publishing one's adjustment model is rarely done. I don't trust any adjusted models unless the researcher discloses the equations, the coefficients and goodness of fit statistics. Even better, the researcher should compare the adjusted to unadjusted results, note how the adjustments affected the outcome, and explain why these adjustments make sense.

Handwaving about the structure of the adjustment model, referencing other papers that supposedly use similar methods, etc. suggest that the researchers lack confidence in their methods.

In this blog post, I demonstrate how to evaluate statistical adjustments, using the recently released, highly controversial results from the colonoscopy clinical trial. (See my related writeup here.) It took a lot of work to put this together, because the disclosure of details in the paper is scarce (this is the norm rather than the exception, unfortunately.)

***

The colonoscopy trial was the first randomized clinical trial (RCT) conducted to measure the effect of colonoscopy screening on risks of diagnoses or deaths from colon cancer. The trial enrolled two groups of participants: those invited to screening, and the "usual care" group which means no screening (as is the practice in Poland, Norway and Sweden, where the trial took place).

For this post, I focus only on risk of colon cancer diagnoses (i.e. cases). That's because the disclosure on mortality outcomes is even worse, making it impossible to judge the adjustment procedure. The trial specifies an intent-to-screen analysis (more generally known as intent-to-treat, ITT), The ITT analysis shows that 0.93% of the invited group were diagnosed during the 10-year follow-up window, compared to 1.13% of the usual-care group. This result is described as a 18% reduction in risk of colon cancer, even though the difference in diagnosis rate is only 0.2%, that is to say, out of 1,000 people invited to colonoscopy, only two fewer cases will be detected relative to usual care.

As discussed in the previous blog, the medical profession was none too happy with this outcome because doctors, esp. American ones, have convinced themselves that colonoscopy could reduce both cases and deaths by 70% or more. Neither did the authors of the paper disclosing the trial results - in the section of the paper in which they discussed the findings, they did not mention the primary endpoint, but only talked about an "adjusted per-protocol" analysis, which moved the reported result from 18% reduction in risk to 31%. Still far from the 70% previously believed, but deemed more palatable.

The media then mis-reported this analysis as "per-protocol" analysis, dropping the all-important word "adjusted". By the end of this post, you'll understand why I say this.

***

Intent-to-treat vs per-protocol is a long-running debate but any serious researcher goes with the former, which is exactly what the designers of the colonoscopy trial did. Let's look at this debate using a marketing example.

Let's say Target is running a Black Friday promotion in which online customers get an extra 30% discount if they type the code SAVE30 when checking out. In order to measure the effectiveness of this promotion, Target data scientists design an A/B test in which shoppers are randomly divided into the exposed group and the non-exposed group upon visiting Target's website. They intend to analyze the results of this experiment by looking at metrics such as proportion of visitors who made a purchase, and average purchase amount, comparing the two groups. This analysis is known as an intent-to-treat (ITT) analysis.

Ouch! The ITT analysis showed that the additional sales generated by exposure to the SAVE30 promotion failed to cover the cost of giving out these extra discounts. Inevitably, there will be at least one business manager who complains that the wrong analysis was done. They argue that we should not include people who did not notice the promotion. Being exposed to the promotion just means Target shows notifications while the customers browse around but some fraction of customers did not pay attention. So, what do they think is the right analysis? We should only analyze those people for whom we know for sure they noticed the promotion - in other words, we only analyze those people who entered the promotion code on checkout (or other variants, such as only those who clicked on a promotion banner). This style of analysis is "per protocol".

Why would all serious researchers shy away from per-protocol analyses? In this example, everyone in the exposed group who subsequently entered the promotion code on checkout is someone who is surely, or almost surely, making a purchase. In the non-exposed group, there is no such thing as entering the promotion code, thus we effectively narrowed the exposed group to almost sure shoppers against the unexposed group that contains the average online shopper. Which group will show better metrics? Will the better results on the exposed group be due to the promotion or inherent differences in the group composition?

***

Back to the colonoscopy trial. In the per-protocol analysis, the data analyst threw out roughly half of the invited-to-screen group because they did not "comply". The reasoning is that they could not have benefited from the screening (read: SAVE30) and thus, they drag down the efficacy. But the "compliers" typically have different characteristics than the "non-compliers", and so per-protocol destroys the randomization aspect of the "randomized" controlled trial. In other words, the complier subset is different from the usual-care group not just in being screened but in unknowable dimensions. We've turned the RCT into an observational study.

People who advocate PP analysis say not to worry - they will throw a bunch of demographic variables into a regression model, and all biases will disappear. I find this quite fascinating. If such adjustment strategies indeed can cure all biases, then why would the community embrace RCTs as the gold standard? It would appear that observational data blended in a mixer known as regression modeling is equally as good as a randomized experiment!

In fact, the colonoscopy controversy clearly shows the opposite. Prior observational studies have consistently shown extremely positive results of 70% or so, and the first time a research team conducts an RCT, the outcome is below 20%. If anything, this proves that carefully analyzed observational studies cannot replace the gold-standard RCT.

***

Even if all biases are cured, and we are able to believe the PP analysis, the real-world results are guaranteed to disappoint.

Let's go back to the Target example. Let's say the PP analysis showed that customers who used the SAVE30 promotion were 80% more likely to complete the transaction than the customers who were not shown the promotion. So now, the head of marketing tells everyone if all shoppers are shown the promotion, then the experiment proves that they should see an 80% jump in number of transactions.

Of course, reality bites. The problem is that the 80% incremental sales apply only to the subset who used the promotion code. If only 10% of those exposed used the code during the A/B test, then only about 10% would contribute an 80% jump in sales. The other 90% are exposed but would not be using the code.

If half the trial population elect not to take the colonoscopy screening during the trial, it is likely the case that half the population would make this decision if colonoscopy was rolled out to all. The 30% reduction in colon cancer risk would still only apply to the subset choosing to be screened, and would not make a dent on those who chose to forego it.

***

It looks like I have to defer showing you my analysis till the next post. The plan is to compare the ITT analysis and the "adjusted PP analysis", pointing out how the adjustments affected the outcomes. Come back tomorrow.

(By the way, all Covid-19 vaccine trials were per-protocol analyses as well, even though such analyses are problematic.)

 

 

 

All evidence are not created equal and some evidence are more equal than others

A casualty of the Covid-19 emergency is scientific rigor, and in particular, certain long-established statistical principles in evaluating evidence, and specifically, that the weight of evidence depends on the method of research, and how data are collected and analysed.

The media, and the scientists they employ as talking heads, rarely identify the nature of the evidence behind health policies. On air, they make little or no distinction between results obtained from randomized clinical trials (RCTs), observational studies, surveys, lab experiments, animal experiments, etc. Like a drumbeat, they promote the impression that all evidence are created equal, despite their disparate origins.

***

While such an attitude towards scientific evidence is not new, it became normalized during the pandemic. A great illustration is found in this recent article reporting the results of a clinical trial on colonoscopy (screening) for colon cancer (link).

In the U.S., the standard of care calls for all men above 45 years old to get a colonoscopy every 10 years. Colonoscopies are both expensive and invasive. In some other countries, less invasive alternatives like stool testing are preferred.

American doctors have pointed to strong evidence that colonoscopies work. According to the article:

Past research always showed that colonoscopy could put a huge dent, on the order of 70%, in the incidence and mortality from colon cancer.

In other words, we can reduce colon cancer deaths by 70% - purely by screening, and presumably catching cancer in its early stage. A researcher was quoted saying the profession has expected colon cancer to become "extinct" if every man were to take colonoscopy.

***

Chances are men were not informed that all past evidence about this practice has come from observational studies only.

Observational studies involve comparing the outcomes of people who have taken colonoscopies and those who haven't. However, because people self-select into these procedures, a direct comparison of the two groups is biased. The people who choose to do it are likely to be different from those who don't, beyond the decision to be screened. Analyzing observational data involves finding ways to remove biases.

Take a current example. It is obvious that when comparing the Covid-19 outcomes of vaccinated and unvaccinated people, these two groups are not identical because self-selection introduces biases. There's age bias, political bias, etc. etc. It's challenging enough to enumerate all sources of biases. The hard nut to crack is to learn the magnitude of such bias. Even if we know older people are more likely to be vaccinated, and more likely to suffer more severe consequences from Covid-19, how do we know the degree of bias? If we don't know the size of the bias, how can we tell if we have adjusted too much or too little?

It's like asking the tailor to adjust the length of a pair of pants without telling her the height of the person wearing those pants. It's just too long, the tailor is told.

Back to the observational studies of colonoscopies. Doctors have relied on a series of peer-reviwed observational studies that converge on the conclusion that colonoscopies are extremely effective. What can go wrong?

***

The profession ignored the nature of the evidence, the deficiencies of observational studies.

Finally, some researchers in Norway had the good sense to check these results by running the gold standard of medical evidence, a randomized clinical trial. In these trials, patients are randomly selected to get invitations to take colonoscopies. The random assignment of treatment is an automatic adjustment for known and unknown biases. (Read this previous blog for a primer.)

And... after 10 years of observation, the RCT showed that colonoscopies has essentially zero effect on mortality from colon cancer, and only about 20% reduction in incidence, both metrics coming in far below 70%, the efficacy found in the observational studies. I'll dig deeper into the results in a future post. For this post, I take these findings at face value.

***

What gave me feelings when reading the Stat article is the response from various experts who effectively advocate trashing the RCT result in favor of the observational studies. This reaction is shocking given the RCT's status as the gold standard of medical evidence.

This RCT is not some summer project by a student intern. It includes 85,000 people, with data collected over 10 years. And yet, the article ends with a quote that says "colonoscopy is still king".

The negative reaction reflects the belief that all evidence are created equal, and illustrates the skepticism toward counter-intuitive scientific findings. It's not just in medicine. Anyone who have run experiments in industry has encountered this attitude.

The arguments for rejecting the inconvenient results can be classified into the following types:

(a) Some other related evidence exists that contradicts the current study - e.g. a randomized trial of a related procedure has previously shown better results

The problem with this type of argument is that it says nothing about the current study. A Chinese term for this strategy is "raising East to beat West". This strategy works under the doctrine that all evidence are created equal. The other study is on a different procedure and it is brought into this conversation purely because it obtained the more "acceptable" result. Imagine this: if that trial has produced worse outcomes than the current study, would the expert have mentioned it to the reporter?

If we allow prior experiments to overrule new experiments if the latter's conclusion differs from the former, then what's the point of running the new experiments?

(b) The treatment worked better for a subset of the test subjects - e.g. if we restrict the analysis to the older population, the positive effect of the treatment is higher

Whenever the overall results of an experiment disappoint, the sponsor of the experiment almost always asks for "deep dive" analyses, which typically means, hunting for a subset of the population for which the treatment made a difference. What's wrong with this? Imagine, what if the researcher discovered a subset of patients for which the effect of colonoscopy on cancer diagnosis was not 20% reduction but 5% reduction? This finding will be dismissed and ignored because it adds "nothing new" to the overall conclusion. The only additional findings that can get the attention of the sponsor are those that validate the desired outcome.

Besides, any such deep dive analysis runs into statistical problems. The sample size of the subset is usually too small, harming the reliability of the result. The filtering effect of statistical significance means any such finding is likely to be an outlier for the experiment.

Worse of all, the sponsor may take the positive outcome for the subset, and create the impression that it works for everyone. This actually happens a lot more frequently than one thinks. Take a Covid-19 vaccine study that uses matching to remove biases. While the matching procedure increases our confidence that the two groups being compared are similar, many records are dropped from the analysis because they are unmatchable. For example, because most older people got vaccinated, and many younger people did not, the matched subpopulation has a different age profile than the full population. While the comparison of the matched groups is not biased, the extrapolation from matched groups to the general population is biased. Nevertheless, experts used the matching study to proclaim that the vaccine is effective for all, rather than that the vaccine is effective for the matchable subpopulation.

(c) Our hypotheses are better than your data - e.g. because treatment methods are improving over time, we believe the benefit will increase over time, therefore in five years' time, the outcome will be better

Everyone is entitled to their hypotheses but only those running the experiment has submitted their hypothesis to testing with real-world data. The experts offer no data to support the alternative hypotheses. The experts are also ignoring hypotheses that reinforce the study's findings. Imagine this: what about the possibility that someone would develop a new screening method that might dominate colonoscopies in the next five years?

(d) The data should be analyzed in some other way - e.g. we should use a per-protocol analysis

This reaction makes a mockery of pre-registering the study methods. We face the same problem  of ex-post hunting for a specific outcome. If the per-protocol analysis had yielded similar or "worse" results as the intention-to-treat analysis, the same expert would not have raised this critique.

***

Imagine this: the RCT for colonoscopy has confirmed the profession's longstanding view that the procedure cuts cancer deaths by 70% or more. What would the article look like? Would there be a line-up of experts raising concerns that the RCT outcome might be wrong?

All four types of counter-arguments could still have been raised. There is likely going to be at least one randomized trial of a related procedure that showed much lower efficacy. If the analysts did a deep-dive analysis, at least one subset of the patients probably experienced much lower benefits than the average. Various hypotheses could be proposed that suggest the RCT over-estimated the efficacy. Some other way of analysing the data would have yielded lower efficacy.

Nevertheless, we wouldn't be hearing these arguments.

***

Lost in this debate is the real question raised by the RCT; in fact, it hasn't even been asked. So I'm asking it here: What went wrong in those prior observational studies to have over-estimated colonoscopy's effectiveness to such a large extent? How did the machinery that "adjusts" for biases fail?

 

 

 

 

What about safety

Readers may have wondered why I have said little about vaccine safety in my reviews of vaccine trials. Today's post, which fills that gap, also serves to explain why.

As is well known, the FDA proclaimed that the mRNA vaccines (marketed by Pfizer and Moderna) are both efficacious and safe. Each trial enrolled about forty thousand participants, split randomly into vaccine and placebo groups, with the latter receiving a saline solution instead of the vaccine.

For efficacy, the primary outcome of interest was infections (measured by its proxy, reported symptomatic cases). As observational data drifted away from this narrative, the outcome shifted to hospitalization and death attributed to Covid-19. Whether the endpoint is cases, hospitalization or death, the clinical trial design ensures that an analyst can look at the difference (ratio) of the event rates to assess whether the vaccine performed better than the placebo. [That is, until after the emergency use approval was granted, because the FDA then allowed the pharmas to dismantle the placebo groups, destroying the trial design.]

Measuring safety - on the surface - may proceed in a similar way. However, safety outcomes present two enormous challenges that make safety analyses much harder - and thus less convincing - than efficacy analyses for the same cohort of trial participants.

First, it's difficult to know which safety outcomes are of greatest concern. It's a bit of a fishing expedition. Lots of outcomes are measured to screen for signals. Such data mining always raises the probability of false signals. Thus, all else equal, one would require a larger sample size to analyze safety outcomes. Since the sample size is fixed for both safety and efficacy outcomes, this implies that the confidence we have for safety is necessarily lower.

Second, the truly horrible safety outcomes - known as "severe adverse events" (SAEs), which include hospitalization and death - are expected to be rare events. Since a vaccine is indiscriminately injected into the population, most of whom are healthy people, even a small rate of occurrence can be alarming. Let's assume a hypothetical vaccine causes death with probability 1 in 500,000 shots (this is a thought experiment for illustration purpose only). Bloomberg reported that over 600 million shots have been administered in the U.S. alone. On average, 1,200 people would have died from this vaccination.

I chose 1 in 500,000 as the hypothetical rate of vaccine-induced death for a reason - the vaccine trials as designed would not have been able to detect this signal, let alone death rates below that level. For an event that happens at a rate of 1 in 500,000 attempts, the chance of observing zero occurrences in 25,000 people is 95%, and the chance of observing a maximum of two occurrences is 99.99999%. In other words, any SAE with a rate of occurrence of 1 in 500,000 would likely produce no events during the vaccine trials, and given the amount of shots that went into arms, that level of harm would have killed 1,200 people (while the analyst legitimately touts a perfectly safe vaccine).

***
In their respective trials, at the interim analysis that happened when half the participants have reached two months of observation beyond the second shot, Pfizer and Moderna observed few SAEs, and concluded that these event rates were "balanced" between the vaccine and placebo arms. This was the basis of the safety claim.

A group of researchers have recently revisited the analysis, and drawn a different conclusion in a peer-reviewed article published in the journal Vaccine. (link) They argued that the mRNA vaccines caused an excess of 10-15 extra SAEs of special interest relative to placebo. That's a difference. As a ratio, the mRNA vaccines caused a 43% increase in SAE risk. Most of the excess risk in the category known as "coagulation disorders".

It's instructive to learn why the same data led to different conclusions.

The first difference is what is counted as an SAE. It turns out that the FDA and U.S. pharmas created their own definition of what counts as SAEs. Meanwhile, the international community, something called the "Brighton Collaboration", agreed on a priority list. The new analysis reclassifies the events reported by Pfizer and Moderna in an attempt to perform an analysis that follows the international standard. Even here, the researchers conducted two analyses, based on the "original" list and an "expanded" list.

[For those wanting to peep into the sausage factory, the reclassification involves three experts readjudicating the events based on subjective evaluation of case history and other information. The original classification during the trials also involved such subjective judgment.]

The second difference is the proper handling of differential observation time among participants. Remember that the interim analysis were conducted while people were still in trial. I noted when I read the efficacy analyses the fact that the earlier participants were observed for longer periods of time, so that, even though the "median" participant reached the two-month point post second dose, the interim analysis involved many participants who had not yet reached the start of the case-counting window, which means that for the purpose of the interim analysis, these participants were "immortal"!

It turned out that the FDA-endorsed analysis included participants who have not yet received the second shot, since they counted any SAE at data cutoff, i.e. the same cutoff date applied to all participants regardless of enrollment date. The new analysis removed participants who had not yet taken the second shots.

***

A few other noteworthy nuggets can be found in the article. (link)

First, the authors noted that the Pfizer (Moderna) vaccine reduced risk of hospitalization by Covid-19 infection of 6 (2) per 100,000. So, for every 100,000 vaccine recipients, around 2-6 are expected to have avoided hospitalization relative to not getting vaccinated, but 10-15 are expected to experience serious adverse events relative to not getting vaccinated. As the authors noted, such a benefit-harm analysis should be conducted by age group but one of the enduring mysteries of Covid-19 vaccine science is the lack of available data of sufficient granularity for third-party researchers to conduct independent studies - all the more baffling given the claim of massive benefit and zero harm, which presumably should withstand even moderate- to large-sized error bars!

Just for illustration. The 2-6 hospitalizations are likely concentrated in older people and if the 10-15 excess SAEs occur mostly in younger people, then the risk in the young is higher than average while the benefit is lower than average.

Second, Pfizer stopped reporting safety outcomes beyond one month after the second shot.

Third, Moderna counted Covid-induced SAEs in the SAE analysis, which the authors pointed out is not the "traditional" analysis. (Since supposedly the vaccine's side effects are similar to the effect of Covid-19 infection, differentiating between them requires experts to determine the cause of the observed SAEs. That's definitely not something you or I am qualified to do!)

 

 

 

Andrew screaming about professional fouls

In a recent blog post, Andrew announced that "he will keep screaming about this sort of thing forever" (link).

And I will be right there with him.

The "sort of thing" that upset him this time is when authors of a peer-reviewed journal article chose a misleading title. The accused title is "....: A long-term experimental study of ..." Anyone who reads more than the abstract learns that by "long term", the authors meant three days. As Andrew quipped, since they aren't studying amoeba, three days do not long term make.

Frustratingly, this paper is not a special outlier in the world of academic publishing. Such misleading titles or abstracts are common enough, and no one seems to be willing to call them out.

This is a nice example of what I've been calling "professional fouls" (see this post), a term borrowed from sports. A professional foul is an act recognized by outsiders as a foul but not considered a foul by insiders since "everyone does it".

In football (soccer), the so-called diving is a professional foul. Most players perform this act and most players aren't agitating for strict refereeing to root out this practice. In pro basketball, I find the last few minutes of a game unwatchable because it consists of a series of professional fouls leading to free throws. The entire profession pretends that this is "smart strategy". Smart strategy is if the community agrees to forbid this ridiculous tactic.

I complained about similar abuse of journal titles before. For example, what Pfizer titled "six-month safety and efficacy" suggests that participants in the clinical trial has been observed for at least six months to determine the vaccine's efficacy after six months. Based on my analysis, 80 percent of the participants in that study were observed for five months or fewer - and would never arrive at the 6-month point because the FDA had allowed placebo participants to get vaccinated, and almost all decided to take the shots. Just because a small number of participants were observed for six months, Pfizer called the study a "six-month" study.