Big Data, Plainly Spoken (aka Numbers Rule Your World)

Better late than never

Quick quiz. Who wrote this?

I did not test the data for irregularities, which after this painful lesson, I will start doing regularly.

Of course, you won't know. But which of the following people is the most likely to have said such a thing?

A) A Stat 101 student

B) A data scientist working in industry during the first year of employment

C) A graduate student research assistant

D) An assistant professor in the publish-or-perish game

E) A senior tenured professor with multiple best-selling books and tens of thousands of citations

***

If you think the least likely is the most likely, then you'd be right. The answer is (E).

The person who said the sentence (in 2021) is Dan Ariely, currently a Duke professor of economics but probably known to my readers as author of a series of best-sellers in behavioral economics, starting with Predictably Irrational. (You can find the quotation in this PDF.)

Andrew has documented a series of scandals in which several of his seminal studies have been called into question. The latest one involves data obtained from an insurance company by Ariely. He now claims to have no role in collecting, and processing the data.

Neither did the people who uncovered this potential research fraud. Hence, the ability to suss out data problems does not require first-hand involvement in collecting or processing data.

To read about how the fraud was detected, see here.

***

Next time you hear about a publication in a "peer-reviewed" scholarly journal, think about this case. It is entirely possible to publish data analysis in a peer-reviewed journal without "testing the data for irregularities". In fact, researchers with hundreds of peer-reviewed publications may have never once tested data for irregularities! Better late than never, right?

 

P.S. (1) If you need something light for the holidays, here is an advice column Andrew, hmm, wrote for the WSJ.

(2) One of Ariely's best-sellers is "The Honest Truth about Dishonesty". He is an expert on honesty.

 

Thirty percent unvaccinated in healthcare: less than meets the eye

When I saw this "news" on my Twitter feed, I knew I have to write a blog post about validity of research results. There is an ongoing plague in which science reporters broadcast top-line results without ever checking the underlying papers - which disclose how limited most of these results really are.

The most important words in the above excerpt is not that 30 percent of U.S. hospital-based healthcare workers are still unvaccinated - they are "for whom data on vaccination status was available".

The published paper is here. I'll summarize what you'll learn from it.

***

The report covers only 40% of U.S. hospital-based healthcare workers. The source is a database in which hospitals (sometimes through the state health departments) voluntarily enter aggregate data.

About half of the hospitals in the U.S. contribute data to this database. Another 10% of hospitals were dropped because of exclusions.

"Psychiatric, rehabilitation, and religious non-medical facilities were excluded." I really don't know why.

Also, hospitals for which "total HCP" (healthcare personnel) or "HCP vaccination" fields are missing or zero are excluded. The confounding of missing and zero is a common database problem. Does zero mean zero or missing? By using this exclusion, the analysts made the assumption that zero means missing, and not zero.

Yet another exclusion applies when unvaccinated personnel and fully vaccinated personnel summed to less than 75% of the total number of HCP. This means they excluded any hospital in which a significant number of HCPs have taken one shot or have taken two shots but not reached 14 days. I really have no idea why this exclusion is needed.

P.S. On second thought, I think this provision is associated with the desire to put "unvaccinated" into the headline instead of "not fully vaccinated". However, when such an exclusion has been added, then the first part of the sentence should really say 30% of hospital-based healthcare personnel excluding those who are partially vaccinated. So, no, I don't like this exclusion, and the headline is inaccurate.

***

What is the implication of exclusions? It causes the study findings to become less generalizable. It also confuses readers. When people read "30% of hospital-based healthcare personnel are unvaccinated", they think 30% of all such personnel. When they do that, they made the implicit assumption of no non-response bias, i.e. the hospitals excluded from the study are the same as those included.

Ex ante, we know that assumption to be false. If that assumption held, then there would be no need to exclude hospitals in the first place.

If excluded hospitals (including those who chose not to participate) are different from included ones, then the study finding can only apply to included hospitals, which represent less than half of total hospitals. In statistics, we say exclusions reduce the "validity" of results.

***

Many statisticians, including myself, prefer to make educated guesses to close the gap. We'd like our overall finding to generalize to the whole country. We'd therefore need to predict what answer we would see from the excluded hospitals should we have their data. This prediction can use data from the included hospitals with appropriate adjustments.

A key to such a prediction is to find differences between the excluded and included hospitals. According to the paper, excluded hospitals are less likely to be critical access hospitals, more likely to be non-rural, and more likely to be larger in size. (The implication of this last factor is that the 40% of hospitals represent less than 40% of all hospital-based HCPs.) The authors stated that the first two factors are associated with higher vaccination rates, and so the excluded hospitals may in fact have higher rates than those included.

Here is where you find heated debates between statisticians. Some argue that any such prediction injects subjectivity into the analysis. I take the other side of this argument. Not injecting subjectivity is a mirage. When you don't make adjustments like these, your results assume no adjustments are necessary, which is to assume no bias exists, and allow readers to implicitly extend biased results to the entire population. The adjustments may fail entirely or partially but I don't think the adjusted results are any worse than the unadjusted ones.

***

The adverse impact of hidden biases in the dataset is easily missed.

Consider the following sentence from the paper:

Coverage was higher among HCP working in facilities located in metropolitan counties (71.0%; 95% CI: 70.9, 71.0), followed by HCP working in facilities located in rural counties (65.1%; 95% CI: 64.8, 65.3), and HCP working in non−metropolitan urban counties (63.3%; 95% CI: 63.1, 63.5).

We know that rural hospitals have a higher response rate than urban hospitals. Responders are likely to be different from non-responders. So we don't know whether vaccination rate by degree of urbanicity would be in that precise order were we to have complete data.

The same problem applies to every conclusion of the study, e.g. children's hospitals have higher vaccination rates. In some cases, the connections are not obvious. (Children's hospitals are not likely to be equally distributed in urban and rural areas.)

 

Further adventures of case-counting windows

If it's not already clear, observational data is a great playground for statisticians. We sniff out where biases are hiding, and we devise methods to correct such biases. It's a lot of fun. It's also frustrating - because there are endless ways to make adjustments.

In today's post, I examine a couple of curious observations made by the Israeli researchers when they analyzed the data from vaccine boosters (link).

***

The researchers shrank the follow-up window when counting severe cases (relative to counting cases)

This decision surprised me because severe cases are a fraction of all cases so I'd want to have a longer counting window, not a shorter one.

Recall that in the analysis of cases, the Israel booster real-world study pulled data from the national database for the month of August. Because of a case-counting window that starts 12 days after the booster shot, in effect, the study only counts cases from August 11 to August 30. Most people did not get the booster on July 31st, and so the average follow-up time is probably around 10 days. (See the previous post for more.)

For severe cases, the case-counting window shrank by 4 days, as the researchers stopped counting on August 26 instead of August 30. Thus, the average follow-up time for severe illness is around 6 days. You're right - protection against severe illness was shown for less than a week.

Here is a direct quote that explains why the case-counting window was shortened for severe cases relative to all cases:

In order to minimize the problem of censoring, the rate of severe illness was calculated on the basis of cases that had been confirmed on or before August 26, 2021. This schedule was adopted to allow for a week of follow-up (until the date when we extracted the data) for determining whether severe illness had developed.

In the Appendix, the researchers offered the following chart as supporting evidence:

This chart shows the lag between testing positive and the onset of severe illness for Covid-19 cases between November and March 2021 (conditional on people subsequently getting severely ill). The median lag is 4 days, meaning among those who eventually got severely ill, half got severely ill within 4 days of testing positive, and half got severely ill after 4 days (up to almost 30 days).

I believe the logic is that for anyone who got sick in the final 4 days of August and became severely ill, the onset of severe illness would have taken place after the data cutoff for this study.

Consider the following implications:

• According to the chart above, half of the past patients took over 4 days and up to 30 days to develop severe illness. If we want to capture all severe cases, we ought to extend the follow-up period to at least 30 days - it should be 60 days if we use a one-month cohort. The methodology based on median lag time would capture only half of the severe cases.

 

• Think about the severe cases that occurred between Aug 26 and Aug 30. These severe cases are connected with infections that occurred prior to August 26, most likely during August. They are excluded from the analysis. All of these severe cases should count. The severe cases that happened on the first 4 days should be dropped, if we use their logic. (Note that the infection rate was skyrocketing during August.)

 

• Now think about someone who took the booster shot on August 16. If this person became severely ill on August 26, the severe case is not counted. But this severe case represents a lag time of 10 days from detection of infection, which according to the chart above, happens more than 10% of the time. This issue arises because the case-counting window is given a fixed ending date regardless of when the booster was administered.

***

The researchers disclosed they faced an incurable imbalance problem.

Part of the paper's Appendix is devoted to alternative analyses, one of which is a matching study. I discussed examples of matching studies before (Mayo Clinic, and Clalit). Unfortunately, this disclosure did not contain enough details to evaluate the quality of matching.

The following statement caught my eye: "Due to the very small number of severe cases, and the high censoring proportion, calculation for severe COVID-19 was not possible."

In my comments on the Clalit study (link), I mentioned a limitation of matching studies: if selection bias is severe, it may be hard to find acceptable matches, leading to dropping many unmatched cases ("high censoring proportion"). In this case, the matched population ended up being much younger than the vaccinated population.

In the Israeli booster study, the researchers attempted to find one person who hasn't taken the booster to match with each person who took the booster. Matching variables are the usual suspects like age group, second-dose date and so on. They appear to be saying that they failed to find adequate matches for a large proportion of the booster group. This is a sign of incurable covariate imbalance. This also suggests that those regression adjustments used in the main analysis are insufficient.

 

 

 

Boosters in Israel illustrate statistical issues with real-world effectivness analysis

I previously discussed the evidence that Pfizer submitted to the FDA to get approval for booster shots (here and here). The studies were small, and circumstantial. It appears that the experts were partially or completely swayed by "real world" data coming out of Israel so I looked at what is in that influential Israeli study (link to report).

The story of vaccinations in Israel is one of high hopes unfulfilled. Here is their full case curve for 2021. 

Between April and July, it appeared that the fast vaccination campaign yielded dividends; public health officials were satisfied that the vaccines beat Covid-19. Like other countries, Israel moved to remove most of the mitigation measures. Not so fast, as in July, cases started to tick up again. The rate of growth was so alarming that they became the first country to approve booster shots. Other countries, including the U.S., were curious what the outcomes would be - since the vaccine developers were not going to run proper randomized experiments to prove the value of boosters. Even after administering boosters, the case curve didn't start bending till late September. To this day, despite the general acceptance that vaccines are effective, no one can reliably predict what proportion of the population needs to get injections before infections start to decline, or how much time it would take to start seeing cases come down.

***

This Israel study (link) was published in October, and only concerned one month of data (August 2021). Approximately 1.1 million Israelis were fully vaccinated by July 30 and formed the study population. However, we do not know how many of those subsequently took the booster shot. This is one of those key omissions that have pervaded all the vaccine studies - basic check-off-the-box data that are withheld from the scientific reports.

The data came from the Ministry of Health. Data cutoff for cases was Aug 30, 2021, and these researchers invented a new case-counting window that starts at 12 days after the booster shot. Since the first booster shots analyzed by this study happened on July 30, that means no cases are counted until 12 days later (Aug 11, 2021). For those who took the booster on July 30, the study has a maximum follow-up window of 20 days. The graph above shows a linear growth in administered boosters during the study period. For someone who took the booster on Aug 15, the case-counting window opens on Aug 27 and closes on Aug 30, thus the follow-up period was 3 days. Anyone who took the booster after Aug 18 are "immortal": if they get sick after 12 days, those cases won't feature in the study data. 

This little calculation reveals that the follow-up period for the booster group is between 1 and 20 days, and that the study population is effectively three weeks, 12 days fewer than advertised (people who received boosters in those 12 days have not reached the beginning of the case counting window). That means, if we believe everything else in this study, the researchers have shown that the booster shot is effective for an average of 10 days.

The headline result of the study is that those who took the boosters are 11 times less likely to get infected than those who didn't take the boosters (note the unspoken: effective for 10 days, after ignoring anyone who gets sick during the first 12 days). For severe illness, the factor was 20 times.

As explained in my previous post, the use of these multiplicative factors (x times) is a fad. They could have expressed the same data in the usual way: the booster effectiveness was 90% for cases, and 95% for severe cases. In other words, they are claiming the same effectiveness as the original two shots.

The above calculations were "adjusted" by including main effects for age group, gender, calendar day, timing of 2nd dose, and "sector" (Arab/Jewish/Ultra-Orthodox).

***

A vaccine effectiveness metric is a relative ratio between the (severe) case rate of the treated group and that of the untreated group. In the original vaccine trials, the treated group took two shots while the untreated group took the placebo shots. In the real-world booster study, the treated group took the booster shot while the untreated group didn't; both groups previously received two doses of an mRNA vaccine.

The VE metric is merely a correlation between cases and vaccination status. In order to move from correlation to causation, statisticians want to establish that the booster group and the no-booster group are comparable; that the observed difference in case rates isn't driven by factors other than vaccination.

To understand the nature of this challenge, let's pretend that the FDA required vaccine developers to conduct a randomized controlled trial (RCT) to determine the causal effect of the booster shot on cases (The FDA didn't require it.)

We would have recruited participants from the group that has previously took two vaccine shots. Then, they are randomly divided into two groups: booster, and placebo. All participants are then followed for a period of time, and at around the two-month time point, an interim analysis is conducted. This analysis is simple, because by virtue of random assignment, the two groups have co-variate balance (same mix of age groups, genders, occupations, date of second doses, etc.), and thus, any variation in case rates is due to the booster.

Instead, we have "real world data". On the surface, we have two groups: booster and no-booster, which sounds similar to booster and placebo. But labels deceive. The lack of a random assignment device means the two groups are not apples to apples. Note that public health policy prioritizes high-risk groups, and some people have reasons to want early boosters (e.g. work on the front lines, want to stop wearing masks).

I find it rewarding to think about the study population in terms of three subgroups (bottom row of the left chart).

Start with anyone in the study population on July 30, the start of the study period. This person is in the no-booster group - until s/he decides to get the booster shot, at which point the same person migrates to the booster group. There are some who never took a booster during the month of August so they remain in the no-booster group throughout the study.

Let's consider the following three subgroups:

(NB1) People who never took a booster during August

(NB2) People who received a booster during August - they belonged to the no-booster group until the day of booster

(B) People who received a booster during August - they migrated to the booster group on the day of the booster.

(NB2) and (B) are "twins". They are the same people, counted twice, the first time as a member of the no-booster group, and the second time as a member of the booster group.

Thus the no-booster group (NB) in the study consists of (NB1) and (NB2) while the booster group is just (B). Now, let's address whether (NB) can be compared to (B).

***

Consider first (NB2) vs (B). These are twins, they are the same people counted twice. Thus, by definition, we have 100% co-variate balance, if we consider the typical demographic co-variates, such as age, gender, and occupation. Even date of second dose is also perfectly balanced since that's a variable about historical behavior prior to the study.

Nevertheless, (NB2) differs from (B) in pivotal ways. The exposure time windows are disjoint. The (NB2) twin always have earlier exposure than the (B) twin. Besides, the length of exposure is negatively correlated: the longer the follow-up time for the (NB2) twin, the shorter for the (B) twin. Thus, the booster status variable is confounded with exposure time and duration - and this is a hopeless confounding, 100% confounded.

Next, compare (NB1) with (NB2). Are the two partitions of the no-booster group similar? The answer is a resounding no. (NB2) are those who got the booster in August while (NB1), not in August. Unless one believes that the timing of the third dose is chosen completely at random, one must conclude that those two subsets are dissimilar due to selection biases.

In addition, (NB1) invariably has longer follow-up time than (NB2), and by extension, (B). The start of the follow-up window is fixed for everyone in (NB1) and (NB2) but variable for those in (B).

To summarize, (B) is identical to (NB2) on many co-variates due to twinning but differs from (NB2) pivotally on exposure metrics. (NB2) is different from (NB1) due to selection bias, thus these two subsets do not have co-variate balance; they also have different exposure durations.

When we compare (B) with (NB), we may be tricked into thinking we have good co-variate balance because of the (NB2) component. This statistical deception intensifies as more people get boosters. The exposure-related complete confounding is, to my knowledge, impossible to cure. Regression adjustments via main effects are insufficient to solve these complex statistical problems.

The differences between (B) and (NB) - in addition to booster status - include the exposure-related confounding with (NB2), the selection biases with (NB1), and the exposure-related bias with (NB1). All three are affected by the proportion of people who have taken booster shots.

Any serious causal analysis of real-world booster data must offer adjustments that deal with the above statistical issues.

P.S. [11/9/2021] In the following post, I looked at a couple of observations from this study that caught my eye.