Big Data, Plainly Spoken (aka Numbers Rule Your World)

Sloppy language misleads

One of the most frustrating aspects of reading Covid-19 related publications is reading the conclusions. A summary of results is supposed to provide readers with the minimally necessary information to understand the full contents of the paper - but what we encounter is "story time": almost always, these conclusions are stated in the most general sense, ignoring all the caveats, the exclusions, the imperfections of the methodologies and analyses. They are "stories" that go far beyond the limited evidence presented.

I've been reading the Clalit study that was a major force behind the approval of the first booster shots back at the end of 2021/start of 2022. This paper made the following one-line conclusion:

Participants who received a booster at least 5 months after a second dose of BNT162b2 [i.e. Pfizer] had 90% lower mortality due to Covid-19 than participants who did not receive a booster.

Is it possible to misread this sentence? Yes, if you literally read the sentence.

***

If we know something about vaccine efficacy/effectiveness, we immediately recognize that these authors are selling the same story - that the vaccine has VE 90%, the only difference here being the outcomes are deaths attributed to Covid-19, rather than reported cases. As such, this VE is driven by the relative ratio of the death rates of the booster group vs the no booster group. They didn't use the term VE as it typically connotes cases not deaths.

A death rate is a fraction composed of two parts: people who are counted as part of the comparison group, and people within that group who died during the study period. In this post, I'll pay attention to both these parts.

From my last post, we already know one issue with the above statement. If we take any participant at the start of the study and ask whether this person is counted as part of the booster group or the no-booster group during the study, the answer, for 90% of the participants, is both. 

Since at the start of the study, 100 percent of the participants had not received the booster, and by the end of the study period (a mere 54 days), 90 percent of them got the booster, the researchers could not have divided up the study population into two distinct groups of people - each person (except the 10% who didn't get the booster during the study period) first appeared in the no-booster group, and then transitioned to the booster group.

But the concluding statement reads like the study enrolled two distinct groups of participants. This misreading is even more likely for knowledgable readers because ordinary studies, such as well-designed clinical trials, use distinct groups.

Note that I'm not criticizing the methodology up to now, I'm describing what it is, and criticizing how the language in the summary statements sends readers the wrong message.

***

The above detail may sound trivial but it has major implications. Think about the regression adjustments. As with all observational studies, researchers claim that by adding standard covariates to their regression models (age, gender, ethnicity, prior health conditions, etc.), they have corrected substantively all biases. For example, if older participants are more likely than younger ones to want the booster, rather than being assigned a treatment at random as in a randomized clinical trial, then it's possible that the age distribution in the booster group does not mirror that in the no booster group, i.e. an age bias exists.

Look at the setup here: Ninety percent of the participants appear once in the treated group and once in the not treated group. By the design of this study, there is little variation across the comparison groups in the distribution of everything, including age. If we check "covariate balance", it's balanced because the two groups are by and large identical. So this type of studies underestimates the variability of everything - which leads to p-values that are too small, and regression adjustments that are meaningless.

***

In the paper's conclusion, the phrase "participants who did not receive a booster" is incorrect in more ways. We already know almost all of them ended up getting boosters before the end of the study period, which was fewer than two months long. I have previously said that a participant transitions into the booster group when s/he takes the booster shot.

This is in fact not true. That's because this study also imposes a "case counting window," under the claim that the booster shot takes 7 days to become effective. In other words, if someone takes a booster shot, and falls ill with Covid-19 within the first 7 days, this case is not counted as a case in the booster group.

Negating this case is one thing - but this study (and many others) took another step: the case is not negated but it is counted as a case in the no-booster group!

How do they pull this off? The study design says not to transition the participants to the booster group until the 8th day after the booster shot.

So, let's review this. Each participant starts off in the no-booster group, then the person took the booster shot; this person is kept in the "no-booster" group for an additional 7 days before being moved into the "booster" group. During these 7 days, if the person gets infected or died from Covid-19, the case or death is counted as "no booster".

Remember that we can only talk about exposure time for this type of study since almost everyone is eventually exposed to both conditions. The exposure time of the "booster" group therefore excludes the first seven days after the booster while the exposure time of the "no booster" group includes the first seven days after the booster.

This is an instance of the additional bias that Lataster described in his comment on our Three Biases paper.

***

As the story goes, even the above description is inaccurate. We haven't dealt with the following puzzling disclosure:

participants who received the booster and had a confirmed case of Covid-19 within 3 days before the effective-booster date (defined as 7 days after the booster was administered) were excluded

The analysts further subdivided the seven days after the booster shot is administered into two parts: Days 1-4 and Days 5-7. Let's follow a participiant who did not have the booster at the start of the study. During the study period, this person took the booster shot. If the person got sick during Days 1-4, the person is counted as a member of the no-booster group with infection as outcome. However, if the person got sick during Days 5-7, this person is excluded from the study, i.e. not counted as part of the no-booster group, and not counted as a case. Finally, if the person did not get sick by Day 7, the person transitions to the booster group.

There is a subtle but important difference. The Days 1-4 treatment is applied to every participant, regardless of outcome. If they got the booster, they continued to be called "no booster" for seven additional days. The Days 5-7 treatment is an exclusion based on the outcome. If the person got sick, the person is removed while if the person did not get sick, the person is retained. For me, it is never acceptable to re-define treatment groups after observing the outcomes. And yet, this paper (and many others) engages in this practice.

To understand the impact of these data processing procedures, you can use the same device we used in the Three Biases paper: imagine a vaccine that is ineffective, apply the analysis, does it result in no effect, or something else?

This is a great example of why researchers must disclose the code used to process the data. It's hard to believe these special adjustments did not get the attention of reviewers. But I can think of no other way to interpret those words. (In fact, the waterfall chart in the paper confirms the exclusions.)

***

Now that you have heard how they measure outcomes and define "no booster", it's time to revisit the conclusion:

Participants who received a booster at least 5 months after a second dose of BNT162b2 [i.e. Pfizer] had 90% lower mortality due to Covid-19 than participants who did not receive a booster.

Does this sentence correctly summarize the study?

 

 

How hard can counting be?

Raphael Lataster has written a response to our JECP paper, which discusses several biases in Covid-19 vaccine observational studies that have not been properly accounted for in many notable studies.

He augmented our first example to show another source of bias in later studies.

In our example, we demonstrated how the "case counting window bias" could cause vaccine effectiveness estimates to be over optimistic. In typical clinical trials, one starts counting cases (events) from the day the treatment is finished, so in the case of two-shot Covid-19 vaccines, this should be the day after the second dose. However, none of the Covid-19 VE results you have read about include all these cases!

All studies have imposed a case-counting window that always starts X days after the second dose. X is most often 14 days but sometimes it's 7 days, sometimes it's 28 days, or whatever the researchers deemed appropriate, frequently after computing VE for different window sizes.

The problem is that in observational studies, for the unvaccinated group, there is no such thing as 14 days after second shot (because they did not get any shots, unlike those in the clinical trial who got placebo shots). In many studies, the case-counting window is applied asymmetrically to the vaccinated group (reducing the case count) but not to the unvaccinated group.

Lataster pointed out another problem that creeped in with later studies. Many prominent studies started to define the "fully vaccinated" group as those having two doses (within some allowable dose duration) and who did not get infected prior to 14 days after the second shot, and then label the complement of the fully vaccinated group as the "unvaccinated" group.

This creates a host of problems. For the sake of discussion, let's focus on one subset: the group of doubly vaccinated people who got infected during the first 14 days after taking the second shot. In our original example, these cases are taken out of the case count in the vaccinated group. Effectively, these cases disappeared from the analysis.

In Lataster's example, these cases did not disappear. They got shifted into the unvaccinated group because that group now included doubly-vaccinated people who got infected in the first 14 days.

***

The decision to apply a case-counting window is tragic. It lands researchers into a dead end. This is a point Peter and I made in our reply to Lataster's response (here, paywalled).

The key of understanding our argument is to pay attention to both numerators and denominators in the case infection rates, which underlie the VE formula. The numerators are the cases, and the denominators are the subjects from which cases accrue.

In typical clinical trials, the treatment populations are fixed in advance. In an intent-to-treat analysis, one must use the treatment groups as originally randomized. Even in a per-protocol analysis, we remove subjects who did not complete the treatment (at the risk of adulterating the randomization). This procedure is different from removing subjects who completed the treatment (2 doses), and then got infected (i.e. have the primary outcome) within X days of finishing the treatment.

However, if the analyst uses the per-protocol population, and also applies the case-counting window, then these subjects who got infected during the first 14 days disappear from the numerator (case count) but not the denominator. In effect, they have become "immortal". Further, in observational studies, for which we cannot apply the case-counting window to the unvaccinated group, the analyst has made some immortals in the vaccinated group but not the unvaccinated group!

In Lataster's example, not only are the cases moved to the "unvaccinated" group but also the subjects. The analyst has taken the doubly-vaccinated person who got infected during the first 14 days out of the vaccinated group and added this person to the unvaccinated group. The case count follows along. This procedure makes me very uncomfortable. What happened here is that the analyst altered the treatment group assignment after looking at the primary outcome (Covid-19 cases).

This is why I called this a dead-end. If you just move the case not the group assignment, then you insert an immortal bias. If you move both the case and group assignment, you're manipulating the group assignment based on observing the primary outcome of the analysis. And, as Peter and I explained in our response, we are not convinced by arguments for case-counting windows.

***

Another part of Lataster's comment concerns empirical evidence that showed negative VE. I'm unwilling to go as far as he does because those negative VE numbers are observed in subgroup analyses for selected studies. Much more work needs to happen before we can establish generality of those results.

 

The hate campaign against negative reviews

A friend sent me this New York Times article about a "hate campaign" orchestrated on the book reviews website, GoodReads (acquired by Amazon a while back, no surprise), that allegedly suppressed sales of a new book.

I'm definitely a biased observer here since I have authored two books, and am affected by online book reviews. I also have emotional responses towards negative reviews, and like other humans, I exercise far less scrutiny of positive reviewers. It's a difficult issue to grapple with, and instead of offering conclusions, I want to suggest some topics for discussion.

(This is an example of a 1-star review on Amazon. When I wrote this post, the GoodReads site was malfunctioning and no reviews would load for me on multiple browsers. In my experience, GoodReads reviewers give fewer stars relative to Amazon reviewers.)

 

(A) The gripe that the negative reviewers published their opinions even before the book's publication date.

By itself, this is an invalid complaint, because publishers routinely send out advanced copies to friendly reviewers who are expected to help generate buzz prior to and leading up to the publication date. The real gripe is that they don't want to hear negative buzz, but if in an alternate universe, it's not a "hate campaign" but a "love campaign", I don't think this gripe would emerge.

Should there be an embargo on all reviews before the publication date?

The novel was published on June 6, 2023. The first review on Amazon was on June 6, 2023. Of the nine reviews posted on the first day, four were 5 stars, three 4 stars, and none below 3 stars. As of this writing, about 25% of the total reviews are below 3 stars. Who are the first-day reviewers? Should there be a reasonable X number of days to allow people to buy the book, read it, and write a review?

 

(B) The gripe that the negative reviewers objected merely to the outline of the book, and have not read the book itself.

According to the journalist, after the friendly reviewers put up a summary of the plot of the book on GoodReads, some unsettled readers wrote bad reviews because they hate the concept. This gripe is more valid. But... realistically, how many of the people who write reviews (including positive ones) have read the book? How much of the book must be read to be considered "qualified" to have an opinion? Did the famous people who put up blurbs on the book covers read the books first page to last?

The same question can be asked of not just early reviewers but any reviewers.

 

(C) The gripe that GoodReads do not ban these allegedly invalid reviews or reviewers.

The problem is how to set the censorship criteria. Should the authors and their publicists have the right to decide which reviewers or reviews are invalid?

Take, for example, the criterion that GoodReads should ban reviewers because they have not read the book. How is GoodReads going to know that? Do the publishers send GoodReads a list of names to whom they have sent advanced copies? What if one of the advanced readers pass the book on to someone else, who then write a review after reading the book?

If someone puts up a positive review after reading just the first 10 pages, is the review valid?

Take the other example of banning all reviews before the publication date. Surely, the publishers didn't mean that because they actively seek out advance praise. Emphasis on the word praise.

My own favorite censorship criterion is to delete all content-empty reviews. You routinely see reviews that have no value at all - "This ia a wonderful book", "This book is terrible. I hate it!" Having read the review, I have gained no appreciation for whether the reviewer read one page of it. But be careful what I reap! There are at least as many content-empty positive reviews as there are negative ones.

Here are two examples from Amazon:

Should online platforms censor content? This is a challenging question.

***

The journalists lamented that the book in question sold only 1,000 copies in the first 10 days, attributing the supposedly poor performance to the "hate campaign". Too bad the readers who purchased the book did not organize a "love campaign" on GoodReads.

 

 

It's random because we say so

Some of you are wondering why I haven't written about Covid-19 studies recently. Lamentably, the quality of such studies has not improved. It may have deteriorated even more. I wrestle with whether it's a waste of time to read them. The problems with many studies are apparent from the first page: how do they pass the peer review process?

This post concerns a half page of a recent jourmal publication called "Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data" (link). It's only half of one page because there are already enough problems with what I read that I don't see the point of continuing.

A friend sent me this paper because the authors share my concern about significant misrepresentation of vaccine effectiveness in observational studies of Covid-19 vaccines. (Here is a post about my paper published last month with two co-authors that highlighted three common biases that are not properly handled in studies.)

My comments are limited to the section with the header "Sequential Target Trial Emulation" on page 2 of the paper. In this section, the authors describe the first of two proposed analytical devices for dealing with biases in observational data.

For any observational study, there must be a well-defined starting and ending date of study enrollment. For each day in this enrollment window (and for each region), they pretend that a new target trial is being launched. This is how they put it:

on each day and in each region, eligible persons who have not yet been vaccinated are randomly assigned to receive immediate vaccination or to remain unvaccinated throughout follow-up and are then followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first

They then suggest that each daily trial can be analyzed on its own, and then the results averaged to obtain an overall effectiveness. They claim this outcome is representative of the "intention to treat" effect in a hypothetical randomized clinical trial. That's a bold claim! (Not even Pfizer, Moderna, etc. claimed to have produced ITT vaccine efficacies even in their RCTs.)

I'm not making this up. They literally claimed that "The trial-specific estimates of the effect of vaccination from each sequential trial can then be combined to estimate an observational analogue of the intention-to-treat effect of assignment to the intervention."

***

Why am I shaking my head violently?

Their statement that people are "randomly assigned to receive immediate vaccination or to remain unvaccinated" is unserious.

We are talking about observational data - data analysts poking around in a database after the fact. It doesn't matter what analytical technique is used - it remains the case that the vaccinated people self-selected the treatment. They were not randomly assigned to receive immediate vaccination.

In fact, the authors contradicted themselves in the very next sentence when they explained (my bolding):

Each calendar day is considered as time zero for a new emulated trial, with persons who are determined to be eligible assigned to the vaccination group if they were vaccinated on that day or to the no-vaccination group if they were not vaccinated,

For the vaccinated, there is no "assignment", let alone random assignment. Whoever decided to get vaccinated lands passively in the treatment group for that day. Because the vaccinated are self-selected, the unvaccinated are self-selected as well. Whatever selection biases exist in the raw data persists in this ex-post experimental setup. This step absolutely does not support presumption of random treatment assignment.

If this procedure resulted in random treatment assignment, then the original dataset contains no biases, which annuls the central premise of the entire paper.

***

Of course, they aren't taking every unvaccinated person. So the data analyst actively selects who comprise the unvaccinated group. The authors give two mechanisms for selecting unvaccinated.

The first is random selection. I had to take a deep breath before reading on. I doubled back to see if I read this correctly. "One or more persons can be selected at random from all eligible persons who were unvaccinated on that day and in that region, with exclusion of unvaccinated persons selected for a comparison made on a previous day."

If they randomly selected unvaccinated, then the random sample of unvaccinated would have the same statistical composition as the population of unvaccinated. This ensures no correction for any kind of biases. And yet at the end of this process, they claim to have found the ITT effect as if a randomized clinical trial were run!

The second mechanism for selecting unvaccinated is by matching on key "characteristics that reflect Covid-19 risk". This method is definitely more acceptable as it at least attempts to make the two treatment groups more comparable.

In practice, the matching method faces several big challenges (many of which I have discussed before on this blog):

• the matchable population is a small proportion of the total population, e.g. most studies using matching heavily undersamples older people and oversamples younger people. The method is guaranteed to undersample demographic segments in which vaccination rate is high, and oversample demographic segments in which vaccination rate is low
  
  
• VE in the matched population may be different from VE in the general population. The estimated VE in the matched population is surely not the ITT for the entire population
  
  
• most important confounders are unmeasured (e.g. taking other mitigation measures, vaccine hesitancy, exposure to virus). Matching does not infer balance on non-matching variables, a key reason why ex-post matching on observational data cannot replace randomized treatment assignment
  
  
• most countries enforce highly targeted vaccination programs which generate irreducible biases, e.g. if older people are encouraged to get vaccinated first, and have done so in large numbers (say 80%), what's the chance that the 20% unvaccinated older people are comparable to the other 80% vaccinated older people?

***

There is another aspect of the observational studies that are buried away from the summary of the methodology. In the sentences cited above, they said that the study subjects were "followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first." That surely cannot be true!

Another event that happens is vaccination. For people in the unvaccinated group, by definition, they have not been vaccinated up to the day of enrollment; however, many or most of them would get the vaccine during the follow-up period. How did the data analyst deal with this event? We are not given a hint in this methodology section.

Upon vaccination, clearly the follow-up period for the unvaccinated person must end. We simply don't know the counterfactual: what would have happened if the person did not get vaccinated? If we follow the time series of vaccine efficacy from day of vaccination forward, more and more people in the unvaccinated group are dropped from follow-up because they got themselves vaccinated. The people who stay in the unvaccinated group for a long time are the vaccine holdouts, who are clearly different from the general population, e.g. they are much less likely to take other precautions such as wearing masks, distancing, avoiding crowds, etc. How well does matching on demographics and basic health data control for these differences?

If those differences are not adequately accounted for, then the right side of the VE curve cannot be trusted.

***

Also, note that variability is artificially suppressed in this type of study. Anyone who got vaccinated during the study period appear once in the vaccinated group on the day of vaccination, but also may appear in the unvaccinated group on any day prior to vaccination. Unlike a typical RCT in which there are no overlaps in the group membership, in this type of observational studies, the majority of the people show up in both groups. So, if we subsequently run regression analysis comparing the two groups, adjusting for demographics and other factors, the variance between groups is markedly under-stated!

 

 

Game of telephone for data messaging

We all know about the game of telephone. If one person says something to another, and the second person to a third, and so on, the message received at the end of the chain will be substantively different. What gets me thinking about this today is an article by Gelman and friends, which essentially describes the game of telephone as it applies to data messaging: "Through endless repetition, numbers of dubious origin take on the veneer of scientific fact, in many cases in the context of vital public-policy debates."

I love their leading example. Apparently, (at least one) university in the U.S. circulated a public-service notice telling people to wear a hat on cold days because "most body heat is lost through the top of the head." The authors attempted to trace the origin of this message, and discovered that the advice purportedly came from a statistic, which claims that 50 or even 80 percent of body heat is lost through the (apparently cap-less) head.

[Most of the article is about tracing statistics on "corruption", and about the concern that politicians mislead the public using junk statistics.]

***

This little scenario reveals a number of things about our relationship with data.

First, repeating data seems to follow what Kahneman calls "fast thinking", not "slow thinking". People tend not to slow down, and apply logic, and ask: could we really be losing 80 percent of body heat through our head? Could a human being survive after losing 80 percent of its body heat? (Apparently, the more correct answer is 10 percent.)

Second, because of the fad built upon "unconventional wisdom," spread by the likes of Malcolm Gladwell and Steven Levitt, people may believe illogical data -- the more bizarre, the better. Shock value is the currency of social media, so this fad lives.

Third, I'm interested not just in the likelihood that the message has changed through repetition, but in what direction the message shifts. My working assumption is that the default setting of most people is story-first (as opposed to data-first). The way data are consumed is typically to support some hypothesis, i.e. preconceived notion. In the game of telephone, the message is continuously shaped by the story-first mentality of the messenger.

This idea is an expanded version of what the authors called "decorative statistics", i.e. making numbers "sound big and impressive".

Each person receives a message, but only partially due to memory failure. Each person has its own belief, such as about the effect of not wearing hats on cold days. The outgoing message will tend to be pulled toward those prior beliefs.

When I find some time, I should run a simulation using Bayes' Rule to propagate messages at each stage and explore the above idea more rigorously.

***

Another factor enables this game of telephone in data stories. Every statistic ever reported has nuances - it is only true for specific populations, after certain exclusions, accounting for particular biases, etc. However, in messaging data, we don't have the luxury of attaching two paragraphs of explanations for each number. Therefore, the nuances are lost.

The orphaned data are out in the open. This invites anyone to adopt these statistics and shape them to their own liking. Fill in the blanks. If one is completely rigorous, one plugs the gaps by researching the origins of the data, but who has the time, right? So, instead, one just adds whatever one thinks "makes sense"... this is where we ends up in a story-first mode, because making sense is in the context of that preconceived story. If we want people to wear hats, we like the story that hats keep heat contained in the body, and if we can't quite remember the statistic, we are susceptible to guesstimating a number that supports our story.

 

Covid vaccines contain PEDs, according to an athlete

Another sport has the doping demons. News arrived that U.S. triathlete Collin Chartier has been banned for three years after testing positive for EPO, a favorite performance-enhancing drug for endurance sports.

Doping is featured in my book Numbers Rule Your World (link), written before Lance Armstrong finally admitted to doping after vehement denials for years. I made several surprising observations. Firstly, if you or I are running an anti-doping lab, we'd calibrate the test to reduce false positives at the expense of false negatives, since a doper who is getting away with it isn't going to expose our error but a falsely accused clean athlete will surely erode our reputation. Secondly, and consequently, we expect most positives to be true positives. Thirdly, a high proportion of dopers are getting away with it.

To Chartier's credit, he did not invent excuses but admitted to the offence. As reported in the article, other triathletes have paraded out the usual unlikely excuses - one even claimed that a "tainted burrito" and a "tainted Covid-19 vaccine" were responsible for the positive tests.

Now, if we believe Chartier (and many other athletes who got caught doping), he only took EPO once and only after the two greatest wins of his career. He also claimed that no one else but him knew about the doping, and his coach, who is related to the world champion of the sport, said he was blindsided by his own athlete.

***

I'm still waiting for the day when an athlete shakes up the establishment by admitting all the negative tests s/he got while doping. That would take courage. Wait - Armstrong frequently said he was the most tested athlete on Earth while he was getting away with it. Does that count?

Long live anonymity

Anonymity is in my thoughts at the moment.

First, I learned that the U.S. Census Bureau has moved from paper surveys to online surveys. What does this mean for its central promise of confidentiality? Here is NPR's description of this promise:

Under current law, the federal government is not allowed to release personally identifiable information from the census until 72 years after it's gathered for the constitutionally mandated tally. The bureau has relied on that promise of confidentiality to get many of the country's residents to volunteer their information once a decade, especially among people of color, immigrants and other historically undercounted groups who may be unsure about how their responses could be used against them.

Responding to the Census is mandatory, and yet the Census Bureau does not offer anonymity - instead, it claims to protect our sensitive data. This is exactly what the tech industry says, and yet we all know our data aren't really protected: the truth is that sooner or later, our data will get leaked.

Anything online is much more easily leaked than anything on paper. If the information is on paper, the thief has to be present physically, and s/he would need a truck to steal them - after which it would take a lot of time to scan or copy the data. The efficiency and compression gains plus worldwide access from powerful technologies make large-scale theft much quicker to execute.

***

Second, at a recent conference, I heard about a scheme designed to protect "sensitive" variables from online/mobile advertisers. They notably missed the simplest solution, which is to not collect, store, or use such sensitive data.

By "they", think any of the big tech players, like Google, Microsoft, Meta, etc. In their world, they want to use sensitive data to target advertising and extract revenues from advertisers, while they "protect" their users from the prying eyes of these advertisers. So, the advertisers are cast as the bad players while they themselves are good.

The speaker did not define what he meant by sensitive data. I imagine things like someone being a political activist (especially if they live under a repressive regime), sexual orientation, divorce status, disability status, etc.

The main idea of the talk appears to be using clustering to hide sensitive information about individuals. Each user has an array of sensitive and non-sensitive variables. A cluster analysis forms groups of users that are similar across the entire array of variables, and the advertisers will be provided cluster identifiers for users, rather than the variables.

There seems to be a flaw in this logic. For example, if the sensitive variable we are trying to protect is pro-choice activism, and the clustering analysis is sharp, then we may find a cluster in which all or almost all individuals are pro-choice activists, i.e. the cluster is dominated by a single sensitive variable. In this case, not only is the sensitive information exposed but the clustering strategy has backfired - as an adversary is now able to learn the sensitive variable for a whole group via the cluster identifier.

The definition of sensitive information is itself a complex problem. For example, if I am short, and our society discriminates against short people, I might want to hide my height, which makes it a sensitive variable for me - although the tall people would see it as an advantage, and not think of height as sensitive.

 

 

 

Too good to be true instinct

Good data analysts have developed an instinct of perking up when results are "too good to be true". This is a must-have attribute that I'm always looking for in a job candidate. It is a protective instinct, a response that protects the analysts from getting fooled by the data. Nassim Taleb called this type of thing "fooled by randomness". The temptation is not randomness itself but the lack thereof.

(Credit: Sonja Alves, Flickr)

Observational studies of the Covid-19 vaccines are a case in point. I have been spooked by the consistency of estimates of vaccine effectiveness. Almost all peer-reviewed studies and most working papers that get mainstream attention contain values of VE that are in the 70-90 percent range. (Since these are relative ratios, a 50-percent multiple improvement is already a huge effect.) There is a consistency across headline results of different studies, and there is also a consistency of values inside individual studies, across segments, using different outcome metrics, different methodologies, etc. etc.

Below is Table 2 from the traffic accidents study that I have been reviewing (link):

The aggregate result (derived from the inadmissible non-adjusted comparison discussed in the first post of this series) is shown in the last row and represents a value of 1.7x. The rows above come from applying the same methodology to subsets of the study population, some of them are mutually exclusive while others are overlapping. Most of the values line up close to or near that 1.7x value. There is only one segment (over 65) that has the opposite direction but barely. What does this consistency tell us?

Shown below is the top part of Table 3. Here, the researchers analyzed many variants of the outcome, applying the same analytical strategy. Whereas Figure 2 above shows any involvement in traffic accidents, Table 3 below breaks down these accidents into subtypes, like involvement as driver versus pedestrian, time of day and so on.

The consistency is once again remarkable. Whereas the aggregate number is shown as 1.7x (top row), all the other numbers ranged from 1.5x to 5.1x. Almost all the variance is on the high side of the aggregate. What does such consistency tell us?

In the Appendix, the authors rolled out even more rows of analysis:

Again, all effects are in the range 1.4x to 1.7x. What does this consistency mean? That's what I want to explore in this post.

***

In the mainstream, such consistency is portrayed as resounding evidence. Each individual study that generates another impressive relative ratio similar to the previous result is regarded as another confirmation. The authors of the traffic accidents study believe that because everywhere they looked, they saw the same consistent number, it must mean that they have found something really important - like Newton's Law of Gravity.

Unlike Newton, they aren't studying the realm of physics. The study's data concern human behavior, and social behavior, and it is actually shocking, rather than comforting, to find a law of nature in this arena. My prior - from decades looking at data about humans - is that (a) aggregate effects are typically small, so that even a 10% improvement is remarkable, and (b) there is substantial variability when looking across different segments, different metrics, etc., which is another way to say that "interaction effects" are important.

Were these researchers, mainstream scientists, and the mass media being fooled by (lack of) randomness?

***

Other than discovering some immutable law of nature, what are other possible explanations for the remarkable consistency?

File Drawer Effect

Let's say there is a medical journal editor who only publishes vaccine studies that show high vaccine efficacy - since anything else has little societal value. The result of this editorial practice is that (almost) every study that sees the light of day will definitely have a VE estimate above some lower bound (say, 70%). In this scenario, the consistency of all the peer-reviewed results may have nothing to do with the vaccines themselves, but can be explained by publication bias that is filtering out studies below that bound. This bias is sometimes called the "file drawer effect," describing researchers who stack away "failed" studies (e.g. p-value > 5%) in their file cabinets since journals would reject them.

A consequence of the file drawer effect is that the average of the published results over-estimates the true vaccine efficacy.

In the context of Covid-19 vaccines, the file drawer effect may be related to the phenomonon of cargo cult science.

 

Cargo Cult Science

Cargo cult science is a term coined by physicist Richard Feynman. Roughly speaking, he warns against the potential of bad science even when scientists follow so-called best practices.

Other than the initial clinical trials conducted in 2020 (which were scientifically aborted upon emergency use authorization when the placebo groups were destroyed), all evidence of Covid-19 vaccine efficacy have come from observational datasets. All observational data sets either lack controls, or if controls existed, treatment status was not randomly assigned, which means that unvaccinated people differ from vaccinated people in multiple dimensions besides vaccination status. Therefore, it is wrong to attribute all observed differences between the two groups to vaccination only.

Let's teleport back to the first part of 2021, weeks after the mass vaccination campaign commenced, when the first observational study appeared in print. For the sake of argument, let's say the first analysis showed a VE of 50%. How do we know whether this estimate is to be believed since the people who got vaccinated were clearly different from those who didn't? The honest answer should be no one knows because there is no ground truth in an observational study.

What circumstantial evidence do we have? The only breadcrumbs are interim analysis results from randomized clinical trials that produced VE of 90%+. We face a gnarly problem because our hypothetical observational study led to a VE estimate of 50%. One of the numbers is not like the other; they cannot both be correct. But which study is wrong?

Not surprisingly, most would give more credence to the RCT results, since RCTs are considered the gold standard of medical research. Therefore, since our observational study came in much lower, we'd go back to the drawing board, and massage the dataset some more... until the estimated VE is closer to 90%, making it more credible. The additional data massaging is justified by best practices - we are simply addressing residual confounding bias.

(It is important to realize that randomized experiments can be analyzed in a straightforward, standardized manner but observational studies rely on statistical adjustments that involve subjective judgment from researchers. Analysts try to guess what biases exist in the data, and device ways of correcting for them. It's an iterative process with limited guardrails.)

To make sure you get what's going on, think about what happens if the first analysis of an observational dataset leads to a VE value of 90%. This result would be immediately accepted. Nevertheless, it's not clear that this research team did a better job adjusting for bias than the previous team who came up with 50%.

Since the RCT result was the only piece of available evidence when the first observational study showed up, it became a filter for all subsequent observational studies. Any observational study that produces a number close to 90% feels comfortable, and any study that shows a number far from 90% feels "wrong". Such an editorial process directly triggers the file drawer effect.

***

I don't know how we landed on this stupid state of affairs. Using an observational study to "confirm" an RCT result - the gold standard - is like asking a C student to grade an A student. After the teacher has already given the A student a top grade, what additional information is gleaned by having a C student grade the A student's exam?

What's happening is even more dubious. The C student is not grading the same exam that the teacher saw - the C student is asked to grade a different exam submitted by the A student. By this, I mean: the observational studies are different from the RCTs in meaningful ways, e.g. timing of vaccinations, incidence of Covid-19, types of people eligible for vaccination, residence of patients, no placebo shots... There is scant justification for assuming that the observational studies should produce VE values in the same ballpark as the RCTs!

When another observational study comes out proclaiming 70-90% VE, it doesn't mean much. It just tells me that the research team has selected a set of adjustments so that the VE estimate lands in the zone that is considered acceptable for publication, i.e. it falls in the vicinity of the RCT result.

 

Lurking Variables

Another way in which we can get fooled by the consistency of results is via unknown or unexplored sources of bias. All observational studies can only control for factors for which there are measurements. I haven't encountered yet an observational study of Covid-19 vaccines in which the researchers collected their own data. The current practice is to pull data from pre-existing, government databases. So the variables available are typically limited to basic demographics (age, gender, etc.) or health-related data (prior health concerns, healthcare usage, etc.). Important factors such as attitudes toward freedom and government authority, risk tolerance and exposure to the virus, are not measured, and not adjusted for.

If key sources of bias are not adjusted for, they could explain why the results are so consistent among the subgroups; each such result contains residual confounding from the missing variables.

The missing variable may be a common cause of both vaccination status and infection outcome, for example, people who are generally more health conscious and healthier pre-Covid. They are both more likely to get vaccinated and less likely to get infected (regardless of vaccination status). Since the government databases have no labels for this type of people, there is no way to correct this bias. It is delusional to think that dumping age and gender variables into a regression equation properly adjusted for such a bias. There are health-conscious people in every agexgender segment! In each such demographic segment, vaccination status is still confounded with health-seeking status.

 

Multiple Comparisons

When so many analyses are performed using so many metrics, the best practice is to adjust for multiple comparisons. The general idea is that if one screens a lot of metrics, a few of them will accidentally show statistical significance, and those results are highly likely to be spurious. In other words, without adjustment, traditional statistical methods underestimate the degree of variability in the outcomes. That is to say, results appear more consistent than they are. Yet, most of these vaccine studies sidestep this issue.

I'd be shocked if elementary medical statistics courses don't cover the problem of multiple comparisons. Practitioners in this discipline seem to think that fishing for significance is fine so long as they pre-specify all their intended analyses. Actually, pre-registration does not solve the problem of multiple comparisons at all, unless the pre-registered statistical analysis explicitly outlines a strategy for dealing with it. (Unfortunately, the randomized trials also pre-registered multiple comparisons without correction.)

 

Poorly Defined Metrics

Another reason for consistent results across many analyses is a poorly chosen scale for one's outcome metric. An analogy to grade inflation in schools is telling. American schools has a grade-point average (GPA) typically defined on a scale of 0-4, 4 being grade A. Due to the grade inflation trend, most of the grades given out are As so in reality, the majority of the scale 0-3.5 is basically empty, with almost all students squeezed into the 3.5-4.0 range. Of course, all students look more or less the same.

I think something like this is at play with the relative ratio scales used to measure vaccines. I think an ineffective vaccine will show a VE substantively higher than 0% so the grading scale is compressed just like in schools.

***

In this post, I have offered five possible reasons why every research study of Covid-19 vaccines seems to offer similar effectiveness estimates, and why there appears to be little or no variability when aspects of the methodology are altered. As a statistician who have analyzed a lot of human data, I find it least likely that the reason for the consistency is that these vaccines are like a law of nature, such an overwhelming force that flattens all human variability.

Is this data science professor a victim or a villain?

Apparently, some secretive entity is trying to maximally embarrass Forbes magazine as much as possible. One by one, they are exposing people on Forbes' 30 under 30 or 40 under 40 list of young entrepreneurs as fraudsters. Elizabeth Holmes has already been found guilty of fraud in her medical tech startup. Sam Bankman-Fried is being sued for running a type of Ponzi scheme in his crypto startup. And now, Charlie Javice is being sued by JPMorgan for faking customer lists in her college financial aid app startup (named Frank).

According to the Wall Street Journal (link with paywall), an unnamed data science professor played a role in this fraud, and that definitely caught my attention.

The unnamed professor, who was paid $18,000, used computer-generated information to show Frank’s purported user base, including details like names, birthdays and schools they attended, the suit alleged...“Our plan was to sample first name and last name independently and then ensure none of the sampled names are real,” the suit alleges the professor wrote to Ms. Javice.

This sounds very bad. Apparently, JPMorgan alleged that about 90% of the data were faked.

However, I don't think we can read from this excerpt whether the professor has also been duped, or the professor was part of the con. It could be the case that Javice contracted the professor to generate simulated data, supposedly for some legitimate purpose.

The other reason this case strikes a chord with me is that many of the emerging data science/AI technologies can easily be exploited to deceive and hurt people, and there does not appear to be much serious discussion of this problem.

 

P.S. [1/19/2023] My friend Mark P. sent me a note saying that some people on Twitter who has read the lawsuit learned that Excel is featured in it. Apparently, JP Morgan got suspicious when they received an Excel spreadsheet with the customer data, which cut off at ~1 million rows, which is the row limit for Excel files.

I have to say I don't really understand this point because I'd have thought that their IT people didn't realize that 4 million names don't fit into an Excel spreadsheet; I wouldn't have immediately assumed they had fewer customers than they claimed.

Beneath the surface of recent Google, Elon, FCC news

Several recent news items caught my eye.

First, there's been endless drama coming out of Twitter since Elon Musk acquired it for an ungodly sum. One of his latest bombshells is to promise an email drop that "makes public all internal discussions about the decision to censor @NYPost's story on Hunter Biden's laptop" in order to "restore public trust" (link). It's just one example of how corporate executives can weaponize data.

Second, Wired reported that Google handed over the personally identifiable information (PII) and detailed location data of at least 1,500 people to FBI investigators (link), through "geofence warrants", which is a dragnet. Geofencing is a favorite technique also of marketers for finding targets to deliver their marketing messages: the marketer draws an area on the map, and the data providers hand over contact information of every cellphone that is known to have entered the geofenced area. In this case, the FBI requested information on anyone who entered the area around the Capitol with their cellphones on January 6, 2020.

Third, the FCC blocks several leading Chinese technology manufacturers from the U.S. market (link) in the name of protecting national security. This action is taken based on the "possibility" of "espionage, IP theft, blackail, foreign influence campaigns, and other nefarious activities".

***

These stories are all related to our data. What happens to our data. Who has our data. What can people do with our data. What have they done with our data.

Americans have allowed corporations to be custodians of our private data. This was originally cast as a grand bargain, in which corporations get our data for free while they offer us some service of value for free. Think back to the beginnings of Google and Facebook... scratch that, not the humble beginnings when the entrepreneurs lured us in by offering their services for free without a quid quo pro. It was after these companies went public when they realized they must make money, and eventually, every one of these outfits has become a hoarder of private data, which are sold to the highest bidders. Every new digital product now is conceived as a way to augment the profiles of the users. This includes paid services (e.g. Amazon and any number of subscription services) which also collect the same types of data, and trade them in an obscure market... without the grand bargain.

***

Many - most, perhaps - of us have decided to trust Google, Twitter and similar companies with our data. Some of us prefer to trust our corporations rather than our governments. But can we really trust corporations to act in our best interests?

What is our best interest? It depends. If you're one of those people who entered the Capitol on January 6, your best interest is for Google not to send your location data to the FBI. If you happen to be around the Capitol on January 6 but not involved in the election politics, you don't want investigators to be probing you just because you were in the neighborhood.

If you are one of the anti-government protestors in China, Iran, Russia, etc., your best interest is for Google not to send your location data to the investigators. Now, is it any different if you're protesting against a Western democratic government?

Closer to home, if you happen to live in one of those states that have banned abortion, and also encourage your neighbors to report on you to authorities, your best interest is for Google not to sell your location data to the highest bidder. If your neighbors are pro-life activists, their best interest is for Google to release geofenced location data around known abortion clinics so you can fulfil your civic duty to report on people and doctors who skirted the abortion ban.

Do we really believe that corporations are making these decisions based on our individual desires, or based on what their lawyers think is the best course of action for the corporate interest?

If the government entity is collecting our data, there are constitutional rights that govern what this entity can do. There are Freedom of Information Act (FOIA) provisions that citizens can use to discover if laws have been broken. There are no FOIA requirements that regulate corporate behavior. There is no punishment for companies issuing false statements when not under oath.

***

Elon Musk's behavior provides a glimpse at the answer. His threat to release a load of internal emails relating to Twitter's moderation process is a direct threat to weaponize data. He's not the first corporate executive to do this sort of thing: remember how a Uber executive casually told a reporter that she's being tracked during a time when another Uber executive threatened to dig dirt on journalists who wrote critical articles about Uber (link).

If Twitter will release internal emails to make a political point, it certainly has the means to release user data if an executive finds a reason to do so. This is selective, targeted disclosure of potentially private data. Musk is not releasing all emails about all banned accounts, but emails about one particular political figure that happens to belong to a political party he is on the record of opposing.

There is no way to verify that all emails related to that one situation would be released. No one outside of Twitter has access to this data. If there are any emails that could place Twitter in legal jeopardy, what's the chance that the corporate lawyers would allow them to be released? Would these lawyers play government and redact the documents?

Remember, besides the messages that Twitter users post publicly, there is a whole hidden universe of direct messages which are private conversations between people. Activists in authoritarian countries are known to use platforms like Twitter to hide communications from prying eyes.

Musk's announcement is much more than about Hunter Biden's laptop. It is an admission that private messages are not really private. They are in the hands of the companies which own these platforms. It only takes one CEO or executive with a grudge to weaponize someone's data. (Junior staff can also wage their own small-scale vendettas as many people have access to data as part of their work.)

***

The banning of Chinese tech companies is based on the possibility that the Chinese government may get access to private data from Chinese companies, and the possibility that such data would be used to harm people.

In Twitter's story, the CEO articulated a motive and a plan to selectively release data. He just hasn't pushed the button yet.

By contrast, Google pushed the button when they transferred data to the FBI. It's not anonymized data, either. It's personally identifiable information plus detailed maps of these people's locations. (While the Wired article singled out Google, it's likely that other big tech companies, like Microsoft and Apple, and telecoms companies, may have done something similar.)

 

P.S. [12/4/2022] In the evening of the day I put up this post and the next day, Elon Musk pushed the button and released the data so we now have two instances in which prominent U.S. corporations released data with the express purpose of harming the people who generated the data, plus an allegation that Chinese companies may do similar things.