Big Data, Plainly Spoken (aka Numbers Rule Your World)

Critics, stories, and ethics

Chance magazine (link, paywalled) has a nice interview with Howard Hogan, who was Chief Demographer at the U.S. Census Bureau in the current issue. It initially caught my attention because he was a fellow Princeton graduate, who was lucky enough to be there during John Tukey's time. At the end of the interview, he was asked to give his "Sunset Salvo", which was Tukey's term for words of wisdom at the time of retirement.

He made two observations, both of which resonate strongly with me.

"If nobody's criticizing you, that means you're not working on anything important."

It is natural to be defensive about one's work but in a field like data science or statistics, which is about shades of gray - and only interesting because of it - one simply cannot do good work unless one hears about different ways of approaching the data, and what different results one obtains. Providing constructive critique requires the critic to invest significant time to decipher what the analyst has done, in addition to thinking about alternatives, which provides the context upon which an analysis can be judged.

"If you want to make an impact, you don't just get numbers, you get a story."

This is a counter-intuitive point that frequently makes practitioners uncomfortable. Statistical thinking is mostly about generalizing the observed data, drawing conclusions on a wider population than the sample. A "data story" is most often a specific example. In the interview, Hogan explained "imputation" by saying: if in the Supreme Court Library, one finds books labeled Title 11, 12, 14, 15, then one can guess with good confidence that Title 13 exists, i.e. is absent because someone checked it out. So a data story implies a kind of U-turn: the analyst has looked at the specific sample of data, and generalized them to a broader statement, and then selected a specific single example to explain that statement to an audience. It's a bit counter-intuitive. In practice, it's a highly effective technique.

***

In the interview, Hogan described his experience updating ethical guidelines for statisticians. He reported that the committee decided that there is nothing more to say about ethics in the age of Big Data.

I disagree with that conclusion. Big Data raise or exacerbate issues that are not relevant with "small" datasets.

First, the sheer number of variables being collected about each person means that it is much easier to de-anonymize Big Data than small datasets.

Second, the impact of unintended de-anonymization of the data scales with the size of the sample. With small data, a small number of people are adversely affected. With Big Data, a large number of people are hurt.

Third, more data allow more linking. It's easy to mismatch people's data - one of many reasons for this is that many people share the same names. It's true this is a pre-existing problem that has vastly expanded in reach because of the exponentially larger number of linkages made possible by Big Data.

Fourth, more linkages mean more "guilt by association". The impact of this may be mild for Census data which has no individual implications - but when businesses do such linkage with their Big Data, they make decisions that impact individuals.

 

 

What about safety

Readers may have wondered why I have said little about vaccine safety in my reviews of vaccine trials. Today's post, which fills that gap, also serves to explain why.

As is well known, the FDA proclaimed that the mRNA vaccines (marketed by Pfizer and Moderna) are both efficacious and safe. Each trial enrolled about forty thousand participants, split randomly into vaccine and placebo groups, with the latter receiving a saline solution instead of the vaccine.

For efficacy, the primary outcome of interest was infections (measured by its proxy, reported symptomatic cases). As observational data drifted away from this narrative, the outcome shifted to hospitalization and death attributed to Covid-19. Whether the endpoint is cases, hospitalization or death, the clinical trial design ensures that an analyst can look at the difference (ratio) of the event rates to assess whether the vaccine performed better than the placebo. [That is, until after the emergency use approval was granted, because the FDA then allowed the pharmas to dismantle the placebo groups, destroying the trial design.]

Measuring safety - on the surface - may proceed in a similar way. However, safety outcomes present two enormous challenges that make safety analyses much harder - and thus less convincing - than efficacy analyses for the same cohort of trial participants.

First, it's difficult to know which safety outcomes are of greatest concern. It's a bit of a fishing expedition. Lots of outcomes are measured to screen for signals. Such data mining always raises the probability of false signals. Thus, all else equal, one would require a larger sample size to analyze safety outcomes. Since the sample size is fixed for both safety and efficacy outcomes, this implies that the confidence we have for safety is necessarily lower.

Second, the truly horrible safety outcomes - known as "severe adverse events" (SAEs), which include hospitalization and death - are expected to be rare events. Since a vaccine is indiscriminately injected into the population, most of whom are healthy people, even a small rate of occurrence can be alarming. Let's assume a hypothetical vaccine causes death with probability 1 in 500,000 shots (this is a thought experiment for illustration purpose only). Bloomberg reported that over 600 million shots have been administered in the U.S. alone. On average, 1,200 people would have died from this vaccination.

I chose 1 in 500,000 as the hypothetical rate of vaccine-induced death for a reason - the vaccine trials as designed would not have been able to detect this signal, let alone death rates below that level. For an event that happens at a rate of 1 in 500,000 attempts, the chance of observing zero occurrences in 25,000 people is 95%, and the chance of observing a maximum of two occurrences is 99.99999%. In other words, any SAE with a rate of occurrence of 1 in 500,000 would likely produce no events during the vaccine trials, and given the amount of shots that went into arms, that level of harm would have killed 1,200 people (while the analyst legitimately touts a perfectly safe vaccine).

***
In their respective trials, at the interim analysis that happened when half the participants have reached two months of observation beyond the second shot, Pfizer and Moderna observed few SAEs, and concluded that these event rates were "balanced" between the vaccine and placebo arms. This was the basis of the safety claim.

A group of researchers have recently revisited the analysis, and drawn a different conclusion in a peer-reviewed article published in the journal Vaccine. (link) They argued that the mRNA vaccines caused an excess of 10-15 extra SAEs of special interest relative to placebo. That's a difference. As a ratio, the mRNA vaccines caused a 43% increase in SAE risk. Most of the excess risk in the category known as "coagulation disorders".

It's instructive to learn why the same data led to different conclusions.

The first difference is what is counted as an SAE. It turns out that the FDA and U.S. pharmas created their own definition of what counts as SAEs. Meanwhile, the international community, something called the "Brighton Collaboration", agreed on a priority list. The new analysis reclassifies the events reported by Pfizer and Moderna in an attempt to perform an analysis that follows the international standard. Even here, the researchers conducted two analyses, based on the "original" list and an "expanded" list.

[For those wanting to peep into the sausage factory, the reclassification involves three experts readjudicating the events based on subjective evaluation of case history and other information. The original classification during the trials also involved such subjective judgment.]

The second difference is the proper handling of differential observation time among participants. Remember that the interim analysis were conducted while people were still in trial. I noted when I read the efficacy analyses the fact that the earlier participants were observed for longer periods of time, so that, even though the "median" participant reached the two-month point post second dose, the interim analysis involved many participants who had not yet reached the start of the case-counting window, which means that for the purpose of the interim analysis, these participants were "immortal"!

It turned out that the FDA-endorsed analysis included participants who have not yet received the second shot, since they counted any SAE at data cutoff, i.e. the same cutoff date applied to all participants regardless of enrollment date. The new analysis removed participants who had not yet taken the second shots.

***

A few other noteworthy nuggets can be found in the article. (link)

First, the authors noted that the Pfizer (Moderna) vaccine reduced risk of hospitalization by Covid-19 infection of 6 (2) per 100,000. So, for every 100,000 vaccine recipients, around 2-6 are expected to have avoided hospitalization relative to not getting vaccinated, but 10-15 are expected to experience serious adverse events relative to not getting vaccinated. As the authors noted, such a benefit-harm analysis should be conducted by age group but one of the enduring mysteries of Covid-19 vaccine science is the lack of available data of sufficient granularity for third-party researchers to conduct independent studies - all the more baffling given the claim of massive benefit and zero harm, which presumably should withstand even moderate- to large-sized error bars!

Just for illustration. The 2-6 hospitalizations are likely concentrated in older people and if the 10-15 excess SAEs occur mostly in younger people, then the risk in the young is higher than average while the benefit is lower than average.

Second, Pfizer stopped reporting safety outcomes beyond one month after the second shot.

Third, Moderna counted Covid-induced SAEs in the SAE analysis, which the authors pointed out is not the "traditional" analysis. (Since supposedly the vaccine's side effects are similar to the effect of Covid-19 infection, differentiating between them requires experts to determine the cause of the observed SAEs. That's definitely not something you or I am qualified to do!)

 

 

 

The interpretability business

While running down the rabbit hole following a recent post at the Gelman blog, I found an appetizing quote from Martha S. on a post from 2016 (link).

The topic is ostensibly about the Big Five personality test. One of the five dimensions is named "openness to experience". Martha wrote:

If I’m not mistaken, the label “openness to experience” was an ex-post-facto label that then became taken as a definition. To elaborate, my understanding is that the process was as follows:

The initial research started not by making “definitions” of personality traits, but by making lists of questions that the researchers thought might capture the varied aspects of personality. They administered the resulting questionnaire to a group of people and ran a factor analysis on the resulting data. This spit out a list of “factors” (linear combinations of the question responses) that “accounted for” (relatively) high proportions of the total variance in responses. They decided that the first five (ie, the five with the highest amount of total variance accounted for) gave a pretty good list of the major “traits” of personality. They then gave names to these traits.

In other words, they did not start with definitions of traits; this was exploratory research that gave them candidates for traits. The real definition of the traits was “whatever this linear combination measures”. However, the labels they attached to these factors became “reified” — that is, taken to be The Real Thing Measured, even thought the labels were fuzzy terms subject to varying interpretations.

***

I think this quote is very relevant for anyone who's working in the area of "interpretable" models, which has become particularly popular in the deep learning community. Martha's point is that the Big Five personality traits is "an" interpretation of the model (linear combinations), which then got reified into "the" interpretation.

Martha was responding to a comment by another reader Diana S. who complained that the Big Five dimension "openness to experience" is not well defined. She wrote:

researchers seem to take this to mean “openness to an ever-changing array of experiences.” That is, someone who reads five different books would be considered more “open to experience” than someone who reads the same book five times–even if the latter person finds new things with each rereading. As I see it, someone who rereads may be at least as open to experience as the person always reaching for a new book–but the quality manifests itself differently.

This comment was made to illuminate why we need to have precise definitions of things we're measuring.

The generally accepted definition Diana cited has strayed from how the original dimension was conceived. In fact, Martha suggested that the original definition does not actually exist. Well, it exists in a purely mathematical - abstract - form: the precise definition in the original analysis would be the linear combination of responses to the test, i.e. a set of coefficients. But this precise definition is devoid of practical meaning. How does one interpret 5.3258*Q1 vs 6.2335*Q1?

***

Also relevant is Efron's paper which did a number on measures of "importance" that get pumped out with every machine learning black box model. See my previous post here.