Big Data, Plainly Spoken (aka Numbers Rule Your World)

Book review: Don't Trust Your Gut

Seth Stephens-Davidowitz has a new book out early this year, "Don't Trust Your Gut", which he kindly sent me for review. The book is Malcolm Gladwell meets Tim Ferriss - part counter intuition, part self help. Seth tackles big questions: how to find love? how to raise kids? how to get rich? how to be happier? He invariantly believes that big data reveal universal truths on such matters.

In 2013, I speculated about the challenges faced by big data analysts in this piece for Significance. Big Data has become a real thing in the decade since its publication. Seth's book interests me as a progress report on the state of "big data analytics".

If there is a common core of this style of analytics, as demonstrated by Seth's examples, I'd include the following elements:

a) Big Data is less defined by the amount of data, but by several characteristics I previously labeled OCCAM (see this post).

The data are typically collected by passive observation (e.g. tax records, dating app usage, artist exhibit schedules). Meaningful controls are absent (e.g. no non-app users, no failed artists). The dataset is believed to be complete. The data aren't specifically collected for the analysis (an important exception is the happiness data collected from apps for that specific purpose). Several datasets are merged to investigate correlations.

b) Much - though not all - of the analyses use the most rudimentary statistics, such as statistical averages.

This can be appropriate, if one insists one has all the data, or "essentially" all. An unstated axiom is that the sheer quantity of data crowds out any bias. This is not a new belief: as long as Google has existed, marketing analysts have always claimed that Google search data are fully representative of all searches since Google dominates the market. (Seth's previous book covers his analyses of Google search results.)

c) If the analyst incorporates model adjustments, these adjusted models are treated as full cures of all statistical concerns.

The last few chapters on activities that cause happiness or unhappiness report numerous results from adjusted models of underlying data collected from 60,000 users of specially designed mobile apps. The researchers broke down 3 million logged events by 40 activity types, hour of day, day of week, season of year, location, among other factors. For argument's sake, let's say the users came from 100 places, ignore demographic segmentation, and apply zero exclusions. Then, the 3 million points fell into 40*24*7*4*100 = 2.7 million cells... unevenly but if evenly, each cell has an average of 1.1 events. That means many cells contain zero events. By the magic of model adjustments, all cells have non-zero estimated activities and associated happiness quotients. The estimates in many cells reflect an underlying model that hasn't been confirmed with data - and the credibility of these estimates rests with the reader's trust in the model structure.

I observed a similar phenonmenon when reading the well-known observational studies of Covid-19 vaccine effectiveness. Many of these studies adjust for age, an obvious confounder. Having included the age term, which quite a few studies proclaimed to be non-significant, the researchers spoke as if their models are free of any age bias. Taking this line of logic further, such a belief implies that no one should bother with running expensive randomized clinical trials because a self-selected trial coupled with post-hoc regression adjustments is a perfect substitute!

d) A blurred line barely delineates using data as explanation and as prescription.

Take, for example, the revelation that people who own real estate businesses have the highest chance of being a top 0.1% earner in the U.S., relative to other industries. This descriptive statistic is turned into a life hack, that people who want to get rich should start real-estate businesses. Nevertheless, being able to explain past data is different from being able to predict the future.

A kind of feedback loop is also at play. The average odds of getting rich surely depends on how many people are entering the industry. So if people read Seth's book, and rush into real estate, the data may no longer rank this industry as best.

e) Most of the featured big-data research aim to discover universal truths that apply to everyone.

For example, an eye-opening chart in the book shows that women who were rated bottom of the barrel in looks have half the chance of getting a response in a dating app when they messaged men in the most attractive bucket... but the absolute response was still about 30%. This produces the advice to send more messages to presumably "unattainable" prospects.

Such a conclusion assumes that the least attractive women are identical to the average women on factors other than attractiveness. It's possible that such women who approach the most attractive-looking men have other desirable assets that the average woman does not possess.

It's an irony because with "big data", it should be possible to slice and dice the data into many more segments, moving away from the world of "universal truths," which are statistical averages, said differently. In the last chapter of the book, Seth describes one study that tackles this problem.

***

I wish Seth had devoted more space to explaining model adjustments and statistical issues, and discussing whether the remedies are sufficient. His target audience is clearly people with minimal mathematical training, and I understand his decision not to open such cans of worms. The references at the end should satisfy those interested in exploring further.

As with Gladwell, I recommend reading this genre with a critical eye. Think of these books as offering fodder to exercise your critical thinking. Don't Trust Your Gut is a light read, with some intriguing results of which I was not previously aware. I enjoyed the book, and have kept pages of notes about the materials. The above comments should give you a guide should you want to go deeper into the analytical issues.

I think there is a lot more that can be done with big data, we are just seeing the tip of the iceberg. So I agree with Seth that the potential is there. Seth is more optimistic about the current state than I am.

The role of prior infection in the vaccine trials

In the final analysis of the J&J vaccine trial (link), the study authors suggested that their vaccine benefits those who have previously been infected with Covid-19:

We observed that participants with previous asymptomatic infection (defined by serologic positivity for SARS-CoV-2 N protein and an absence of history of symptomatic Covid-19) can benefit from immunization with a Covid-19 vaccine. In a post hoc analysis, previous infection alone provided 90.4% protection against symptomatic infection, and after administration of Ad26.COV2.S in seropositive participants, 97.7% protection was observed in a comparison with seronegative placebo recipients.

Attentive readers should be raising their eyebrows as they digest this paragraph. The headline vaccine efficacy (VE) reported in this study was 56%. Against this backdrop, the above claim of 98% efficacy for people who have already caught Covid-19 before (aka seropositive) is quite simply astounding.

[The authors, nevertheless, chose their words carefully, as the statement does not actually make a causal claim, but merely promotes such an interpretation.]

***

Let's unpack the underlying data.

The randomized trial design included the usual pair of treatment groups (vaccine and placebo). However, FDA allowed excluding many subgroups post-randomization in the statistical analysis. One such exclusion are people who were seropositive at baseline (actually, it appeared that they may have excluded also anyone who got infected before 14 days after the shot, an additional complication I will ignore in what follows).

We can subdivide the trial participants into four subgroups, as shown in the following table:

Roughly 10% of the vaccine and placebo groups were deemed seropositive (previously infected) at baseline and removed from the headline analysis. The "post-hoc" analysis brings the seropositive subgroups back into consideration.

The headline analysis ignores the first row of the table, and reports on the difference between the case rates between the placebo and vaccine groups in the second row (expressed as a relative ratio). As mentioned already, the VE is 56%, which means that the case rate of the vaccinated was roughly 44% that of the placebo group. Since we are allowed a causal interpretation in the context of randomized trials, the vaccine roughly cuts the case rate by half, from 5.5% to 2.5%.

If the VE were to be 98% instead of 56%, the case rate of 5.5% would have to drop to 0.1%. This is the first place where the paper's aforementioned claim can easily be misinterpreted. Because the VE is expressed as a relative ratio, a 98% drop is not the same as a 98% drop - because the base rates are not equal.

In this example, the base rates cannot be more different. The average case rate for sero-positive (i.e. with prior infection) is only one-tenth of that for sero-negative people. As shown below, on the placebo arm, those with prior infections had a case rate of 0.6% (despite not receiving the vaccine) compared to 5.5% if they did not have a prior infection.

As the study authors acknowledged, natural infection acts like a vaccine with 90% efficacy. Well, they used more scholarly words:

Previous infection alone, in an analysis involving seropositive and seronegative placebo recipients, was found to provide 90.4% (95% CI, 83.2 to 95.1) protection against moderate to severe–critical Covid-19.

Now, where did the 98% claim come from? It turns out the authors were making a comparison between two unlikely subgroups: (seropositive + vaccine) vs (seronegative + placebo).

This comparison is odd in a number of ways. Two conditions are varied at the same time. The 98% VE represents the aggregate effects of the vaccine plus prior infection. The study makes no attempt to assign relative importance between these two changes. In theory, the effect of the vaccine can range from all of the 98% to none of it.

From a practical view, each person either has caught Covid-19 before or hasn't. Unlike the vaccine decision, serological status is typically not a manipulable treatment (unless someone deliberately attends a "Covid party".)

If you're unvaccinated with a prior infection, then your decision is whether or not to get vaccinated. The appropriate comparison is (seropositive + placebo) vs (seropositive + vaccine). This is the first row of the table.

If you're unvaccinated without prior Covid-19, then your decision is - also - whether or not to get the vaccine. The appropriate comparison is (seronegative + placebo) vs (seronegative + vaccine). This question is the primary endpoint in the trial.

The vaccine efficacy for seropositive people is 76% (first row). This is quite a bit higher than the 56% for seronegative people (second row). Remember though that 76% represents a reduction of case rate by 76% - off a base of 0.6% while the 56% represents a reduction of case rate by 56% - off a base of 5.5% so they are not directly comparable values. Besides, at the time of the trial, seronegatives outnumber seropositives 9 to 1.

In glorifying numbers > 90%, the study authors picked the larger case rate from the placebo column and the smaller case rate from the vaccine column to produce a meaningless 98% number. A fuller analysis shows that the vaccine is decently effective for those who have been infected but the improvement is on top of a tiny case rate.

***

If we are serious about learning the effect of vaccination on those who have already been infected, what experiment should we run?

It's straightforward. Recruit people who had previous infections. Randomly divide them into two groups, give one group the vaccine and the other group a placebo. In such a trial, you are effectively comparing (seropositive + vaccine) and (seropositive + placebo). This is precisely the first row of the table shown above.

The only difference is that the sample size of seropositive people was too small during the original vaccine trial. There were simply not enough people with Covid-19 when the trial was initiated.

***

At this juncture of the pandemic, experts have acknowledged that these Covid-19 vaccines do not stop infection or transmission - they say the purpose of these vaccines is preventing hospitalization and deaths.

In the vaccine trials, basically no one with prior infection subsequently was hospitalized for or died from Covid-19. In the J&J trial, only 15 participants out of over 42,000 were both seropositive at baseline, and subsequently counted as cases. Therefore, we have no data on how the vaccine affected the chance of hospitalization or death among the seropositive subgroup (but we know it's not a big number).

The 98% VE number mentioned in the opening quote of this blog is a metric about preventing infection, not severity of disease. Thus, we have been served incongruent messages that (a) previously uninfected people should not worry about getting infected (and only worry about getting severe disease) while (b) previously infected people should worry about getting infected, even though their case rate is 10 times lower than the previously uninfected.

 

Why you must know how analytical results were obtained

A friend sent me to this Vice article, admitting abashedly that she got clickbaited (link).

[right here is supposed to be an image showing the article's headline - but it's not here because Typepad's image loader is failing yet again!]

How did she get baited?

Of course, the MTA did not "deliberately slow down buses to punish riders". What the public bus operator did was to suspend so-called all-door boarding. All-door boarding was being tested on a small number of "Select" bus routes: on these routes, all passengers are required to pay using machines installed on the sidewalk before boarding, and thus, they no longer line up at the front of the bus, and can board through any of the buses' doors. As expected, Select bus rides experienced lower delays.

When this experiment was suspended, the Vice reporter "interpreted" this as MTA deliberately slowing down buses.

Why was the experiment stopped? Certainly not because the operator thought NYC buses were running too fast. Clue: what might happen when passengers are allowed to enter buses through the back doors without having to show they paid the fare?

You got it. The MTA is forced to suspend the all-door boarding test because of fare evasion.

***

The Vice reporter dismissed that possibility by citing an earlier MTA study which concluded that all-door boarding reduced fare evasion. It's as counterintuitive a finding as one could imagine. In fact, that study found that fare evasion was 10 times lower on Select bus routes than on regular bus routes!

[right here is supposed to be an image showing a data graphic from the study - but it's not here because Typepad's image loader is failing yet again!]

This is the kind of analytical finding that smells like your fridge after you return from 15 days' vacation.

The last column suggested that fare evasion was almost 20% on all non-Select bus routes. The table below showed fare evasion went down to 2% on SBS bus routes. In the famous words of your favorite Covid-19 talking head, SBS is the miracle vaccine that has 90% effectiveness in curing fare evasion!

Want to get rid of fare evasion? Install more machines on the sidewalk, and let everyone board without showing proof of purchase!

What does one do when the stats stank like rotting food?

How about turning to the "methodology" section?

There, I learn that two completely different methods were used to collect data on "Select" buses and regular (front-door boarding) buses.

The method used for regular buses appeared rigorous and intentionally designed:

A randomized quarterly sample of observation surveys is generated; peak and non-peak, high and low volume stations, all boroughs represented

The sample consists of approximately 180 station control areas and 140 bus routes per quarter

Staff visit several assigned subway stations / bus routes each day to observe and record evasion of different types, e.g. illegal turnstile / service gate entry, entering bus through back door, etc.

Here is how the study authors described the sampling method for "Select" buses:

Eagle teams conduct periodic “surge” exercises, where in addition to enforcement activity they count paid vs. unpaid passengers on board the bus

So, what are some of the differences?

The regular bus routes have daily data collection while the Select bus routes are sampled "periodically" (with an undisclosed frequency and cadence).

The regular bus routes are represented by a random sample from all routes. We have no idea if every Select bus route is sampled (it's possible since there are not that many). However, since these "eagle teams" had other "enforcement activities", one wonders if the frequency of sampling depended on the frequency of "enforcement activities".

Having to both conduct "enforcement activities" and data collection suggests that the data collectors for regular bus routes are properly trained and specialized in data collection while those for Select bus routes may have less training and experience, and are responsible for multiple tasks.

Let's see: enforcement taking place alongside data collection. What about having police officers standing on every street corner, and then measuring the prevalence of crime in those areas at those times? Well - they knew about this bias because they listed it under "limitations":

The mere presence of field staff may limit evasion.

This is what I call a "professional foul". This type of language appears on almost every journal article - important biases that likely invalidate results are simply ignored after the authors acknowledge that they exist. Most analysts keep quiet since everyone else follows the same practice.

***

I'm also struggling to imagine how these "Eagle teams" collected the fare evasion data on Select bus routes. I can see two possibilities:

(a) They position themselves at the bus stops, and observe passengers as they board the bus. In this case, they can see who paid at the roadside machines and who didn't. (In recent years, though, the MTA also installed scanners that allow people to board through back doors using Applepay or chipped credit cards so that significantly complicates the counting. The study was conducted in 2018 which may have preceded this new payment method.) I'm not sure what they mean by "enforcement actvity" - could it mean enforcing fare payment? If so, this is truly Orwellian.

(b) They position themselves on the buses, presumably performing their "enforcement activities". At each stop, a crowd of people enters from all doors. None of them is required to show tickets, in the name of speeding up boarding. How do they figure out what proportion paid? Do they do a show of hands? Do they ask every individual rider to show proof of purchase? If so, do they hold up the buses and not allow anyone to get off until they have finished tallying?

The language used in the Methodology section suggested (b) since they "counted" "passengers on board the bus".

Readers - maybe I'm missing the obvious. How do you think they collected the data?

***

Even if the same sampling methodology was applied, and the data were properly collected, the next challenge is that the Select bus routes serve a small portion of the area served by all bus routes, and the demographics cannot be assumed as identical.

 

 

 

The difference between prediction and estimation

I saw an interesting talk by Brad Efron on the differences between the tasks of prediction, estimation and attribution (link). He made some provocative points. The notes below include my interpretations of what he said. (Efron is most famous for inventing the bootstrap.)

To nail down the setting, let me describe the example he used. The data came from a clinical study in which there are 100 subjects, half of whom developed cancer. For each subject, genetic data were available, which contain > 6,000 dimensions. So, this is a case in which p > n, i.e. the number of variables is larger than the number of subjects.

A classicial statistician approaches this problem as "estimation" of global (average) risk factors - the goal is to find a small set of genes that have high explanatory power: these may be single genes (main effects) or clusters of genes (interaction effects) that are strongly associated with subjects who developed cancers. The goal of such models isn't to predict the fate of individuals.

A machine-learning engineer tackles this as a prediction problem - the goal is to predict whether or not a subject will develop cancer. These complex prediction models do not lend themselves to simple interpretation, and so have little value as scientific explanations. The initial progress in ML arose by explicitly discarding the scientific imperative to focus solely on predictions: who cares if the average is biased if the individual predictions are better?

Both groups frequently deploy similar analytical machinery. For example, they may run logistic regressions. Thus, many practitioners see estimation and prediction models as substitutes, basically different ways to solve one underlying problem. The estimation model is applied to individuals to generate what look like predictions while the ML modeling software computes "variable/factor importance" which are interpreted as global "risk factors".

The value of Efron's presentation is to clarify that prediction and estimation are separate pursuits. At the end, he held out hope that a unifying theory may emerge but for now, it doesn't exist.

(I) Prediction errors from prediction models are often smaller than those from estimation models

Prediction models have proven their value in predicting outcomes. For Efron's example problem, a random forest model makes 1 error out of 50 in the test set while a GBM (gradient boosting) model makes 2 errors. Estimation models applied to make individual predictions do quite a bit worse.

(II) Prediction models have dubious value in estimation

Prediction models ignore bias but that's not what Efron focused on. Prediction models are "explained" by looking at variables/factor importance values. Something like this:

It turns out that these charts are misleading.

Since Efron had 6,000+ variables, he could afford to throw out a bunch of them. When he removed the top 10 variables, and re-built the random forests (or boosting) models with the remaining, the new model has similarly strong predictive power - except, of course, the most important variables has changed. The same thing happened when he removed the top 50, top 100, etc.

So, in fact, the set of "important" variables is not important in the general sense but important only to the particular model.

This finding implies that such tabulation of variable importance is useless. They don't contribute to "explaining" or "interpreting" models.

(III) Prediction is "easier" than estimation

Efron noted that estimation models attempt to fit the average value while prediction models fit individual values. Specifically, accuracy of prediction models is computed on a "test" dataset.

A test dataset can be thought of as a sample from the underlying population. In the cancer example, the boosting or random forests models are evaluated on 50 patients randomly selected from the set of 100. Repeated random selection results in different test sets of 50, which results in different values of predictive accuracy. Some of the prediction errors come from variability of the test datasets. This portion of the prediction errors does not depend on the predictive model itself. As a result, many predictive models end up with similar error rates.

The estimation error does not depend on test data variability since estimation models are judged against the average. With estimation models, there may be a few models that perform markedly better than others, and the modeler has to find those. This is one reason Efron said prediction is "easier". 

Efron hopes that people making estimation models can figure out how to make more accurate predictions, and that people making prediction models can figure out how to draw scientific knowledge from them.