Big Data, Plainly Spoken (aka Numbers Rule Your World)

It's random because we say so

Some of you are wondering why I haven't written about Covid-19 studies recently. Lamentably, the quality of such studies has not improved. It may have deteriorated even more. I wrestle with whether it's a waste of time to read them. The problems with many studies are apparent from the first page: how do they pass the peer review process?

This post concerns a half page of a recent jourmal publication called "Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data" (link). It's only half of one page because there are already enough problems with what I read that I don't see the point of continuing.

A friend sent me this paper because the authors share my concern about significant misrepresentation of vaccine effectiveness in observational studies of Covid-19 vaccines. (Here is a post about my paper published last month with two co-authors that highlighted three common biases that are not properly handled in studies.)

My comments are limited to the section with the header "Sequential Target Trial Emulation" on page 2 of the paper. In this section, the authors describe the first of two proposed analytical devices for dealing with biases in observational data.

For any observational study, there must be a well-defined starting and ending date of study enrollment. For each day in this enrollment window (and for each region), they pretend that a new target trial is being launched. This is how they put it:

on each day and in each region, eligible persons who have not yet been vaccinated are randomly assigned to receive immediate vaccination or to remain unvaccinated throughout follow-up and are then followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first

They then suggest that each daily trial can be analyzed on its own, and then the results averaged to obtain an overall effectiveness. They claim this outcome is representative of the "intention to treat" effect in a hypothetical randomized clinical trial. That's a bold claim! (Not even Pfizer, Moderna, etc. claimed to have produced ITT vaccine efficacies even in their RCTs.)

I'm not making this up. They literally claimed that "The trial-specific estimates of the effect of vaccination from each sequential trial can then be combined to estimate an observational analogue of the intention-to-treat effect of assignment to the intervention."

***

Why am I shaking my head violently?

Their statement that people are "randomly assigned to receive immediate vaccination or to remain unvaccinated" is unserious.

We are talking about observational data - data analysts poking around in a database after the fact. It doesn't matter what analytical technique is used - it remains the case that the vaccinated people self-selected the treatment. They were not randomly assigned to receive immediate vaccination.

In fact, the authors contradicted themselves in the very next sentence when they explained (my bolding):

Each calendar day is considered as time zero for a new emulated trial, with persons who are determined to be eligible assigned to the vaccination group if they were vaccinated on that day or to the no-vaccination group if they were not vaccinated,

For the vaccinated, there is no "assignment", let alone random assignment. Whoever decided to get vaccinated lands passively in the treatment group for that day. Because the vaccinated are self-selected, the unvaccinated are self-selected as well. Whatever selection biases exist in the raw data persists in this ex-post experimental setup. This step absolutely does not support presumption of random treatment assignment.

If this procedure resulted in random treatment assignment, then the original dataset contains no biases, which annuls the central premise of the entire paper.

***

Of course, they aren't taking every unvaccinated person. So the data analyst actively selects who comprise the unvaccinated group. The authors give two mechanisms for selecting unvaccinated.

The first is random selection. I had to take a deep breath before reading on. I doubled back to see if I read this correctly. "One or more persons can be selected at random from all eligible persons who were unvaccinated on that day and in that region, with exclusion of unvaccinated persons selected for a comparison made on a previous day."

If they randomly selected unvaccinated, then the random sample of unvaccinated would have the same statistical composition as the population of unvaccinated. This ensures no correction for any kind of biases. And yet at the end of this process, they claim to have found the ITT effect as if a randomized clinical trial were run!

The second mechanism for selecting unvaccinated is by matching on key "characteristics that reflect Covid-19 risk". This method is definitely more acceptable as it at least attempts to make the two treatment groups more comparable.

In practice, the matching method faces several big challenges (many of which I have discussed before on this blog):

• the matchable population is a small proportion of the total population, e.g. most studies using matching heavily undersamples older people and oversamples younger people. The method is guaranteed to undersample demographic segments in which vaccination rate is high, and oversample demographic segments in which vaccination rate is low
  
  
• VE in the matched population may be different from VE in the general population. The estimated VE in the matched population is surely not the ITT for the entire population
  
  
• most important confounders are unmeasured (e.g. taking other mitigation measures, vaccine hesitancy, exposure to virus). Matching does not infer balance on non-matching variables, a key reason why ex-post matching on observational data cannot replace randomized treatment assignment
  
  
• most countries enforce highly targeted vaccination programs which generate irreducible biases, e.g. if older people are encouraged to get vaccinated first, and have done so in large numbers (say 80%), what's the chance that the 20% unvaccinated older people are comparable to the other 80% vaccinated older people?

***

There is another aspect of the observational studies that are buried away from the summary of the methodology. In the sentences cited above, they said that the study subjects were "followed until COVID-19 diagnosis, death, or the end of the study period, whichever occurs first." That surely cannot be true!

Another event that happens is vaccination. For people in the unvaccinated group, by definition, they have not been vaccinated up to the day of enrollment; however, many or most of them would get the vaccine during the follow-up period. How did the data analyst deal with this event? We are not given a hint in this methodology section.

Upon vaccination, clearly the follow-up period for the unvaccinated person must end. We simply don't know the counterfactual: what would have happened if the person did not get vaccinated? If we follow the time series of vaccine efficacy from day of vaccination forward, more and more people in the unvaccinated group are dropped from follow-up because they got themselves vaccinated. The people who stay in the unvaccinated group for a long time are the vaccine holdouts, who are clearly different from the general population, e.g. they are much less likely to take other precautions such as wearing masks, distancing, avoiding crowds, etc. How well does matching on demographics and basic health data control for these differences?

If those differences are not adequately accounted for, then the right side of the VE curve cannot be trusted.

***

Also, note that variability is artificially suppressed in this type of study. Anyone who got vaccinated during the study period appear once in the vaccinated group on the day of vaccination, but also may appear in the unvaccinated group on any day prior to vaccination. Unlike a typical RCT in which there are no overlaps in the group membership, in this type of observational studies, the majority of the people show up in both groups. So, if we subsequently run regression analysis comparing the two groups, adjusting for demographics and other factors, the variance between groups is markedly under-stated!

 

 

Residual problems of intelligence

Modern data science/machine learning/AI has achieved much through efficiently performing certain tasks to an acceptable level of quality. It remains to be seen whether there are means to tackle the residual problems, or perhaps, whether there is the will to do so.

I ran into a couple of problems while using Yelp.

First, I sorted the restaurant reviews by date, and the first two reviews that popped up were these:

These two reviews are almost identical. If we take them at face value, these are two people, one from Midtown West and the other from Staten Island, who both walked in for lunch two days apart (assuming they wrote the review on the same day of their respective meals), and managed to grab one of the last tables. They both ordered steak frites and loved it, especially the fries. And, they both decided to come back for breakfast in the future. They also learned English composition from the same teacher. What a coincidence!

Yelp has long had implemented AI that supposedly detects fake reviews but apparently something that is so obvious is missed by the algorithm.

There's a lot to unpack here but one key immutable fact is that all review sites attract fake reviews because good reviews are great marketing for businesses. Review sites that stringently authenticate reviewers will have much lower volumes of reviews, which creates a negative network effect. Authentication does not solve the root cause anyway, since businesses hire real people to write fake reviews.

***

The next situation I ran into isn't really about Yelp but about Google Maps. Within Yelp, I clicked on the address to see the location of a restaurant. I read in one of the reviews that this restaurant is inside a hotel. In order to confirm that, I entered the hotel name and asked for directions inside Google Maps.

Here is the recommended route:

It literally says walk out of the building (the hotel), make a circle around the block, then re-enter the building (the restaurant). It doesn't realize that the beginning and ending addresses are identical.

***

It's surprising that these issues persist so many years after these services were launched. Make no mistake - most users have most of their problems solved using Yelp and Google Maps. What's the incentive to solve the residual problems? That's what's on my mind today.

Game of telephone for data messaging

We all know about the game of telephone. If one person says something to another, and the second person to a third, and so on, the message received at the end of the chain will be substantively different. What gets me thinking about this today is an article by Gelman and friends, which essentially describes the game of telephone as it applies to data messaging: "Through endless repetition, numbers of dubious origin take on the veneer of scientific fact, in many cases in the context of vital public-policy debates."

I love their leading example. Apparently, (at least one) university in the U.S. circulated a public-service notice telling people to wear a hat on cold days because "most body heat is lost through the top of the head." The authors attempted to trace the origin of this message, and discovered that the advice purportedly came from a statistic, which claims that 50 or even 80 percent of body heat is lost through the (apparently cap-less) head.

[Most of the article is about tracing statistics on "corruption", and about the concern that politicians mislead the public using junk statistics.]

***

This little scenario reveals a number of things about our relationship with data.

First, repeating data seems to follow what Kahneman calls "fast thinking", not "slow thinking". People tend not to slow down, and apply logic, and ask: could we really be losing 80 percent of body heat through our head? Could a human being survive after losing 80 percent of its body heat? (Apparently, the more correct answer is 10 percent.)

Second, because of the fad built upon "unconventional wisdom," spread by the likes of Malcolm Gladwell and Steven Levitt, people may believe illogical data -- the more bizarre, the better. Shock value is the currency of social media, so this fad lives.

Third, I'm interested not just in the likelihood that the message has changed through repetition, but in what direction the message shifts. My working assumption is that the default setting of most people is story-first (as opposed to data-first). The way data are consumed is typically to support some hypothesis, i.e. preconceived notion. In the game of telephone, the message is continuously shaped by the story-first mentality of the messenger.

This idea is an expanded version of what the authors called "decorative statistics", i.e. making numbers "sound big and impressive".

Each person receives a message, but only partially due to memory failure. Each person has its own belief, such as about the effect of not wearing hats on cold days. The outgoing message will tend to be pulled toward those prior beliefs.

When I find some time, I should run a simulation using Bayes' Rule to propagate messages at each stage and explore the above idea more rigorously.

***

Another factor enables this game of telephone in data stories. Every statistic ever reported has nuances - it is only true for specific populations, after certain exclusions, accounting for particular biases, etc. However, in messaging data, we don't have the luxury of attaching two paragraphs of explanations for each number. Therefore, the nuances are lost.

The orphaned data are out in the open. This invites anyone to adopt these statistics and shape them to their own liking. Fill in the blanks. If one is completely rigorous, one plugs the gaps by researching the origins of the data, but who has the time, right? So, instead, one just adds whatever one thinks "makes sense"... this is where we ends up in a story-first mode, because making sense is in the context of that preconceived story. If we want people to wear hats, we like the story that hats keep heat contained in the body, and if we can't quite remember the statistic, we are susceptible to guesstimating a number that supports our story.

 

How do storytellers use data?

A friend forwarded an article about minimum wages in Hong Kong published by CNN (link). The article is filled with statistics and data. It's interesting to observe how the reporter chose to present them.

(excerpt from this amazing infographic by SCMP)

The following list is in order of appearance in the article:

• minimum wage went up by 32 cents (in the title of the article)
• one of the world's priciest cities
• the least affordable housing market on the planet
• parking spaces that go for nearly a milion dollars each
• new minimum wage is 40 HK dollars (US$5.1) per hour
• previous minimum wage was 37.5 HK dollars (US$4.78) per hour
• minimum wage was first established in 2011
• minimum wage was frozen to 2021 levels since the pandemic
• tied with Los Angeles as the fourth place on the Worldwide Cost of Living Index in 2022, with only New York, Singapore and Tel Aviv in front of it
• New York's minimum wage is US$15 per hour, Tel Aviv's about US$8.27-8.45, depending on total hours worked, Singapore "offers different minimum wages depending on sector"
• the median hourly wage last year was 77.4 HK dollars ($9.86)
• severe wealth gap, with the richest households making 47 times more than the poorest
• the income of the poorest decile (bottom 10%) fell by more than 20% compared to their incomes before the Covid outbreak
• the proportion of people living on minimum wage is actually lower than a decade ago: only 2.6% of employees earned less than $40 an hour in 2021, compared to 6.4% in 2011 [for some reason, this number is stated in HK dollars and not translated into US dollars, as are all the other values in the article. the $40 an hour is the new minimum wage equivalent to $5.1 US]
• the gap between median and minimum wage has increased
• food costs have risen more than 7% from December 2019 to December 2022
• energy costs have also increased by more than 11% during the same period

***

What should we make of the 32-cent increase in the headline? It is designed to shock American audiences and the context for interpreting that number is not provided until later in the piece, much later.

The first piece of hidden data is the absolute value of the minimum wage, which was US$4.78 prior to the raise. This means that the raise was about 7%. Scroll to the bottom of the list, we find that food inflation was 7% aggregated over a 3-year period, so roughly 2.5% per year. Also for context, for most American corporate workers, it's been a long time since we got salary increases in the 7% range.

Another important piece of data is that the median wage was less than US$10 per hour. Knowing this statistic did not stop the reporter from comparing Hong Kong's minimum wage of $5 to New York's of $15. Half the population in Hong Kong earn less than two-thirds of New York's minimum wage. According to this site, the median hourly wage in New York City is $28. So those numbers aren't directly compared.

Further, below the fold, the reporter mentioned that only 2.6% of people earn less than the minimum wage in Hong Kong. The proportion earning below minimum wage has been cut by 60% in the last decade. In New York City, according to that same site, about 20% earn less than the minimum wage, which is almost 10 times higher than in Hong Kong.

Singapore is perhaps the most comparable place to Hong Kong. Singapore is described by most writers as not have a (universal) minimum wage, which is euphenism for saying the minimum wage is $0. I wonder what is the story behind that.

Of course, it's natural for labor activists to push for bigger increases in minimum wages. I just find it fascinating how the statistics were processed, selected and sequenced for publication. A different selection and sequencing would have yielded a different story line.