Big Data, Plainly Spoken (aka Numbers Rule Your World)

There is no such thing as objective data

One of the persistent delusions about people running numbers is that they produce "objective" data. This delusion is frequently delivered in a self-righteous manner, directed at people who advocate adjusting the raw data. No no no, they say, you can't change the data because you are injecting subjectivity into the sacred raw data!

They claim that any adjustment is like injecting additives to pure, wholesome, raw foods.

In fact, all data are subjective. Even if the data quality is perfect, all data are still subjective because of measurement. This is one of the running themes in my second book  Numbersense (link). The first chapter addresses how school administrators can manipulate formulas to gain an edge in school rankings. The formulas are fixed but an endless number of tactics  delivers data that result in different values for the same underlying condition.

The latest demonstration of this malfeasance is trending news due to an investigation by a Columbia math professor (link). US News has now suspended Columbia from this year's rankings. What's shocking is their one bad apple framing. As documented in Chapter 1 of Numbersense (link), fraudulent data collection pervades all rankings at all levels of education: in any jurisdiction that matters enough to merit investigation, fraud has been revealed.

One of my favorite examples of fraudulent data collection from that chapter also happens to be an example of a "professional foul", i.e. a foul that is obvious to any outsider but deliberately oblivious to all insiders who partake and benefit from the foul.

This amusing example is the pushing of the "Common App", which lowers the barrier of application for any college applicant. Whereas in the past, each college has its own idiosyncratic application, post Common App, it takes about the same time to apply to one school as 15 schools. 

What the emergence of the Common App does is to "raise all boats". All selectivity rates (proportion of applicants admitted) will go up at all schools that participate in the Common App because the denominator (number of applicants) is jacked by the Common App. If schools are increasing their class sizes at all, the rate of increase in enrollment does not catch up to the rate of growth in applicants due to the Common App.

So - is selectivity rate of colleges an objective data point? In my view, it is a subjective metric, just like every other metric, easily manipulated.

***

This post is inspired by a statement I read about Wework - the controversial startup that rents office space without long-term contracts. (Disclosure: I'm a fan of Wework. I think it markets a product that delivers clear value to users without doing harm, unlike many startups that either delivers zero value or hides the harm done behind "convenience" and other positives. Wework's problem has always been figuring out a business model.)

The CEO claims that Wework's occupancy rate in this last quarter matched that of prepandemic levels, defined as late 2019.

Occupancy rate sounds like one of those pure, raw metrics that has an official definition, thus an analyst can report an objective value for it, beyond scrutiny.

Far from it. Let me point out a couple of ways to dress up this number. I'm not accusing anyone of doing anything. The following is merely a demonstration of possibility - as I disclosed above, I hope Wework finds a business model.

Like selectivity rate, occupancy rate is a fraction: the proportion of available office space that has a tenant. Like selectivity rate, let's focus on the denominator, the amount of available offices.

The occupancy rate may be 72% in both Q4 of 2019 and Q1 of 2022, while at the same time the number of tenants is lower in 2022 than 2019. It all depends on the number of offices available. Since the pandemic, Wework has closed locations - incidentally, any business should want to get rid of its lowest performers so this statement isn't about gaming numbers; I'm pointing out how the metric attaining the same value does not contain the same meaning.

Nevertheless, managers can choose to game the metric. If you get rid of the bottom 20% instead of the bottom 10%, you'd have pushed the occupancy rate even higher.

Now, let's take a tour of one of these buildings. You'll notice that some of the offices are being used for storage. There are chairs, desks, etc. stacked up in them. If I am the accountant, I'd remove these offices from "available" space.

There are also offices that have been converted to common areas for users who are just there for the day. An accountant has at least two options: a) remove these offices from available space, or b) count them as available space, and count them as "occupied". To legitimize option b), the accountant can also stipulate that such an office is counted as "occupied" so long as there is at least one user working in that space for x% of the days of the month.

***

If you came here because you're read Numbersense (link), those last two paragraphs might remind you of Chapter 6, in which I discuss the official unemployment rate. That chapter also portrays data's subjective nature. Unemployment levels depend on how the government defines employed and unemployed. In the book, you learn that someone may have only worked for one hour in a month, and the US government counts him/her as "employed". In addition, the government removes people from the employment count, classifying them as "not in labor force", just like the accountant who subtracts offices used as storage space as "not available".

tldr; If you don't know how something is measured, you don't know anything about the data.

 

 

 

 

Reclassifying data to manufacture statistics

In a previous post, I reviewed an analysis by Columbia professor Michael Thaddeus of Columbia's data that underlie its fast rise up the US News undergraduate ranking table. Even seemingly simple metrics such as class size are subject to subjective manipulation.

Thaddeus discovered the existence of two datasets, one for Columbia College and Columbia Engineering, and one for the School of General Studies (SGS). It turns out that SGS houses a lot of transfer students. Overall, 30% of Columbia undergrads are transfers, the highest proportion among Tier 1 private universities; in SGS, >75% of the students have some transfer credit.

This little detail has huge implications on Columbia's ranking. The US News ranking assigns about 23% of the weight to "outcomes", one of which is the graduation rate within six years. Notably, the graduation rate formula excludes transfer students.

From data Columbia submitted to the government, we learn that transfer students at Columbia (mostly at SGS) have a graduation rate of 85% compared to 96% at Columbia College and Columbia Engineering. The blended graduation rate is thus 92%. That apparently little difference of 4% would have shifted Columbia's graduation rate ranking from 6th to 26th.

Reclassifying data to game statistics is a professional trick. I covered an analogous situation in Chapter 1 of Numbersense (link). Claremont McKenna College was caught inflating SAT scores when reporting data to US News. The President of the College said it was just small potatoes as inflating the average score by 10-20 points on a scale of 800 was nothing (see Figure 1-8 if you have the book).

The scale of 800 is meaningless when the average SAT scores of CMC applicants fluctuate within a tight range (dozens of points) from year to year. In statistical terminology, we should look at a standardized scale that takes into account variability.

Similarly, the graduation rates at top-tier universities tightly bunch up in a narrow range. That's why a decrease of 4% drops Columbia by 20 spots.

This type of gaming has real effects on students. Thaddeus explains:

The picture coming into focus is that of a two-tier university, which educates, side by side in the same classrooms, two large and quite distinct groups of undergraduates: non-transfer students and transfer students. The former students lead privileged lives: they are very selectively chosen, boast top-notch test scores, tend to hail from the wealthier ranks of society, receive ample financial aid, and turn out very successfully as measured by graduation rates. The latter students are significantly worse off: they are less selectively chosen, typically have lower test scores (one surmises, although acceptance rates and average test scores for the Combined Plan and General Studies are well-kept secrets), tend to come from less prosperous backgrounds (as their higher rate of Pell grants shows), receive much stingier financial aid, and have considerably more difficulty graduating.

No one would design a university this way, but it has been the status quo at Columbia for years. The situation is tolerated only because it is not widely understood.

***

There is a final wrinkle to this issue of transfer students. They are not created equal. What makes Columbia special is that the transfer students tend to have lower achievement and higher financial need. This contrasts with transfer students at the UC system, for example, where they have higher graduation rates than non-transfers.

Again, Thaddeus provides useful background:

Some leading public universities, including several in the University of California system, have a higher proportion of transfer students than Columbia. Their transfer students largely come from two-year community colleges in the same state, providing a route to a bachelor’s degree for students at these colleges.

I have met this type of transfer students, and they are good students who work very hard to compete for a second chance to enter a school of their choice. So I'm not surprised that their outcomes are comparable to, or even better than, non-transfer students.

From an analytical perspective, this exercise demonstrates that one sometimes has to dig several layers deep in the data. Columbia has more transfer students, unlike its peers; plus, Columbia's transfer students have worse outcomes than its non-transfer students, unlike other schools.

 

 

 

Manufacturing statistics, US News style

US News school rankings are rigged. That much you should know after reading Chapter 1 of Numbersense (link).

Is it the school administrators' fault that they one-up each other gaming the ranking to the nth degree? Or is it the people's fault for selecting one school over another because it's ranked higher on US News? Should US News have fact-checked submitted data? Can we blame US News for providing a product that apparently is deemed highly valuable?

This investigative report by Columbia professor Michael Thaddeus (tip from Andrew Gelman's blog) serves two purposes:

a) it exposes the tricks of the trade that have pushed Columbia up the ranking table from #10-ish to #2 over the last two decades

b) it confirms that prior expose of bad behavior have not deterred gamers (and Columbia is certainly not the only school doing this)

***

Prof. Thaddeus started out by repeating US News's selling point that 80% of the ranking is "based entirely on numerical data collected by the institution" - data which are presumed to be "objective." (N.B. self-reported, unaudited)

He proceeded to follow the numbers trail, and in each case, discovered that such supposedly "objective" "numerical data" are not objective at all - which is a key lesson of Chapter 1 of Numbersense (link). In the professor's words, "several of the key figures supporting Columbia’s high ranking are inaccurate, dubious, or highly misleading." Most data are subjective, period.

Here as in Chapter 1, it is reasonable for a casual consumer of data insights to wonder how data such as "undergraduate class size", "percent of faculty who are full-time", and "student-faculty ratio" can be subjectively interpreted.

Aren't there clear, simple formulas for each item?

Undergraduate class size is the total number of class enrollments by all undergraduates divided by the total number of classes.

Percent of faculty who are full-time is the number of full-time faculty divided by the total number of faculty.

Student-faculty ratio is the number of enrolled students divided by the total number of faculty.

Is there any room for interpretation for such simple statistics?

***

Let Prof. Thaddeus tell you. It's more than your eyes can see!

According to US News, Columbia said 83% of its undergraduate classes have <20 students while 9% have over 50 students, so about 8% have between 20-50 students.

Prof. Thaddeus pointed out that the 83% number is well above most colleges in the U.S. The median value for all National Universities was 48%. The second and third highest numbers behind Columbia in the top 100 were both at 78%.

This statistic seems to contradict hearsay as I have never heard someone describe Columbia as a place with small class sizes, in fact, the opposite (admittedly, most of my sources are in graduate schools).

Prof. Thaddeus learned that US News applies the same set of criteria to determine class size as used in the "Common Data Set", which publishes standardized data on U.S. colleges. (Columbia is not part of this group, and is one of the eight schools in the Top 100 that do not belong.)

This is where you learn that "undergraduate class size" is not what you think it is. It's not what the simple formula above computes.

First, an undergraduate class is "a class that enrolled at least one undergraduate". This means that many graduate school classes are counted as undergraduate just because one out of 20 students is an undergraduate. Further, if a student is doing a joint bachelor's/master's program, then that student by him/herself will cause all his/her graduate-level classes to be classified as undergraduate!

Second, a long list of class types is excluded no matter how many undergraduates enroll in them. An internship class that grants course credit, for example, is not considered an undergraduate class.

Back to Columbia. Thaddeus now made a major discovery - that Columbia prepares two separate datasets (non public): one for "Columbia College and Columbia Engineering" and one for "School of General Studies". You may be reminded of 'In labor force' and "Not in Labor Force" when governments compute unemployment rates (Chapter 6 of Numbersense (link)). As I read this part, I'm anticipating that as this story unfolds, the students in the SGS will be forgetten and abandoned.

***

Leaving Columbia aside for a second. The class size metric is misleading even if schools provide honest data. The problem is the units. When US News say 83% of Columbia's classes have fewer than 20 students, the base unit is a class. At most schools, class size has a skewed distribution: a small number of courses (such as ECON 101) have enormous enrollments while a long tail of courses have relatively few students.

Let's say a school with 100 students offers 20 undergraduate classes. Each student takes 4 classes, resulting in 400 total enrollments. Two of these classes are compulsory so the enrollment is 100 each. The other 200 enrollments are split between 18 classes, so each of the smaller classes enrolls 11 students. Each student therefore has the same schedule, 2 large classes (n=100) and 2 small classes (n=11). Therefore, the average class size experienced for each student is (100+11)/2 = 56.

What if we apply the industry formula for computing class size? It's based on classes not enrollments. So 18 of the classes have size 11 while 2 of them have size 100, thus the average class size is 20.

Ninety percent of the classes have enrollment <= 20 students and yet each student experienced an average class size of 56. Ouch!

[P.S. If enrollments rather than classes were counted, then 50% of enrollments are in classes with size 100 while 50% were in classes below 15. This presents a more accurate description of the reality.]

***

Undeterred by the lack of disclosure, Prof. Thaddeus does the prototypical data science thing - scrape the web, in particular, Columbia's Directory of Classes, for information on every class and its maximum enrollment size.

While analyzing scraped data provides a good approximation, this method (like most web scraping exercises) cannot claim to be accurate. One problem is that capacity is not the same as actual enrollment. In fact, some classes may be offered in the catalog and later cancelled. But this information is sufficient to establish an upper bound for the class size, as well as a reasonable guess at the true average. Using this, the professor concluded roughly 63 to 67% of classes are <20 students, "nowhere near the figure of 82.5% claimed by Columbia."

For the other metrics, I'll refer you to Prof. Thaddeus report.

 

 

Changing metrics when you dislike the results

(Source: xkcd)

 

Chapter 2 of Numbersense (link) is titled "Can a new statistic make us less fat?" In the context of recent remarks by the acting New York State Governor, perhaps I should have called it "Can a new statistic save some people from dying?"

Trick question:

An executive tracks performance using a key metric displayed on a dashboard. The executive requests a review of the definition of the metric, then decides the definition must be revised, and after the revision, the new metric looks better than before. Which of the following is true?

a) Using the prior definition, the key metric was outperforming expectation

b) Using the prior definition, the key metric was underperforming expectation

***

The New York TImes reported that "Ms. Hochul said that some hospital executives have told her between 20 and 50 percent of their Covid patients are not suffering from severe symptoms, but that they are testing positive in the hospital incidentally, after being admitted for other reasons such as car accidents...As a result, beginning Tuesday, the state will begin to ask hospitals to break down how many patients are being admitted for acute Covid-19 symptoms, in an effort to further decipher this wave’s severity."

This is statistical alchemy that is frequently requested by people who don't like what the numbers are showing. It is a trick because they did not commit to restating the entire data series back to the start of data collection. This aforementioned issue is a "problem" only if they had recently changed the counting rules to include admissions for non Covid-19 reasons, therefore causing recent data to be inflated relative to older data, but that's not what they're saying.

Further, notice that they are commingling two separate re-definitions: one is Covid-19 vs not Covid-19; the other is mild vs "acute" Covid-19. I have never heard that "cases" only refer to "acute" Covid-19 cases, until now. The new definition relies on a subjective definition of "acute" and so they can report whatever number they want going forward.

To have any credibility, they need to (a) publish all the definitions that have been deployed in the last 2+ years, including the latest one, and (b) apply the latest definition to restate all historical data. Otherwise, no one can tell whether the trend is up or down since we are not comparing apples to apples!

To have maximum credibility, they should do (c) a comprehensive review of the definition, and correct for problems that undercount as well as overcount. For example, are they missing lots of cases found by people doing at-home testing? How are they dealing with people with multiple test results? (This is no small matter: if an infected person will take further tests till s/he tests negative, then each positive test is likely to be balanced by one or more negative tests in the near future. This dynamic affects how we can interpret the time series of positive tests divided by number of tests.)

***

In Numbersense (link), I trace how every few years, someone writes an article saying we need a new definition of obesity. The typical reason is that BMI classifies fit athletes as obese. I have never seen someone provide an analysis of how BMI misclassifies the average person. It's always the outliers. Notice in the above quote, the Governor did not give an aggregate proportion of what she considers misclassified. She said "some" hospitals gave her a concerning statistic.

As a data analyst, you have to think about the likelihood that the sample of hospitals she heard from is representative of all hospitals. Imagine you're working at a hospital for which most of the patients testing positive for Covid-19 are being treated for Covid-19. How likely are you to pick up the phone and call the Governor to tell her that you do not have a problem?

Thus, the sample she is working with is biased towards those hospitals experiencing this "problem". The real proportion of cases being "misclassified" is surely lower than what she announced.

Sharper understanding of vaccine effectiveness

Recently, some researchers changed the way they describe vaccine effectiveness. It used to be the Pfizer vaccine is 90% effective at preventing (first it was cases, then it was severe cases, finally it was) deaths. Now, they are saying the vaccine cuts (cases, severe cases, deaths) by 11 times. (See, for example, this recent report on the booster shots in Israel.)

Cutting some bad metric down by 11 times sounds extremely impressive. Is 11 times better than 90%?

The answer: they are saying exactly the same thing. If the rate for unvaccinated is 11x, and that for vaccinated is x, then the vaccine effectiveness is 1 - (x/11x) = 91%.

Cutting a number by 11 times is great but hard to achieve. Now, if we have a vaccine that cuts infection rate in half, that is still quite good, right?

That vaccine would have a VE of 50%. CDC doesn't even want to talk about such a vaccine.

***

I will say it again. A 90% vaccine effectiveness does not mean that 9 out of 10 people who got vaccinated are protected. As explained here, VE is a ratio of two infection rates. Both of these infection rates are very small, typically below 10%, and sometimes below 1%, depending on which clinical trial the data came from. In other words, over 90% of the people in the placebo arm did not get sick during the study period. It's logically impossible for 90 out of 100 people to be protected when fewer than 10 unvaccinated people would have gotten sick.

Statements such as "99% of vaccinated people are not hospitalized with Covid-19" are not very informative, unless we are also told what proportion of unvaccinated people are not hospitalized. If we have both, and flip the focus from not hospitalized to hospitalized, then we are looking at a ratio of rates, which is the vaccine effectiveness measure.

Those alternative descriptions are distracting and harmful when misinterpreted.

***

One last thing. A 90% effectiveness against cases and a 90% effectiveness against deaths are not at all similar things.

That's because the rate of death is a fraction of the rate of cases. When talking about cases, a 90% VE may mean pushing case rates from 10% to 1%. When talking about deaths, a 90% VE may mean pushing death rates from 0.5% to 0.05%. It is ludicrous to say that 99.95% of vaccinated people are protected from deaths when 99.5% of unvaccinated people would not have died.

A big drop in death rates is impressive on its own. Less effective is not the same as not effective. I don't like these alternative descriptions of effectiveness because the math is incorrect.

If you're not counted, you don't count

If you're not counted, you don't count.

The first law of counting is to count everything, and I do mean everything - not just the things you think you care about.

***

Imagine - as I did in Chapter 1 of Numbersense (link) - that a college advertises that 95% of its graduates found jobs within 6 months of graduation. It turns out the school collects data from a post-graduation survey - it's not disclosed anywhere in the marketing materials that it's 95% of survey respondents, not 95% of graduates. What's also not reported is the survey's response rate, which was 20%.

Nevertheless, the college administrators stand behind their statistics. They say the respondents and non-respondents did not show statistically significant differences in their gender distribution, average age, and type of degree; therefore, they assume that non-respondents also have a 95% employment rate within 6 months.

Do we think graduates who failed to land jobs have the same chance of filling out an employment survey as those who are gainfully employed after graduation? You can draw your own conclusion.

If you only have biased data on 20% of the graduates of this school, do you think you have a full understanding of the employment prospects of its graduates?

***

At the start of the Covid-19 pandemic - think March or April 2020, the U.S. suffered from a shortage of diagnostic tests. The official policy at the time was to test only people with severe symptoms.

We knew next to nothing about the novel coronavirus at the time. The data on cases became the basis of our knowledge - not only of the severity of the pandemic but also its trends, its demographics, its symptoms and injuries, and so on.

But triage testing leads to biased data. When the pandemic also grows quietly through asymptomatic transmission, our data led to a partially blind, misleading assessment of the situation. I warned about this in a Wired column - subsequently, colleges have shown that comprehensive testing is crucial to controlling the spread of Covid-19. Random testing of small samples of the population is a viable alternative we can still implement today to monitor new variants.

***

A curious thing happened during the pandemic to our unemployment statistics. Eight million Americans not only lost their jobs overnight but suddenly lost all interest in working. That's according to official statistics, which showed that the number of people in the "civilian labor force" dropped by 8 million.

People "not in the labor force" have always been a strange species. The government claims that these people either do not want work at all, or have not been looking for a job for an extended period of time. Their unemployed status is ignored in the unemployment rate. Oddly, as more people drop out of the labor force, the unemployment rate improves, all else being equal. This group is simply not counted; they are neither employed nor unemployed.

If one has the responsibility of reducing the unemployment rate, one will focus on unemployed people who are counted in the data, not the ones who are not counted. When you are not counted, you don't count.

In Chapter 6 of Numbersense (link), I explain how the government computes the unemployment rate in detail. Not counting a group of unemployed people hides some of the bad news. As I pointed out in this previous blog post, the sharp exodus out of the labor force at the start of the pandemic proves the serious shortcoming of the concept. None of the people who just lost their jobs due to Covid-19 are people who do not want work at all; their recently-ended employment proves otherwise. There also was not enough time that passed to qualify them as having given up on a job search.

***

The Bureau of Labor Statistics (BLS) does count everyone, including data on how many people have been removed from the labor force tally. These data just show up in an appendix, with much less attention and care.

We can't say as much about how the CDC counts Covid-19 cases in 2021. As this New York Times story disclosed, the CDC "stopped investigating breakthrough infections among fully vaccinated people unless they become so sick that they are hospitalized or die."

Just as BLS defines unemployed people as not really unemployed if they "don't want to work", CDC defines infected people as not really infected if they are fully vaccinated and are not hospitalized for severe Covid-19 symptoms. In this way, our data have come full circle, back to the same sad state that it was in March or April 2020. It tells us only about severe cases. To the extent that new variants spread through asymptomatic infections, we are as blind to that possibility as we were a year ago.

Furthermore, the change in counting rules upsets any trend analysis. Consider a real-world study of vaccine effectiveness. If the researchers report that the case rate of fully vaccinated people is much lower than the case rate of unvaccinated people, is it because of the vaccine or because we count mild or asymptomatic cases for unvaccinated people but not for vaccinated people?

It's not just fun and games with counting. Don't forget: infected, hospitalized, or dead people who got sick before 14 days are also treated as not really infected, hospitalized or dead. When you're not counted, you don't count. The CDC turns a blind eye to them because they happen to catch the disease prior to reaching so-called full vaccination status. Why do their lives not matter? That's something I have been wondering about throughout the pandemic, and I still don't have a good answer.

Note on a Simpson's paradox in real-world vaccine effectiveness studies

You have heard it in the media: real-world studies of high-risk people such as nursing home residents or healthcare workers offer definitive proof that the vaccines' true efficacy is <insert a percentage above 90%>!!

These were the people who got the vaccines first, they are at greatest exposure to the virus, more of them are fully vaccinated (two shots), they have the highest risk of death, they are easier to track down. Ergo, the result is rock-solid, beyond criticism!

I wish someone had the foresight, and did a quick randomized trial while delivering vaccines to these groups, then we'll have a good dataset to draw some useful conclusions.

Alas, no one did (as far as we know) - which means all these studies are real-world observational studies. I discussed two examples of these studies (Mayo Clinic, Israel's Clalit health fund), both analyzing everyone in their customer database using the matching technique. Matching is used to create a synthetic control group (after the fact) in the hope of emulating a randomized control group as in a randomized, controlled trial (RCT). After matching, we apply methods commonly used to analyze RCTs, as if the conditions of RCT have been satisfied.

Today, I'm examining a Danish study of the Pfizer vaccine in two high-risk groups, nursing home residents, and healthcare workers. (link). This is an example of those real-world studies of high-risk groups that have so excited the punditry.

Counting People No Longer Works

Let's start with a simple calculation, and I'll explain why it fails. The study period is from December 27, 2020 to February 18, 2021, just under 2 months starting from the day Denmark initiated vaccinations. Based on the Figure 1 at the back of the paper, I estimated that during this period, there were 1,675 confirmed cases in Denmark among the nursing home population. The researchers said 382 cases were discovered among the vaccinated nursing-home residents, so there were 1,294 cases confirmed for unvaccinated residents. (The actual numbers used in their calculations are slightly lower because of exclusions but this doesn't matter for what I'm about to say.)

During the study period, 37,172 nursing home residents got at least one Pfizer shot so the case rate for vaccinated is 382/37,172 = 1%. For the unvaccinated group, I use the total number of nursing home residents who have not been infected as of December 27, which was 39,050. So the case rate for unvaccinated is 1,294/39,050 = 3.3%. The ratio of those case rates results in a vaccine effectiveness (VE) of 70%.

This calculation sounds vanilla but has a serious problem. Each nursing home resident who is counted as vaccinated is also counted as unvaccinated - because on December 27, 2020, the first day of vaccinations, everyone is unvaccinated. Then, over the course of time, those who have not been infected are eligible to get vaccinated. Upon vaccination, the nursing home resident transitions from being counted as unvaccinated to being counted as vaccinated. By the end of the study period, only about 5% of the residents are still unvaccinated. So 95 percent of this population are counted twice, and in each case, the persons spend the first part of the timeline as an unvaccinated and then the second part of the timeline as vaccinated.

Moreover, the proportion of time spent as unvaccinated varies depending on the day of vaccination. To deal with this issue, we change the denominator of the case rate. Instead of cases per person, we use cases per person-week. Each person is still counted twice, but in any given week, a specific resident is counted just once, either on the vaccinated arm or the unvaccinated arm but not both.

There were roughly 148K person-weeks accumulated on the vaccinated arm, and 164K on the unvaccinated arm, leading to new case rate estimates of 0.26% and 0.79% (interpreted as per 100 person-weeks). This leads to VE of 67%.

(In the second half of this post, I'll show why this 67% number is compatible with a completely useless vaccine. Stay tuned.)

Matching No Longer Works

For any real-world study, we ask why we can believe that the group labeled unvaccinated and the group labeled vaccinated are identical except for their vaccination status. In prior studies I featured on this blog, the researchers rely on the matching technique to assert this claim. Matching is infeasible here because the entire study is centered on two high-risk groups, for whom the progress of vaccination was rapid. Among the nursing home residents, about 5 percent were vaccinated during the first week, reaching almost 95% by the end of the study period.

The Danish researchers treated this issue differently from the Clalit researchers.

In the Israeli study, the investigators matched each vaccinated person with an unvaccinated one on a rolling basis through the study period, and they ended follow-up on both members of each pair when the unvaccinated person received the first shot. This matching and censoring scheme creates many undesirable side effects (for details, see my prior post):

• the follow-up time is severely limited because most matched, unvaccinated patients quickly became vaccinated, stopping the count on both members
   
• cases on the vaccinated arm are not counted because of the early stopping described above
   
• the number available for matching shrinks rapidly throughout the study period, leaving the majority of high-risk subgroups unmatched, and dropped from the study.

By contrast, in the Danish study, few are dropped from the study (only those who have been infected prior to the first day of vaccination). Each person is followed through the entire study period (until they get infected, died or become untrackable). For anyone who eventually got vaccinated, the person contributes to unvaccinated person-weeks until the day of vaccination, after which the person contributes to vaccinated person-weeks. They don't have extra early-stopping rules triggered by something that happens to an unvaccinated person so every case will be counted (except when they play around with case-counting windows, which I'll cover in a future post).

But there is one big problem with this design, which the researchers recognized and tried to resolve.

Vaccination Timing Bias

Because of the design described above, cases counted as unvaccinated typically occur earlier in the study period than cases counted as vaccinated. If you think about any given individual, the time progression is always from unvaccinated to vaccinated. So earlier infection dates are correlated with being unvaccinated, and later dates with being vaccinated.

This is particularly problematic because vaccinations were happening in the backdrop of a general decline in cases post-holiday, at least partially related to a national lockdown that started in Denmark a week before vaccinations began. To deal with this bias, the Danish study added a "calendar-time adjustment" to their regression model.

Unfortunately, such an adjustment is better than nothing but isn't sufficient. In the rest of this post, I'll walk you through a numerical example to show how this timing bias is a real-world example of what statisticians call a "Simpson's paradox". Then, in a next post, I discuss what a regression adjustment does, why it doesn't get rid of the problem, and also address other residual biases.

Simpson's Paradox

For this demo, I start with an assumption that the vaccine is useless (VE = 0). Then, I show that using the usual way that we compute vaccine effectiveness, we will conclude that the vaccine is extremely effective. This "paradox" is of the Simpson's type, well known to statistics students.

As mentioned already, the Danish study attempts to correct for this issue. This demo explains why they must do something about it. It also provides an estimate of the magnitude of this bias (huge).

As background, we start with some real data taken from Figure 1 of the paper: the progress of vaccinations during the study period, and the weekly trend of cases in nursing homes.

Note in particular that the case rate (gray) dropped markedly over the course of the study.

Assuming VE = 0 means each week, the split of cases between the vaccinated group and the unvaccinated group is equal to the split of people between the two groups. This assumption makes the group-level case rates equal each week. The VE in any week is zero since VE is based on the ratio of group-level case rates. Like this:

So far, so good. Now, we compute a cumulative VE using the standard method. We sum the cases and the person-weeks over all weeks, and then divide the two to get an overall VE for the entire study period. This is exactly the calculation I did in a previous section, which led to a VE of 67%.

I hope your head is where mine is: WTF!

We forced VE to be 0% every week of the study period (by assumption), and yet the cumulative VE for the entire period turns into 67%. This is known as a "Simpson's paradox". The aggregated statistic leads to a startlingly different conclusion than the disaggregated (weekly) statistics.

This is the reason why the Danish researchers introduced a "calendar time adjustment" to their analysis. They realized that they must do something to overcome this paradox.

What's Lurking Behind the Scene

We expected the cumulative VE to be 0% since it's just the average of a bunch of zeroes. The problem is it's not a simple average but a weighted average. The weights reflect the progression of vaccination through the study period.

The unvaccinated group is front-loaded, and shrinks over time while the vaccinated group is back-loaded, and expands. So the aggregate case rate for the unvaccinated group is weighted toward the first few weeks while the aggregate case rate for the vaccinated group is weighted toward the last few weeks. Remember that the overall weekly case rate was high in the first two weeks, and then started declining. This explains why the unvaccinated group with a much higher cumulative case rate than the vaccinated group despite our assumption that the vaccine is useless.

In the above diagram, you see that for the unvaccinated group (pink), the largest weights coincide with the several weeks with very high case rates while for the vaccinated group (blue), the largest weights fall on the most recent weeks with close to zero case rates. Simpson's paradox is a situation in which aggregating the data removes too much information, and distorts the interpretation.

The magnitude of this distortion is extreme! What should have been 0% shows up as 67%.

(In the preprint, I only have the proportion vaccinated at the start and end of the study. In the demo, I drew a straight line, which means I assumed an even pace through those weeks. The reality is that the earlier weeks were probably slower than the later weeks, which makes the distortion even worse.)

***

The situation of rapid vaccination and declining overall case rate are prevalent everywhere so every real-world study, particularly ones focused on high-risk groups, faces this problem.

This analysis does not show the vaccine is useless. It shows that if the researcher does not correct for this timing bias, the estimate of vaccine effectiveness is extremely optimistic.

***

My second book, Numbersense, opens with an example of Simpson's paradox, and then it keeps getting better. Get your copy here.

In the next post, I explore how the Danish researchers attempted to fix this bias.

 

 

 

 

 

 

 

 

 

 

Real-world studies: how to interpret them

Recently, two types of data analyses about vaccine effectiveness have flooded the airwaves.

One type is being termed "re-analysis" of data from randomized controlled trials (RCTs). They used to be called "post-hoc", or "not pre-specified", or "exploratory." Serious analysts do engage in this type of analyses but we treat the findings as purely suggestive, requiring confirmation by another study. It's bad practice to treat them as capable of overturning or correcting analyses of RCTs by the original research team. Beware when the analyst is not arranging further studies to confirm such findings.

Another currently popular type of analysis is labeled "real-world studies." Technically called "observational studies," or "causal studies", these are worth the effort. As noted in my previous post, such studies should not be expected to correct or overturn findings from an RCT either. They are most useful in situations when we cannot run an experiment, e.g. if the US Postal Service wants to measure the impact of a price hike, it cannot randomly apply increases to some citizens and not others.

When we can run either RCTs or observational studies, the "real-world studies" benefit from larger sample sizes, especially for subgroups, and a more realistic context. It's a double-edged sword though. When the investigators cannot control the context, there will be more variability in the study population ("statistical noise"). The observed difference between test and control may be explained by biases, both known and unknown ("confounding").

Imagine an observational study that aims to estimate the real-world effectiveness of the Moderna vaccine. The simplest first analysis is to compare the case rate of the vaccinated people to the case rate of the unvaccinated people. This is hopelessly flawed because in a real-world study, we must not assume "all else equal." People who have received the vaccines at this point are apples and oranges to those who haven't. We already know about racial inequities but the differences go beyond that: geographical, cultural, economic, religious, access to technology, access to transportation, age, comorbidities, occupation, etc. may all explain some of the observed difference between the vaccinated and the not vaccinated.

The point of observational study methods is to statistically adjust the data. In Numbersense (link), I spend quite a few chapters explaining statistical adjustments (e.g. marketing and economic data). Critics complain that "massaging the data" is a horrible thing to do but for those of us invested in these methods, it is the only reasonable thing to do. Not adjusting the data when biases are present is to wilfully publish biased and therefore incorrect results.

In this post, I will walk you through an example of an observational study, conducted by the Mayo Clinic in partnership with nference (link). I chose this study because I'm impressed by its quality, and satisfied with the level of transparency into how they massaged the data.

What a real-world study tells us

Let's start with their headline finding:

They provide a "preliminary assessment of real-world vaccination efficacy in 62,138 individuals from the Mayo Clinic... between December 1st 2020 and February 8th 2021... Administration of two COVID-19 vaccine doses was 88.7% effective in preventing SARS-CoV-2 infection (68.4-97.1%) with onset at least 36 days after the first dose."

To a statistician, this is a heavily loaded statement, with every word chosen precisely, its meaning to be revealed in the rest of the paper. Here are the footnotes to save you time:

1. "preliminary": This is an interim analysis in which the individuals under analysis have not all reached the end of the desired observation window. The length of follow-up differs by individual.
   
   
2. "real-world": it's an observational study in which individuals did not get vaccinated at random, and so we cannot analyze this dataset using methods for analyzing RCTs. What determines one's vaccination status is self-selection plus government prioritization rules.
   
   
3. "62,138 individuals": we did not generalize our finding beyond the analysis set of patients in our hospital system. You are on your own if you should extrapolate our finding outside this group.
   
   
4. "December 1st 2020 and February 9th 2021": see point #3. Ditto for generalizing across time.
   
   
5. "administration" [of two doses]: the 88.7% number was arrived at after removing a few cases from the vaccine arm because those people have not received their second shots despite being past the target time for them. Without removing those cases, the effectiveness was 83%.
   
   
6. "onset at least 36 days after the first dose": as with other research teams, they made their own decision as to when to start counting cases. 36 days after the first dose is roughly 2 weeks after the second dose of Pfizer, and 1 week after the first dose of Moderna. (This just happens to flip the official case-counting windows from their respective RCTs, in which Pfizer started counting 1 week after the second dose, and Moderna, 2 weeks.)

How a real-world study assembles the control group

The key to any observational study is how they define the control group (i.e. the unvaccinated group).

They start with a vaccinated group. These are patients in the Mayo Clinic system who has received at least the first shot of either Pfizer or Moderna during their study period, which was 12/1/2020 to 2/8/2021. Each of these patients has a vaccination date, PCR test dates, and whether each test came back positive or negative. For the moment, we will concern ourselves with just the vaccination date (Day 0, day of the first shot), and any positive test result after Day 0 to indicate infected. 

To form the control group, they start with all patients in the system who has not been vaccinated as of 2/8/2021 (subject to exclusions which I'll talk about later).

The researchers collect demographic data and zip codes on all patients in both groups. They then assign "propensity scores" to each individual. These scores project the probability that an individual of given zip code, age, sex, race, ethnicity, and number of prior PCR tests has taken a first shot by 2/8/2021. For example, the model that generates propensity scores might say someone in zip code 55443, age 25, male, white, non-Hispanic and 1 prior test has a 20% chance of having been vaccinated (and 80% chance of not).

The originators of the propensity-scoring method demonstrated that we can form well-balanced control groups in the following way: within each zip code, take each vaccinated individual and find an unvaccinated individual that has the same propensity score. Within this basic framework, there are many variants. What does it mean by "same score"? How many controls to match to a single vaccinated person? (The Mayo Clinic does 1-to-1 matches.) Which individual is selected if several viable candidates are available? (They use a "greedy" method.)

What is the point of such adjustments?

The following excerpt of Table 1 from the paper illustrates the core idea of "covariate balance."

Each of the vaccinated and unvaccinated groups consists of 31,069 people because of 1-to-1 matching. The distribution of age groups, sex, race, etc. are all well-balanced between the two groups. Both groups are approximiately 63% female, 21% 75 years or older, etc. (Sidenote: I flagged the age groups because I was surprised that only 23% of the vaccinations went to 75 years and older, equal to the proportion that went to 18-25.)

This balance condition emulates the randomization in an RCT. But don't mistakenly think they are equivalent. If truly randomized, we can see balance on any variable known or unknown to the investigators. The balance for propensity scores holds for variables used in the adjustment - precisely because we made it happen - but there is no guarantee that there will be balance for any variables not used.

So, it's very important to make sure researchers use as many variables as possible, and include all relevant ones. This is one of these nuggets of advice that sound better than practiced. We're making an assumption that we know all the factors that matter. But we soldier on.

One of the curious variables used by the Mayo-nference team is the number of prior PCR tests (any time before 2/8/2021). They assert that this variable is a good proxy for the level of exposure to the coronavirus.

Matching the case-counting window

Now, we get to a tricky problem that I have discussed before. We have pairs of patients, one vaccinated and one unvaccinated, which we will treat as if randomized. The vaccinated one has a Day 0, which is the day of the first shot. What about the unvaccinated person? Establishing Day 0 is really important - given the desire of these investigators to pinpoint when an infection occurred.

In the Mayo Clinic study, they assign to the unvaccinated person a "study enrollment date" (pretend), which is Day 0 of the paired vaccinated individual. Now, they can start counting cases after Day 0 in both groups. This then lead to a vaccine effectiveness (VE) number.

As the above quote suggests, their headline number ignores all cases that occur prior to Day 36, meaning 2D+14 (2D = 2nd dose) for Pfizer and 2D+7 for Moderna, and Day 36 from the study enrollment date for the unvaccinated. The VE is 83%.

Like every other team doing re-analyses or real-world studies, these investigators computed VE for all reasonable case-counting windows, and selected the one with the maximum VE. They show their homework in Table 3. Here is an excerpt:

The top part of the table shows calculations of VE for cumulative case-counting windows of the type Day X to end of study while the bottom part of the table shows calculations of VE for incremental weekly windows of the type Day X - Day X+7.

Note that there are two rows for "Day 36 onwards". The second row has VE 89%, slightly higher than 83% (but of course, the margin of errors are plus/minus 15%). I'll let the authors explain their reasoning:

Starting 36 days ... vaccine efficacy of 83.4%... Importantly, we found that two of the six infections in the vaccinated cohort on or after day 36 occurred in individuals who had received only one vaccine dose, even though all vaccinated individuals should have received two doses by this time point... Among the properly vaccinated individuals... efficacy of 88.7%.

This style of reasoning is now state of the art. They are not alone in using it. Every recent study I've read is doing the same. I've said a lot about this issue in a prior post (link) so I won't repeat this here. The one addition is that this final adjustment only applies to the vaccinated group because the unvaccinated group has no such thing as missing a second shot (unlike in the RCT, when the control group got placebo shots).

Can this study's finding be generalized outside the study population?

Before ending this post, I want to address one other issue - why the research team was very careful in not generalizing their finding outside the analysis set of people.

There are two challenges to such a generalization: the vaccinated group in the analysis set has to be representative of all vaccinated Mayo Clinic patients; and then the Mayo Clinic vaccinated patients has to be representative of the wider vaccinated population.

(Note that because of matching, the unvaccinated group does not represent the general population. It mirrors the vaccinated group in the analysis set. That is the point.)

The analysis set is actually not representative because of a series of exclusions. They exclude:

• anyone who did not take at least one PCR test prior to February 8, 2021
• anyone who have not previously granted proper research authorization
• anyone under 18
• anyone who tested positive before December 1, 2020
• anyone living in zip codes with fewer than 25 people vaccinated at one of Mayo's facilities

In addition, for the vaccinated patients, they exclude:

• anyone who tested positive prior to their first shot (the data appear to show that people are given shots despite prior infections)
• anyone who got their first shot on the last day of the study period, thus having no follow-up period

The zip-code exclusion is particularly instructive. This creates biases. For example, less populous zip codes may be excluded. Also, it's likely that zip codes that have lower exposure may have fewer people interested in getting vaccinations.

It's very possible that sparsely-populated zip codes pose obstacles to finding matches. This is what is called incurable imbalance. If we can't find a matched control, we have to drop the vaccinated case. Dropping such cases makes the vaccinated group in the analysis set deviate from the vaccinated group in the system. (But it improves the balance of the people who are included.)

One of the key pieces of information you're looking for in an observational study is how many cases were dropped for lack of matches. This affects whether you can generalize the result. Sometimes, lack of matches is masquerading as an exclusion, as in the example above. If those small zip codes are excluded up top, they aren't looking for a match later but the effect is the same; we would have dropped them for lack of matches.

***

If you have come this far with me, I'm very impressed. I ran twice the length as the usual post so I'm going to move the rest into a future post. I will be addressing several concerns I have with this analysis for those who want to jump in the deep end. I believe this paper will stand the test of time as one of the better efforts at real-world studies.

 

P.S. [3/4/2021] The next post is published here. I delve into several technical issues so you can understand the limits to what real-world studies can tell us. It's an exciting area of research that blends art and science.

 

 

 

 

New year, more numbersense

Happy New Year. It's ever more important to use your numbersense when you hear data about the pandemic.

<1> The current count of "vaccinations" of Pfizer or Moderna - in the U.S., Israel, and anywhere else - is zero. No one is "vaccinated" until s/he receives the second shot. (Well, those who got two shots in the vaccine trials can be counted.)

<2> The dashboards are inflicting actual harm on the vaccination campaign, spreading the illusion that one dose is a complete vaccination. The world is soon to face a crisis of half-baked vaccinations.

<3> The government officials and celebrities must now get their second shots on camera to remind people to get them.

<4> In the Euro-zone, the second wave surge are being tamed after imposing more severe "lockdown" measures (mandatory night curfews, closing restaurants and bars, restricting movement, etc.) starting late October.

<5> Mortality trends are still highly predictable, demonstrating "statistical gravity". Current death rates reflect cases from up to a month ago. In the coming weeks, Italy's line would have fallen further, Germany's risen a little, and U.K.'s spiked. (Cross-reference the previous chart)

<6> The data in three key countries do not support the theory that the death rate from Covid-19 has declined. (Remember the U.S. and U.K. lines for the second wave have not yet peaked.)

<7> Rates don't tell the full story. Here are the cumulative death tolls, which show failure to control the spread is tragic.

<8> Death is a choice. Be smart.

 

Netflix, the data pioneer, gets flagged for data offence

The headline says it all. The Hollywood Reporter reported that "'The Witcher' Breaks Netflix Series Viewership Records -- with One Big Catch."

This should come as no surprise to my readers. It proves yet again you can use data to prove anything you want. Apparently, Netflix switched how it counts "viewers" in 2020.

Those not knee-deep in data work might not appreicate how measuring viewership can be so controversial. How hard is it to count how many people viewed a show?

In this case, Netflix announced 76 million households watched the series during the first four weeks of release. This seemingly simple statement includes a load of unspoken assumptions:

• Does each houshold have to watch all four weeks or one out of four weeks? (Nothing prevents one from requiring each household to watch two or three out of four weeks, either!)
  
  
• How much of each show must be watched in order to count as a "view"? (Hint: if the show is 60 minutes, absolutely no one will count 60 minutes. If you want to guess, select a number closer to 0 than to 60. Correct answer is disclosed below.)
  
  
• How many episodes must be be "viewed" to consider the series "viewed"? (The Witcher Season 1 has eight episodes, all released at the same time. I don't think you need a hint for this one.)
  
  
• Should each device and each sub-account be aggregated or not?

The above list isn't even comprehensive; there are even more subtle issues:

• What is being measured is not "viewing" but "streaming". The video player sends back regular signals to Netflix saying the player is still streaming a particular show. To measure actual viewing, the TV would have to track the viewers' eyes. (This is technically feasible although I think these companies aren't doing this.)
  
  
• Do repeat minutes count? If the rule is 10 minutes of at least two episodes, how do we deal with someone who watched 10 minutes of the same episode, twice?
  
  
• How to deal with fast forward, skipping, etc.?

***

As you look at those questions, think about whether there is a principled way to make these decisions. How does one argue that one out of four weeks is not a view but two out of four is? How does one argue that 10 minutes constitute viewing but 5 minutes don't?

It's pretty clear why you can use data to prove anything you want.

Still, what Netflix did is jaw-dropping. The previous definition of a view is "watching at least 70 percent of one episode of a series or 70 percent of a feature film". The new definition of a view - watching at least 2 minutes of one episode. You are reading this right; the 76 million households have "viewed" one Watcher episode for at least 2 minutes.

The first definition is already quite generous. The rule of the game is to get enough people to turn on the first episode, and it doesn't matter that they never return. Since Netflix has the famous recommendation engine, it's got the tool to drive traffic to the first episode of pretty much anything. The audience have to stick around for 70 percent of the running time, though.

How does Netflix justify dropping the bar from 70 percent of one episode to 2 minutes?

From the Hollywood Reporter:

Our new methodology is similar to the BBC iPlayer in their rankings based on 'requests' for the title, 'most popular' articles on The New York Times, which include those who opened the articles, and YouTube view counts," the company notes in its earnings report. "This way, short and long titles are treated equally, leveling the playing field for all types of our content including interactive content, which has no fixed length."

Netflix is right about Youtube. Youtube counts 30 seconds of run time as "viewing". Also, Youtube is autoplay by default. Once you start playing one video, it performs a version of "doom scrolling" so it keeps playing the next clip. I'm surprised people actually suggest that such "autoplay" views are not counted as views. Doing so would disqualify a ton of legitimate viewing as not views - and Youtube makes money from generating views!

The most objectionable part of the quote is "short and long titles are treated equally". According to Netflix, treating someone who watched 2 minutes of a 10-minute clip the same as someone who watched 2 minutes of a 240-minute movie demonstrates fairness.

You know it's a problem when the agreeable press not only noticed the data offense but also decided to stick it in the headline rather than burying it two paragraphs from the end of the article.

***

The issue of definitions is everywhere in data analysis. Covid19 tests and deaths. Flight delays. Mean tweets. Twitter users. Facebook video view time. Wind chill.

In Chapter 2 of Numbersense (link), I featured some doctors who believed that changing the definition of obesity will solve the obesity problem in the U.S.

P.S. [10/20/2020:

Mark Palko alerted me to his take on this news also. I think what he's saying is Netflix knows (of course) how much time each device "viewed" each minute of each episode, it just chooses not to publicize that data. Great point!

Here is another post by Mark that covers how Netflix uses "deceptive messaging and suspicious statistics". 

]