Big Data, Plainly Spoken (aka Numbers Rule Your World)

Swear and gossip: answer to question 1

Last week, I posed two questions related to the Business Insider headline about swearing and gossipping in the workplace (link). Today, I provide my answer to the first question. The answer to the second question is here.

Question #1: Are the following headlines identical (i.e. interchangeable)?

Answer: They are not identical.

More precisely, the two statements can be identical under limited conditions but they are usually not interchangeable. As Scott quipped on Twitter, it depends.

A reason why we may think those headlines say the same thing is that people tend to draw the same causal conclusion from each headline: if team members swear and gossip more, their chance of success increases.

We can unpack the statements. The key difference between them is the order of presentation. The first statement starts with the base of teams that are rated as the most successful by some metric of success, and within this base, there is a high incidence of swearing and gossiping. The second statement starts with the base of teams that engage in swearing and gossiping, and within this base, there is a high incidence of success.

For those familiar with conditional probabilities, the first statement concerns the probability of swearing and gossiping given that a team is "most successful" while the second statement is about the probability of success given a team engages in swearing and gossipping. In general, P(X/Y) is not the same as P(Y/X).

According to the famous Bayes' Theorem, the ratio of those conditional probabilities is equal to the ratio of the prior probabilities so the two statements are identical if and only if the rate of success among all teams is the same as the rate of swearing and gossipping. P(swear and gossip) = P(success) is an unlikely event in real life.

***

Let's bring back the helpful diagram from the answer to Question #2, that lays out the four possible characterizations of any team.

Consider an extreme case in which all teams are either in A or D. Every team that swears and gossips is successful while every team that doesn't fails. So B and C both are empty.

In this case, A+B (teams that swear) and A+C (teams that are successful) are identical. The prior probabilities, and also the conditional probabilities are the same. We conclude that the most successful teams swear and gossip, and also that the teams that swear and gossip are successful.

In the above diagram, the conditional probability of swear and gossip given success is A /(A+C) and that of success given swear and gossip is A/(A+B). For those to be identical, B and C must contain the same number of teams. In the extreme case, B = C = empty. If we put x teams in B, then we must put x teams in C, and then we have to subtract x teams from A and from D to make things even.

There are a small number of scenarios in which this can happen, and a much larger number of scenarios in which the two probabilities aren't the same.

***

Consider the two statements in this format:

Formula (I) comes from the original headline while formula (II) corresponds to the other headline.

Formula (II) predicts success (... leads to success) while formula (I) describes success (a successful team does ...). Formula (II) does not imply formula (I).

In the extreme case, every team swears and gossips, and so it follows that the most successful teams swear and gossip. Since everyone does it, swearing and gossiping will not predict success. Formula (II) does not imply formula (I).

In order to learn Formula (II), we must add other factors that affect team success.

(In the above, I excluded the possibility that success may cause swearing. Allowing this relationship complicates the causal interpretation even more.)

***

The confusion between descriptive and predictive (or causal) statements is endemic. For example, in coronavirus news, we keep hearing statements such as "Most of the hospitalized people are  unvaccinated." This is very different from the related statement of "Most of the unvaccinated people are hospitalized."

If we think of vaccination status as a cause of an effect, and hospitalization as an outcome, then "Most of the hospitalized people are unvaccinated" is a description of people with a given outcome. It is not a predictive statement supporting a claim that vaccination status predicts some outcome. In addition, the first statement does not imply the second statement - that "most unvaccinated people are hospitalized."

 

Sharper understanding of vaccine effectiveness

Recently, some researchers changed the way they describe vaccine effectiveness. It used to be the Pfizer vaccine is 90% effective at preventing (first it was cases, then it was severe cases, finally it was) deaths. Now, they are saying the vaccine cuts (cases, severe cases, deaths) by 11 times. (See, for example, this recent report on the booster shots in Israel.)

Cutting some bad metric down by 11 times sounds extremely impressive. Is 11 times better than 90%?

The answer: they are saying exactly the same thing. If the rate for unvaccinated is 11x, and that for vaccinated is x, then the vaccine effectiveness is 1 - (x/11x) = 91%.

Cutting a number by 11 times is great but hard to achieve. Now, if we have a vaccine that cuts infection rate in half, that is still quite good, right?

That vaccine would have a VE of 50%. CDC doesn't even want to talk about such a vaccine.

***

I will say it again. A 90% vaccine effectiveness does not mean that 9 out of 10 people who got vaccinated are protected. As explained here, VE is a ratio of two infection rates. Both of these infection rates are very small, typically below 10%, and sometimes below 1%, depending on which clinical trial the data came from. In other words, over 90% of the people in the placebo arm did not get sick during the study period. It's logically impossible for 90 out of 100 people to be protected when fewer than 10 unvaccinated people would have gotten sick.

Statements such as "99% of vaccinated people are not hospitalized with Covid-19" are not very informative, unless we are also told what proportion of unvaccinated people are not hospitalized. If we have both, and flip the focus from not hospitalized to hospitalized, then we are looking at a ratio of rates, which is the vaccine effectiveness measure.

Those alternative descriptions are distracting and harmful when misinterpreted.

***

One last thing. A 90% effectiveness against cases and a 90% effectiveness against deaths are not at all similar things.

That's because the rate of death is a fraction of the rate of cases. When talking about cases, a 90% VE may mean pushing case rates from 10% to 1%. When talking about deaths, a 90% VE may mean pushing death rates from 0.5% to 0.05%. It is ludicrous to say that 99.95% of vaccinated people are protected from deaths when 99.5% of unvaccinated people would not have died.

A big drop in death rates is impressive on its own. Less effective is not the same as not effective. I don't like these alternative descriptions of effectiveness because the math is incorrect.

Swear and gossip: answer to question 2

Today, I give my answer to Question #2 in the test presented a few days ago, here.

***

Question #2: Do the following headlines contradict or can both be true at once?

When we are presented with just the first statement that the most successful teams swear and gossip, we are tempted to draw a causal conclusion. We think swearing and gossiping is the cause of success. This motivates the advice given by the headline writer. (Okay, so the writer did not say out loud encourage your team members to swear and gossip more... but is there any other possible interpretation?)

If the cause--effect relationship holds, we may expect the least successful teams to not engage in swear and gossip, which means the truth of the first statement denies the truth of the second, and vice versa. (Even this is not certain since there are other factors affecting team success.)

If we are told both statements are true at the same time, I don't think any of us would conclude that swearing and gossiping determines team success. That's why it's important to know if one statement implies the truth/falsehood of the other.

Answer: Both statements can be true at once. One does not imply the other.

More precisely, one may imply the other but often doesn't. As Scott quipped on Twitter, it depends.

One approach is to lay out the 2x2 grid of all four possible states of being. Like this:

Each team is classified as either successful or not (based on some metric of success). Each team is also classified as either heavy or light on the swear-gossip axis. The sum of the four numbers is the total number of teams.

The statement that "the most successful teams swear and gossip" suggests that A is substantially larger than C, or that A is a large proportion of (A+C).

The statement that "the least successful teams swear and gossip" suggests that B is substantially larger than D, or that B is a large proportion of (B+D).

The question is whether the values of A and C constrain the values of B and D in such a way that if the first statement is true, then the second statement must be false. To invalidate this, we can show scenarios under which both statements are true.

Imagine we have 100 teams. There are only two polite teams that don't swear and gossip, split evenly between successful and not successful. Next, we have 50 teams in cell A (successful and swearing) and 48 teams in cell B (not successful and swearing). Now, among the successful teams, 50/51 swear and gossip while among the unsuccessful teams, 48/49 swear and gossip. Both statements are true. The first statement does not deny the second.

We have a whole class of scenarios that work similarly. The split between A and B can be 49/49, 48/50, 40/58, 55/43, etc.

***

For those who have been exposed to contingency tables like the one above, you may be tempted to fill in the four numbers by multiplying the probability of success and the probability of swearing and gossipping. When you do this, you're assuming independence.

Independence means that knowing whether team members swear and gossip provides no information on team success. This is the opposite of the study's intention. The researchers must believe that team success is dependent on the level of swearing and gossipping.

To model dependence, it's not sufficient to know just the probability of success and the probability of swearing and gossipping. For each level of swearing and gossipping, we assume a different rate of success.

Through this exercise, we learn that three of the four values (A, B, C, D) are free. The only constraint is that they must sum to the total number of teams. In other words, having specified A and C in the first statement, we have only nailed down the sum of B and D but the split between B and D is not constrained. Thus, both statements can be, and often are, true at once.

***

In Numbersense (link), I described a real-life example of this fallacy.

Years ago, the Gates Foundation was putting a lot of money behind the "small schools movement". This was based on some studies showing that the top X schools in district Y have a high proportion of small schools (with small average class size).

The situation can be shown in the following table:

Notice that this is the same table as above, with "small class size" substituting for "swear and gossip".

The Foundation was shown just the first column, from which they concluded that small class size was the cause of high test scores (A being a large proportion of A+C). The statistician, Howard Wainer, looked at the entire distribution and demonstrated that the bottom X schools also have a high proportion of small schools (B being a large proportion of B+D). This corresponds to the second column.

If shown both columns, you'd not conclude that class size drives test scores. If you only know the first column, you'd be tempted toward that conclusion.

If you only know the second column, you'd be tempted to draw the opposite conclusion that small class size depresses test scores!

That's why data analysts want to see all of the data. That's why biased datasets can be extremely misleading. You can make errors of direction, not just magnitude.

***

There is an underlying statistical reason that makes Wainer's discovery inevitable in many real-world datasets. Extreme values (highest or lowest test scores) are associated with small class sizes, because small samples have higher variability.

Imagine a large school with 10,000 seniors, with average GPA of 3.75. Imagine taking a random sample of 1,000 students, and reporting the sample average GPA. It's not going to be exactly 3.75, off by some small amount. Now take a sample of 500 students, and the average deviation of the sample mean from 3.75 will be higher. Now do 100 students. The smaller the sample size, the more likely the sample average runs further away from 3.75.

In the business teams example, I would not be surprised (if they did not control for team size) that the most successful (and least successful) teams are smaller teams.

This is an example of statistical gravity: if the dataset consists of teams of varying sizes, the large teams will be found in the middle of the distribution, and the smaller teams will be found towards the tails.

 

 

Swear and gossip: a test of your statistical acumen

Business Insider has been flogging this headline since late August, and it was still on the front page last week:

I don't have a premium subscription so I don't know what evidence they have for this. I'm imagining that someone collected data, zeroed in on the teams that are deemed successful, and computed the percentage of team members who "swear and gossip". How do they know they "swear and gossip"? I have no idea - the data may have come from one of those Big Brother surveillance technologies.

***

I love this headline because it can be turned into a test of one's statistical acumen.

Question #1: Are the following headlines identical (i.e. interchangeable)?

What's your first instinct? Now, think about it a bit, and see if you change your mind. I'll present some hints in the second half of this post, and the answer in a future post.

 

Question #2: Do the following headlines contradict or can both be true at once?

What's your answer? Come back later in the week to check what my answers are.

For those needing some hints, keep reading.

***

For Question #1, one perspective is whether the order of presentation matters. Is the relationship symmetric? Imagine throwing a dice twice, and getting 1 then 5 in that order. If the question is the sum of the values, then the order does not matter. One followed by 5 is the same as 5 following 1, both adding to 6, in the context of this question. However, if the question is the values are increasing, then order matters. One followed by 5 and 5 followed by 1 are evaluated differently.

One approach that helps with either question involves writing down the entire set of possibilities. When you have two binary variables, there are four possible states. Are the two headlines talking about the same things?

Identity requires that each statement implies the other. Check both directions. If it's not true, then find counter-examples in which one statement is true while the other one is false. Question #2 asks about non-contradiction which is less stringent than identity. Identical statements cannot contradict each other but non-identity does not imply contradiction.

Remember we are doing statistics, not pure math. Almost nothing in statistics apply universally. Think of counter-examples not as a single example but a class of examples sharing the same characteristics.

***

Feel free to use the comments to discuss with other readers.

 

The booster shot data reveal why RCTs are still the gold standard

Observational studies have some common characteristics that annoy the analyst. In the previous post about Pfizer's booster studies, I described a few.

• The sample contains few examples of the people to which you're generalizing your result (e.g. people 65 or older, high-risk individuals of any age).
• The measured outcome is not the obvious, direct outcome (e.g. clinical outcome of infection) but an indirect indicator (e.g. antibody levels).

These problems create the need to extrapolate results. Such extrapolation is usually supported by unverified assumptions (aka expert opinion). Of interest, the FDA allowed Pfizer to impose a causal assumption: that if the booster achieves similar levels of antibodies as the original 2nd dose, then the booster also produces similar clinical outcomes as the original two doses. See the previous post for why modifying an indirect outcome does not invariably lead to a change in the direct outcome. (This matter is yet another real-world example about correlation and causation.)

As I have explained often on this blog, making assumptions is not a sin. All statisticians make assumptions - those who claim they make no assumptions make assumptions! Let me give a quick example. Suppose a college runs a survey asking recent graduates about their post-graduation salaries. Only 50% of the graduates responded with a valid number. Should the college impute the salaries of the non-respondents? If I were the analyst, I would impute the salaries, baking in the assumption that those who don't respond are likely to be earning lower salaries (possibly even zero). However, some statisticians will argue against using imputation. They'll claim that one should just report the average of those who responded, because it is a horrible thing to modify the data - one should only correct egregious data entry mistakes but not otherwise touch the data. Are they making no assumptions? 

They are making a huge assumption: they assume that the non-respondents have the same average salary as the respondents. What is the basis for that assumption? There is none, other than the desire to not tamper with the dataset. In fact, this assumption is almost surely wrong. As I said in Numbersense (link),

Beware of those who don't tamper with the dataset!

***

In this post, I'll cover several additional extrapolations that were used in analyzing the booster data.

The key immunogenicity results all measure antibody levels against the original virus (so-called wild type) but as we've been constantly reminded, the only virus currently in circulation is said to be the Delta variant. How did Pfizer bridge this gap?

According to the FDA Briefing document (link to PDF), "Pfizer proposes to infer effectiveness of the booster dose against the Delta variant from exploratory descriptive analyses of 50% neutralizing antibody titers against this variant evaluated among subjects from the Phase 1 portion of the study." 

"Exploratory descriptive analysis" is not regarded as a serious form of scientific evidence - the rules have changed during this pandemic emergency. I pity the statisticians working on these studies. They were assigned an impossible task. Recall that the gold-standard randomized clinical trial (RCT) has been thrown overboard, replaced by "immunobridging" analysis. The antibody levels stimulated by the booster shot are deemed comparable to those registered (by other individuals) after their second shots, and then the "non-inferiority" in antibody levels is assumed to result in the "non-inferiority" in vaccine effectiveness (a clinical outcome not directly measured).

The Delta variant introduces another complication: as it was not present around the time of the original Pfizer adult trial, researchers do not know what the VE of the first two doses is against Delta. This breaks the causal assumption of Immunobridging.

We do not have even an approximation of the VE outcome against the Delta variant for the 300-odd people in the booster trial. So if one accepts that one can expose the old blood samples to the Delta variant in the lab, and show that the antibody levels are comparable, one has to make an even stronger assumption by claiming that those antibodies will result in the same VE as measured in the Pfizer adult trial.

This line of argument unfortunately results in a contradiction. The original vaccine either works just as well against Delta as it worked against the original strain; or it is less effective, which explains recent surge in cases. Only one of these scenarios can be true.

If the first scenario were true, then the immunobridging assumption holds as VE for Delta is the same as VE for the wild type, but in this scenario, the recent surge in cases has nothing to do with Delta, and there is no need for a booster - since VE is the same.

If the second scenario were true, then the booster may be necessary, but in this scenario, the immunobridging methodology is broken as the prior value of VE no longer applies.   

Since Pfizer applied for approval for the booster, it is assuming the second scenario. One might be tempted to prefer assumption #1 because it permits the immunobridging analysis. That is a popular justification you find in academic papers, in the same vein as "computational feasibility" or "tractability". It's risky to use this logic in studies with real-world consequences. Besides, it creates a contradiction.

Beyond the logic of the analysis method, notice that the Delta analysis is performed using only Phase 1 participants, so we are talking about 11 people between 18 and 55, and 12 people between 65 and 75 years old. I'm not sure why blood from the other 300 people cannot be similarly analyzed. Without those results, the immunobridging assumption is expanded yet again to generalize from the 23 people to the 300-plus people (then to the 20K-plus in the adult trial, and finally to the U.S. population.)

A proper RCT would have yielded a much cleaner analysis of the direct clinical outcome. 

***

Another missing piece of the puzzle is the provenance of the 300 or so Phase 2/3 trial participants who were "selected" to take part in the booster study. Nowhere in the FDA Briefing document can I find the selection mechanism. Usually, this means the selection was not random from the vaccine arm of the earlier trial. If it is not random, what are the criteria? 

The 300 people reduced to about 200 that ultimately were used in any of the primary calculations. That's a drop-off of a third of the starting population, which is a high drop-off rate. (Note: Table 3 claims the evaluable immunogenicity population is 268 but the results shown in Tables 4 and 5 unexpectedly have N=210 and N=179.)

Some of the reasons for exclusion are perplexing. Let's review a few of these reasons.

Six people refused the booster shot. There is an argument that they should be excluded as they did not get the treatment under study, and there is no blood to be analyzed. 

Fifteen were dropped because they did not have "at least 1 valid and determinate immunogenicity result within 28 to 42 days after the booster shot." This sets my alarm bells ringing because this exclusion can only be applied after the primary outcome of the study is measured. It's not clear what constitutes "valid" and "determinate". It's concerning to lose 5% of the study population by effectively saying we failed to obtain the primary endpoint.

Then, they dropped a further 30 people (10% of the study population) because their clinician decided that these people committed "protocol deviations" before the 3D+30 day evaluation time.

Last but not least, they also removed 34 people from the study population (15%) based on a key clinical endpoint. This exclusion criterion is described as "evidence of infection up to 1 month after booster dose". Notice it says "after" booster dose, not before. So, it appears that the following happened: they selected about 300 people to start this study regardless of their prior infection status, various people were excluded for the reasons described above, blood samples were collected, PCR tests were performed (on the day of the shot, and then when the participants self-reported symptoms), and if anyone gets sick within 30 days of receiving the booster shot, they were kicked out of the study.

Remember the design of this study. The indirect outcome of antibody levels is used to infer the direct outcome of infection. So when someone is dropped because of infection, we should infer that the dropped person has inadequate antibody levels. These deletions induce a bias in the primary endpoint, raising the average antibody levels of those who remain in the study.

Why is the case-counting window set to 30 days instead of the 7 days used for the 2nd dose? That's not explained in the Briefing document.  

***

I may come back to the safety data some day but the sample size is so small that only really loud signals can be heard. If no adverse effects are found, one can't conclude that there are no adverse effects; one can only say that there are no adverse effects that can be detected by this study design.

 

Reading the evidence for the booster shot

The FDA Advisory Panel met in mid September to discuss Pfizer's application for a booster shot. This led to a partial authorization (link). The key statement is:

Today, the U.S. Food and Drug Administration amended the emergency use authorization (EUA) for the Pfizer-BioNTech COVID-19 Vaccine to allow for use of a single booster dose, to be administered at least six months after completion of the primary series in:

• individuals 65 years of age and older;
• individuals 18 through 64 years of age at high risk of severe COVID-19; and 
• individuals 18 through 64 years of age whose frequent institutional or occupational exposure to SARS-CoV-2 puts them at high risk of serious complications of COVID-19 including severe COVID-19

In today's post, I read the FDA's Briefing Document (link to PDF), released as part of the September meeting in order to understand the science behind this decision. This briefing represents the FDA's interpretation of data submitted by Pfizer.

***

The most striking discovery is that the FDA no longer requires a randomized clinical trial (RCT) to authorize vaccines. As explained in my research methods talk (link), RCTs - long considered the gold standard for data-driven medical science - have six key components: random sample of underlying population, randomization of treatment, double-blinding, control, placebo, pre-specification.

With the FDA's blessing, Pfizer offered data from two observational studies; these studies do not have most of those key components that make RCTs the standard.

The first study is a Phase 1 trial, consisting of a total of 23 people. All of them were given a third (i.e. booster) shot about 7 to 9 months after their 2nd shot.

The second study involved about 300 participants who enrolled in the big Pfizer adult trial. (Here is the link to results from that trial.) All of them were given a third (i.e. booster) shot, about 6 months after their 2nd shot.

In all, the FDA looked at two studies that enrolled fewer than 350 people, without a control group, without placebo, and not blinded. The investigators wished to generalize from about 300 to about 300 million people in the U.S. I'm unsure as to why they can't find 300 other participants from the big trial so that they could randomize treatment into extra shot versus placebo.

The FDA advisory committee apparently were not fully convinced by the data but they nevertheless authorized the booster shot for 65 plus and others at high risk. This is a curious decision.

The second study with 300 participants contained zero people aged 65 and above. The age cutoff was 55 years old. So the only older participants came from the first study, and there were N=12 such people labelled as 65 to 85 years old. Table 2 in the Briefing document revealed that despite the labeling, the 12 people's ages ranged from 65 to 75 years old so there was no data submitted for people above 75 years old. This is an example of tokenism, which I described here.

Further, the Phase 1 trial excluded all people who are at high risk of Covid-19 infection, and so all the evidence we have for 65 plus are from 12 healthy, low-risk individuals aged 65 to 75. Table 2 confirms that this dozen did not represent the diversity of Americans - not surprising given the tiny sample size. They are 100% white, 0% obese, and 0% with comorbidities.

The evidence for the second recommendation for 18-64 who are at high risk of severe Covid-19 is similarly stretched. Presumably, indicators of risk of severe disease include comorbidities and minorities. The second study only included people aged 18-55 so the only data available for the 55-64 age group came from the first study - however, none of these people are at high risk of Covid-19, let alone severe Covid-19, by virtue of the Phase 1 study's exclusion criterion.

As for those in the 18-55 age range, the second study enrolled 306, of which only 55 have comorbidities, only 28 are black Americans, and only 2 are native Americans. In short, the FDA authorization is based on observational studies that have really small sample sizes, which get attenuated further when the recommendations are restricted to subgroups. The advisors extrapolated these results to subgroups that have few or zero data based on their expert opinion. (I'm not denigrating expertise, just pointing out how it is applied.)

***

The FDA has months ago told the pharmaceutical companies they no longer have to run RCTs. They said that future "modified" vaccines can be approved based on "immunobridging" analysis. In this Briefing Document, the FDA further extends the rule to booster vaccines that are not modified from the original, which is what the Pfizer booster shot is.

The logic of immunobridging is to show "non-inferiority" in antibody levels after taking the booster shot compared to after taking the two-dose series. Instead of comparing vaccinated to placebo as in an RCT, such analyses compare people who took the booster shot to people who took the first two shots. If the antibody levels are non-inferior - as defined by statistical criteria, then it is presumed that the booster shot will provide the same level of protection as the original two doses, i.e. the famous 95% vaccine efficacy.

Pfizer submitted data that showed the antibody levels after the booster shot were non-inferior, using thresholds to which the FDA has agreed.

***

What is "immunobridging"? This bridge is what I'd call a "causal assumption".

You see, neither Pfizer nor the FDA can define the relationship between antibody levels and getting infected. During the Advisory Committee meeting, a Pfizer scientist acknowledged, "We actually looked at our breakthrough cases in our placebo-controlled phase 3 study, and have compared the antibody titers where we had the opportunity in individuals that got the disease versus the ones that didn’t, and we were also unable to really come up with an antibody threshold. So I think there’s probably a much more complex story and not easily just addressed with neutralizing antibodies."

What they were hoping for is a simple rule that says antibody levels > X means vaccinated person is protected. The "much more complex story" implies that the rule must involve not just antibody levels but many other factors.

We therefore know that antibody levels is necessary but not sufficient to induce immunity. The causal model requires other variables, some of which may be unmeasured. The chosen solution is to make a causal assumption: just assume that antibody levels is sufficient for protection. This is the meaning of the following sentence in the summary section of the FDA Briefing Document (link to PDF):

Effectiveness of the booster dose against the reference strain is being inferred based on immunobridging to the 2-dose primary series, as assessed by SARS- CoV-2 neutralizing antibody titers elicited by the vaccine.

Two steps are involved here: the 95% VE estimated in the adult trial is assumed to be directly caused by the vaccine producing enough antibodies to fight off infection - while assuming no other factors play a role; then, the booster shot is assumed to produce 95% VE if it induces "non-inferior" antibody levels compared to the original two shots.

***

The next question is how long will immunity coming from the booster shot last. The short answer is 1-2 months; it could be more but we don't have the data since this is another interim analysis with a brief follow-up period.

The second study has the bulk of the people (about 300). The follow-up periods range from 1.1 to 2.8 months post booster shot. But we should subtract 1 month because there is a case-counting window that would start 30 days after the booster shot. Effectively, the period of time during which these 300 people can accumulate cases is as low as 3 days, and up to 1.8 months (54 days). (The first study with 23 participants has the same maximum follow-up time.)

Where does the 30 days come from? This is analogous to the 14-day period in which cases are removed if they occur in vaccinated people. The antibody measurements were taken 30 days after the booster shot and compared to measurements taken at 2D+30 days in the previous adult trial. Since we are inferring protection based on antibody levels, we can't infer protection for any time prior to 3D+30 days by measuring antibodies at 3D+30 days. Effectively, for the third dose, the case-counting window has been pushed from 14 days after the shot to 30 days after the shot. Anyone who gets sick within one month after getting the booster will not be considered "breakthrough".

***

This post just scratches the surface of all the issues that arise when we don't have RCTs to evaluate medical interventions. These observational studies are small, do not have randomly selected participants, do not have a placebo/holdout arm, and are not blinded. As a result, any observed outcome could have been caused by a host of factors, including unknown or unmeasured variables. This forces data analysts to make assumptions. The quality of the analysis depends heavily on the quality of these assumptions.

 

 

 

 

Measuring value by how much was spent

Over at Andrew's blog, he's checking some numbers from Albert-Laszio Barabasi's latest pop-sci book, and having a load of fun with it. It's something I used to do when I had more time reading pop-sci books.

One sloppy passage from the Barabasi book is this:

It’s possible to put actual monetary value on each citation a paper receives. We can, in other words calculate exactly how much a single citation is worth. Any guesses? Shockingly, in the United States each citation is worth a whopping $100,000. We know this by looking at the amount of money the nation spends on research . . . If we then divide that figure by the total number of citations collectively generated by the papers these funds paid for, we can estimate the cost of a single citation.

Barabasi claims that each citation in each research paper published in the U.S. is worth $100,000. As Andrew and his readers pointed out, the paragraph is filled with loose language and fishy analysis.

The "exact" monetary value is certainly not exactly $100,000. An "estimate" usually isn't "exact". "How much a single citation is worth" is different from "the cost of a single citation". The only thing we know for sure is that Barabasi divided the money spent on research by the total number of citations. When Andrew traced the origin of this formula, he found that Barabasi cited an unpublished report by someone else, and we don't have any details on how it is derived.

Spending is not the same as value. The U.S. spends the most money by far on healthcare amongst peer countries, and as we have learned, our health outcomes are far from world best (link). If an advertiser spends $5 million on a SuperBowl ad, the brand has not created $5 million in value. In fact, the business starts in a $5 milllion hole, and must generate incremental sales above and beyond $5 million to achieve a return in investment! Citations, as far as I know, do not recuperate any investment dollars for the nation.

Another problem is to pretend there is no time lag. This bad assumption is also used by epidemiologists throughout the pandemic when they take last week's new deaths divided by last week's new cases, and call that a case fatality rate. Deaths, in reality, lag infections by weeks so last week's new deaths derive from new cases from many weeks ago. This time-alignment error is most acutely felt in data that exhibit large temporal shifts.

If cases are growing rapidly, this method of computing fatality rate under-estimates it. The analyst may get away with this for a while but when the growth slows, the reverse effect kicks in -- and that's when the analyst will suddenly change the formula. Look out for that!

Barabasi appeared to have assumed that R&D spending in a given year leads to citations in the same year. It's unclear how he dealt with citations this year of a paper published several years ago.

***

The next issue is survivorship bias. A paper with zero citations costs nothing, according to his formula. So if I were the research budget administrator, what should I do?

How about funding lots of researchers who publish useless papers that are neither read nor cited? I can achieve a monetary value of ... infinity! (Divide a big number by zero).

Ideally, I will score exactly one citation so the average value per citation is my entire research budget. Now, I apply for new funding based on value per citation.

The error is to compute the average based on the subset of survivors (those papers with at least one citations) while ignoring the non-survivors (papers without citations).

It's like computing the average rate of return of the hedge fund industry, ignoring any hedge funds that imploded during the year.

 

 

 

Why do we have biased datasets?

Why do we have biased datasets?

The answer is simple. It's usually because biases have been actively injected into the data collection processes.

***

As U.S. colleges reopened to in-person teaching this semester, on the strength of a successful vaccination campaign, some schools continue to enforce testing. Last year, I covered how many colleges - such as Cornell and Georgia Tech, succeeded in keeping on-campus infections down by running a strict testing and tracing program (link, link). Staff and students were tested once or twice a week.

This term, the testing policies have been modified. At some colleges, people are still getting tested weekly. Make that some people. Specifically, these colleges require people who do not show proof of vaccination to get tested weekly. Meanwhile, fully vaccinated people are tested, only when they decide to - which means, when their symptoms have become so severe that they present themselves to a testing clinic.

This is a perfect example of injecting bias into one's data collection process. More testing leads to more reported cases. This bias is due to asymptomatic cases and mild cases, and as described before, is (inadequately) revealed by the positivity rate (what proportion of test results come back positive). Compulsory weekly testing adds asymptomatic and mild cases - as well false positive cases - to the tally. This is actually a good thing as Cornell and other schools demonstrated last year.

The new testing policies at some colleges mandate one set of rules for the unvaccinated, and another set of rules for the vaccinated. Because only the unvaccinated are subject to weekly testing, the case count for the vaccinated will include only severe cases while the case count for the unvaccinated will include everything.

***

Such biased data directly result in biased statistics, which lead to errant decision-making.

The differential testing policy guarantees that most of the reported cases happen to unvaccinated people - even if the vaccine were useless. Assume weekly surveillance testing last year found a run rate of 100 cases. A good working assumption is that half the Covid-19 cases are asymptomatic so 50 asymptomatic cases. Now this year, assume 80% on campus are vaccinated, and one's vaccination status is independent of infection risk. Then if the run rate stays the same, then 80 cases will be among the vaccinated subpopulation while 20 cases will be among the unvaccinated. However, testing policy has changed so that all 20 unvaccinated cases will be detected while only the very sick get tested among the vaccinated. Conservatively, we assume 10% of infections become severe. That means 8 of the 80 vaccinated cases will enter the database, accounting for 8/28 = about 30% of all reported cases.

This mix of cases is then turned into a naive estimate of vaccine effectiveness: the unvaccinated has a (70/20)/(30/80) = 9 times higher chance of getting infected. The trouble is that this result is entirely driven by the reporting bias due to differential testing. No assumption of vaccine efficacy was made in the above calculation.

If we add an assumption that the vaccine is 50% effective at stopping infections, then the 80 cases among vaccinated - the run rate from last year - should have turned into 40 cases this year. Then, only 10%, or 4 cases, would be detected because only very sick people present themselves to be tested. In this scenario, the total number of reported cases is lower, and 4 out of 24 reported cases (17%) occur among the vaccinated. The imputed "vaccine effectiveness" (using the naive methodology) shows that the unvaccinated has a (83/20)/(17/80) = 20 times higher chance of getting infected relative to the vaccinated.

A relative ratio of 9 times corresponds to VE of 89% while 20 times is 95%. In other words, bias in the data due to differential testing explains 89% of the 95% naive vaccine effectiveness. The decision-maker operates on the real-world evidence of 95% VE, unaware that the bulk of this number is explained by the misguided differential testing policy.

***

The potential for harm goes beyond counting cases. There have been some reports that claim that vaccinated people who got sick carry even higher viral loads than unvaccinated people.

Once again, I hypothesize that the differential testing policy may explain some if not all of this effect. The average vaccinated person who gets tested has a more severe case of Covid-19 than the average unvaccinated person.

***

The reason why one should resist all biased data collection processes is that it introduces additional factors that can explain some or even all of our outcome metrics, making it harder to prove that the thing we are interested in (vaccination, in this case) is the true driver of those outcomes.

***

What about those colleges that simply require all community members to get fully vaccinated? In this case, they will only test severely sick people (under the assumption that the vaccine is an invitation to the virus.). There are no unvaccinated people to draw comparisons to, and so it may appear that the above problem has been avoided.

Not so. For these colleges are likely to trumpet a comparison of case rates between this year and the previous year, and conclude that vaccinations work. However, there are other differences between the two years. One glaring difference is the amount of testing, which has declined dramatically. Last year's case count included asymptomatic and mild cases (and false positives) while this year's case count don't. Therefore, even in the absence of vaccinations, we should expect the reported case count to fall significantly.

***

I looked up what Cornell and Georgia Tech are doing in terms of testing. Cornell conducts weekly surveillance testing "regardless of vaccination status". This is excellent.

Georgia Tech makes participation in surveillance testing voluntary, which all but guarantees that their dataset is biased by preferentially selecting people who have experienced symptoms or severe illness, and they also say "You may participate in regular testing even if you have been fully vaccinated, but I especially encourage those who have not been vaccinated to get tested weekly," which means the symptomatic bias is more severe among the vaccinated subpopulation than the unvaccinated.