Big Data, Plainly Spoken (aka Numbers Rule Your World)

Swear and gossip: answer to question 2

Today, I give my answer to Question #2 in the test presented a few days ago, here.

***

Question #2: Do the following headlines contradict or can both be true at once?

When we are presented with just the first statement that the most successful teams swear and gossip, we are tempted to draw a causal conclusion. We think swearing and gossiping is the cause of success. This motivates the advice given by the headline writer. (Okay, so the writer did not say out loud encourage your team members to swear and gossip more... but is there any other possible interpretation?)

If the cause--effect relationship holds, we may expect the least successful teams to not engage in swear and gossip, which means the truth of the first statement denies the truth of the second, and vice versa. (Even this is not certain since there are other factors affecting team success.)

If we are told both statements are true at the same time, I don't think any of us would conclude that swearing and gossiping determines team success. That's why it's important to know if one statement implies the truth/falsehood of the other.

Answer: Both statements can be true at once. One does not imply the other.

More precisely, one may imply the other but often doesn't. As Scott quipped on Twitter, it depends.

One approach is to lay out the 2x2 grid of all four possible states of being. Like this:

Each team is classified as either successful or not (based on some metric of success). Each team is also classified as either heavy or light on the swear-gossip axis. The sum of the four numbers is the total number of teams.

The statement that "the most successful teams swear and gossip" suggests that A is substantially larger than C, or that A is a large proportion of (A+C).

The statement that "the least successful teams swear and gossip" suggests that B is substantially larger than D, or that B is a large proportion of (B+D).

The question is whether the values of A and C constrain the values of B and D in such a way that if the first statement is true, then the second statement must be false. To invalidate this, we can show scenarios under which both statements are true.

Imagine we have 100 teams. There are only two polite teams that don't swear and gossip, split evenly between successful and not successful. Next, we have 50 teams in cell A (successful and swearing) and 48 teams in cell B (not successful and swearing). Now, among the successful teams, 50/51 swear and gossip while among the unsuccessful teams, 48/49 swear and gossip. Both statements are true. The first statement does not deny the second.

We have a whole class of scenarios that work similarly. The split between A and B can be 49/49, 48/50, 40/58, 55/43, etc.

***

For those who have been exposed to contingency tables like the one above, you may be tempted to fill in the four numbers by multiplying the probability of success and the probability of swearing and gossipping. When you do this, you're assuming independence.

Independence means that knowing whether team members swear and gossip provides no information on team success. This is the opposite of the study's intention. The researchers must believe that team success is dependent on the level of swearing and gossipping.

To model dependence, it's not sufficient to know just the probability of success and the probability of swearing and gossipping. For each level of swearing and gossipping, we assume a different rate of success.

Through this exercise, we learn that three of the four values (A, B, C, D) are free. The only constraint is that they must sum to the total number of teams. In other words, having specified A and C in the first statement, we have only nailed down the sum of B and D but the split between B and D is not constrained. Thus, both statements can be, and often are, true at once.

***

In Numbersense (link), I described a real-life example of this fallacy.

Years ago, the Gates Foundation was putting a lot of money behind the "small schools movement". This was based on some studies showing that the top X schools in district Y have a high proportion of small schools (with small average class size).

The situation can be shown in the following table:

Notice that this is the same table as above, with "small class size" substituting for "swear and gossip".

The Foundation was shown just the first column, from which they concluded that small class size was the cause of high test scores (A being a large proportion of A+C). The statistician, Howard Wainer, looked at the entire distribution and demonstrated that the bottom X schools also have a high proportion of small schools (B being a large proportion of B+D). This corresponds to the second column.

If shown both columns, you'd not conclude that class size drives test scores. If you only know the first column, you'd be tempted toward that conclusion.

If you only know the second column, you'd be tempted to draw the opposite conclusion that small class size depresses test scores!

That's why data analysts want to see all of the data. That's why biased datasets can be extremely misleading. You can make errors of direction, not just magnitude.

***

There is an underlying statistical reason that makes Wainer's discovery inevitable in many real-world datasets. Extreme values (highest or lowest test scores) are associated with small class sizes, because small samples have higher variability.

Imagine a large school with 10,000 seniors, with average GPA of 3.75. Imagine taking a random sample of 1,000 students, and reporting the sample average GPA. It's not going to be exactly 3.75, off by some small amount. Now take a sample of 500 students, and the average deviation of the sample mean from 3.75 will be higher. Now do 100 students. The smaller the sample size, the more likely the sample average runs further away from 3.75.

In the business teams example, I would not be surprised (if they did not control for team size) that the most successful (and least successful) teams are smaller teams.

This is an example of statistical gravity: if the dataset consists of teams of varying sizes, the large teams will be found in the middle of the distribution, and the smaller teams will be found towards the tails.

 

 

Swear and gossip: a test of your statistical acumen

Business Insider has been flogging this headline since late August, and it was still on the front page last week:

I don't have a premium subscription so I don't know what evidence they have for this. I'm imagining that someone collected data, zeroed in on the teams that are deemed successful, and computed the percentage of team members who "swear and gossip". How do they know they "swear and gossip"? I have no idea - the data may have come from one of those Big Brother surveillance technologies.

***

I love this headline because it can be turned into a test of one's statistical acumen.

Question #1: Are the following headlines identical (i.e. interchangeable)?

What's your first instinct? Now, think about it a bit, and see if you change your mind. I'll present some hints in the second half of this post, and the answer in a future post.

 

Question #2: Do the following headlines contradict or can both be true at once?

What's your answer? Come back later in the week to check what my answers are.

For those needing some hints, keep reading.

***

For Question #1, one perspective is whether the order of presentation matters. Is the relationship symmetric? Imagine throwing a dice twice, and getting 1 then 5 in that order. If the question is the sum of the values, then the order does not matter. One followed by 5 is the same as 5 following 1, both adding to 6, in the context of this question. However, if the question is the values are increasing, then order matters. One followed by 5 and 5 followed by 1 are evaluated differently.

One approach that helps with either question involves writing down the entire set of possibilities. When you have two binary variables, there are four possible states. Are the two headlines talking about the same things?

Identity requires that each statement implies the other. Check both directions. If it's not true, then find counter-examples in which one statement is true while the other one is false. Question #2 asks about non-contradiction which is less stringent than identity. Identical statements cannot contradict each other but non-identity does not imply contradiction.

Remember we are doing statistics, not pure math. Almost nothing in statistics apply universally. Think of counter-examples not as a single example but a class of examples sharing the same characteristics.

***

Feel free to use the comments to discuss with other readers.

 

The booster shot data reveal why RCTs are still the gold standard

Observational studies have some common characteristics that annoy the analyst. In the previous post about Pfizer's booster studies, I described a few.

• The sample contains few examples of the people to which you're generalizing your result (e.g. people 65 or older, high-risk individuals of any age).
• The measured outcome is not the obvious, direct outcome (e.g. clinical outcome of infection) but an indirect indicator (e.g. antibody levels).

These problems create the need to extrapolate results. Such extrapolation is usually supported by unverified assumptions (aka expert opinion). Of interest, the FDA allowed Pfizer to impose a causal assumption: that if the booster achieves similar levels of antibodies as the original 2nd dose, then the booster also produces similar clinical outcomes as the original two doses. See the previous post for why modifying an indirect outcome does not invariably lead to a change in the direct outcome. (This matter is yet another real-world example about correlation and causation.)

As I have explained often on this blog, making assumptions is not a sin. All statisticians make assumptions - those who claim they make no assumptions make assumptions! Let me give a quick example. Suppose a college runs a survey asking recent graduates about their post-graduation salaries. Only 50% of the graduates responded with a valid number. Should the college impute the salaries of the non-respondents? If I were the analyst, I would impute the salaries, baking in the assumption that those who don't respond are likely to be earning lower salaries (possibly even zero). However, some statisticians will argue against using imputation. They'll claim that one should just report the average of those who responded, because it is a horrible thing to modify the data - one should only correct egregious data entry mistakes but not otherwise touch the data. Are they making no assumptions? 

They are making a huge assumption: they assume that the non-respondents have the same average salary as the respondents. What is the basis for that assumption? There is none, other than the desire to not tamper with the dataset. In fact, this assumption is almost surely wrong. As I said in Numbersense (link),

Beware of those who don't tamper with the dataset!

***

In this post, I'll cover several additional extrapolations that were used in analyzing the booster data.

The key immunogenicity results all measure antibody levels against the original virus (so-called wild type) but as we've been constantly reminded, the only virus currently in circulation is said to be the Delta variant. How did Pfizer bridge this gap?

According to the FDA Briefing document (link to PDF), "Pfizer proposes to infer effectiveness of the booster dose against the Delta variant from exploratory descriptive analyses of 50% neutralizing antibody titers against this variant evaluated among subjects from the Phase 1 portion of the study." 

"Exploratory descriptive analysis" is not regarded as a serious form of scientific evidence - the rules have changed during this pandemic emergency. I pity the statisticians working on these studies. They were assigned an impossible task. Recall that the gold-standard randomized clinical trial (RCT) has been thrown overboard, replaced by "immunobridging" analysis. The antibody levels stimulated by the booster shot are deemed comparable to those registered (by other individuals) after their second shots, and then the "non-inferiority" in antibody levels is assumed to result in the "non-inferiority" in vaccine effectiveness (a clinical outcome not directly measured).

The Delta variant introduces another complication: as it was not present around the time of the original Pfizer adult trial, researchers do not know what the VE of the first two doses is against Delta. This breaks the causal assumption of Immunobridging.

We do not have even an approximation of the VE outcome against the Delta variant for the 300-odd people in the booster trial. So if one accepts that one can expose the old blood samples to the Delta variant in the lab, and show that the antibody levels are comparable, one has to make an even stronger assumption by claiming that those antibodies will result in the same VE as measured in the Pfizer adult trial.

This line of argument unfortunately results in a contradiction. The original vaccine either works just as well against Delta as it worked against the original strain; or it is less effective, which explains recent surge in cases. Only one of these scenarios can be true.

If the first scenario were true, then the immunobridging assumption holds as VE for Delta is the same as VE for the wild type, but in this scenario, the recent surge in cases has nothing to do with Delta, and there is no need for a booster - since VE is the same.

If the second scenario were true, then the booster may be necessary, but in this scenario, the immunobridging methodology is broken as the prior value of VE no longer applies.   

Since Pfizer applied for approval for the booster, it is assuming the second scenario. One might be tempted to prefer assumption #1 because it permits the immunobridging analysis. That is a popular justification you find in academic papers, in the same vein as "computational feasibility" or "tractability". It's risky to use this logic in studies with real-world consequences. Besides, it creates a contradiction.

Beyond the logic of the analysis method, notice that the Delta analysis is performed using only Phase 1 participants, so we are talking about 11 people between 18 and 55, and 12 people between 65 and 75 years old. I'm not sure why blood from the other 300 people cannot be similarly analyzed. Without those results, the immunobridging assumption is expanded yet again to generalize from the 23 people to the 300-plus people (then to the 20K-plus in the adult trial, and finally to the U.S. population.)

A proper RCT would have yielded a much cleaner analysis of the direct clinical outcome. 

***

Another missing piece of the puzzle is the provenance of the 300 or so Phase 2/3 trial participants who were "selected" to take part in the booster study. Nowhere in the FDA Briefing document can I find the selection mechanism. Usually, this means the selection was not random from the vaccine arm of the earlier trial. If it is not random, what are the criteria? 

The 300 people reduced to about 200 that ultimately were used in any of the primary calculations. That's a drop-off of a third of the starting population, which is a high drop-off rate. (Note: Table 3 claims the evaluable immunogenicity population is 268 but the results shown in Tables 4 and 5 unexpectedly have N=210 and N=179.)

Some of the reasons for exclusion are perplexing. Let's review a few of these reasons.

Six people refused the booster shot. There is an argument that they should be excluded as they did not get the treatment under study, and there is no blood to be analyzed. 

Fifteen were dropped because they did not have "at least 1 valid and determinate immunogenicity result within 28 to 42 days after the booster shot." This sets my alarm bells ringing because this exclusion can only be applied after the primary outcome of the study is measured. It's not clear what constitutes "valid" and "determinate". It's concerning to lose 5% of the study population by effectively saying we failed to obtain the primary endpoint.

Then, they dropped a further 30 people (10% of the study population) because their clinician decided that these people committed "protocol deviations" before the 3D+30 day evaluation time.

Last but not least, they also removed 34 people from the study population (15%) based on a key clinical endpoint. This exclusion criterion is described as "evidence of infection up to 1 month after booster dose". Notice it says "after" booster dose, not before. So, it appears that the following happened: they selected about 300 people to start this study regardless of their prior infection status, various people were excluded for the reasons described above, blood samples were collected, PCR tests were performed (on the day of the shot, and then when the participants self-reported symptoms), and if anyone gets sick within 30 days of receiving the booster shot, they were kicked out of the study.

Remember the design of this study. The indirect outcome of antibody levels is used to infer the direct outcome of infection. So when someone is dropped because of infection, we should infer that the dropped person has inadequate antibody levels. These deletions induce a bias in the primary endpoint, raising the average antibody levels of those who remain in the study.

Why is the case-counting window set to 30 days instead of the 7 days used for the 2nd dose? That's not explained in the Briefing document.  

***

I may come back to the safety data some day but the sample size is so small that only really loud signals can be heard. If no adverse effects are found, one can't conclude that there are no adverse effects; one can only say that there are no adverse effects that can be detected by this study design.

 

Reading the evidence for the booster shot

The FDA Advisory Panel met in mid September to discuss Pfizer's application for a booster shot. This led to a partial authorization (link). The key statement is:

Today, the U.S. Food and Drug Administration amended the emergency use authorization (EUA) for the Pfizer-BioNTech COVID-19 Vaccine to allow for use of a single booster dose, to be administered at least six months after completion of the primary series in:

• individuals 65 years of age and older;
• individuals 18 through 64 years of age at high risk of severe COVID-19; and 
• individuals 18 through 64 years of age whose frequent institutional or occupational exposure to SARS-CoV-2 puts them at high risk of serious complications of COVID-19 including severe COVID-19

In today's post, I read the FDA's Briefing Document (link to PDF), released as part of the September meeting in order to understand the science behind this decision. This briefing represents the FDA's interpretation of data submitted by Pfizer.

***

The most striking discovery is that the FDA no longer requires a randomized clinical trial (RCT) to authorize vaccines. As explained in my research methods talk (link), RCTs - long considered the gold standard for data-driven medical science - have six key components: random sample of underlying population, randomization of treatment, double-blinding, control, placebo, pre-specification.

With the FDA's blessing, Pfizer offered data from two observational studies; these studies do not have most of those key components that make RCTs the standard.

The first study is a Phase 1 trial, consisting of a total of 23 people. All of them were given a third (i.e. booster) shot about 7 to 9 months after their 2nd shot.

The second study involved about 300 participants who enrolled in the big Pfizer adult trial. (Here is the link to results from that trial.) All of them were given a third (i.e. booster) shot, about 6 months after their 2nd shot.

In all, the FDA looked at two studies that enrolled fewer than 350 people, without a control group, without placebo, and not blinded. The investigators wished to generalize from about 300 to about 300 million people in the U.S. I'm unsure as to why they can't find 300 other participants from the big trial so that they could randomize treatment into extra shot versus placebo.

The FDA advisory committee apparently were not fully convinced by the data but they nevertheless authorized the booster shot for 65 plus and others at high risk. This is a curious decision.

The second study with 300 participants contained zero people aged 65 and above. The age cutoff was 55 years old. So the only older participants came from the first study, and there were N=12 such people labelled as 65 to 85 years old. Table 2 in the Briefing document revealed that despite the labeling, the 12 people's ages ranged from 65 to 75 years old so there was no data submitted for people above 75 years old. This is an example of tokenism, which I described here.

Further, the Phase 1 trial excluded all people who are at high risk of Covid-19 infection, and so all the evidence we have for 65 plus are from 12 healthy, low-risk individuals aged 65 to 75. Table 2 confirms that this dozen did not represent the diversity of Americans - not surprising given the tiny sample size. They are 100% white, 0% obese, and 0% with comorbidities.

The evidence for the second recommendation for 18-64 who are at high risk of severe Covid-19 is similarly stretched. Presumably, indicators of risk of severe disease include comorbidities and minorities. The second study only included people aged 18-55 so the only data available for the 55-64 age group came from the first study - however, none of these people are at high risk of Covid-19, let alone severe Covid-19, by virtue of the Phase 1 study's exclusion criterion.

As for those in the 18-55 age range, the second study enrolled 306, of which only 55 have comorbidities, only 28 are black Americans, and only 2 are native Americans. In short, the FDA authorization is based on observational studies that have really small sample sizes, which get attenuated further when the recommendations are restricted to subgroups. The advisors extrapolated these results to subgroups that have few or zero data based on their expert opinion. (I'm not denigrating expertise, just pointing out how it is applied.)

***

The FDA has months ago told the pharmaceutical companies they no longer have to run RCTs. They said that future "modified" vaccines can be approved based on "immunobridging" analysis. In this Briefing Document, the FDA further extends the rule to booster vaccines that are not modified from the original, which is what the Pfizer booster shot is.

The logic of immunobridging is to show "non-inferiority" in antibody levels after taking the booster shot compared to after taking the two-dose series. Instead of comparing vaccinated to placebo as in an RCT, such analyses compare people who took the booster shot to people who took the first two shots. If the antibody levels are non-inferior - as defined by statistical criteria, then it is presumed that the booster shot will provide the same level of protection as the original two doses, i.e. the famous 95% vaccine efficacy.

Pfizer submitted data that showed the antibody levels after the booster shot were non-inferior, using thresholds to which the FDA has agreed.

***

What is "immunobridging"? This bridge is what I'd call a "causal assumption".

You see, neither Pfizer nor the FDA can define the relationship between antibody levels and getting infected. During the Advisory Committee meeting, a Pfizer scientist acknowledged, "We actually looked at our breakthrough cases in our placebo-controlled phase 3 study, and have compared the antibody titers where we had the opportunity in individuals that got the disease versus the ones that didn’t, and we were also unable to really come up with an antibody threshold. So I think there’s probably a much more complex story and not easily just addressed with neutralizing antibodies."

What they were hoping for is a simple rule that says antibody levels > X means vaccinated person is protected. The "much more complex story" implies that the rule must involve not just antibody levels but many other factors.

We therefore know that antibody levels is necessary but not sufficient to induce immunity. The causal model requires other variables, some of which may be unmeasured. The chosen solution is to make a causal assumption: just assume that antibody levels is sufficient for protection. This is the meaning of the following sentence in the summary section of the FDA Briefing Document (link to PDF):

Effectiveness of the booster dose against the reference strain is being inferred based on immunobridging to the 2-dose primary series, as assessed by SARS- CoV-2 neutralizing antibody titers elicited by the vaccine.

Two steps are involved here: the 95% VE estimated in the adult trial is assumed to be directly caused by the vaccine producing enough antibodies to fight off infection - while assuming no other factors play a role; then, the booster shot is assumed to produce 95% VE if it induces "non-inferior" antibody levels compared to the original two shots.

***

The next question is how long will immunity coming from the booster shot last. The short answer is 1-2 months; it could be more but we don't have the data since this is another interim analysis with a brief follow-up period.

The second study has the bulk of the people (about 300). The follow-up periods range from 1.1 to 2.8 months post booster shot. But we should subtract 1 month because there is a case-counting window that would start 30 days after the booster shot. Effectively, the period of time during which these 300 people can accumulate cases is as low as 3 days, and up to 1.8 months (54 days). (The first study with 23 participants has the same maximum follow-up time.)

Where does the 30 days come from? This is analogous to the 14-day period in which cases are removed if they occur in vaccinated people. The antibody measurements were taken 30 days after the booster shot and compared to measurements taken at 2D+30 days in the previous adult trial. Since we are inferring protection based on antibody levels, we can't infer protection for any time prior to 3D+30 days by measuring antibodies at 3D+30 days. Effectively, for the third dose, the case-counting window has been pushed from 14 days after the shot to 30 days after the shot. Anyone who gets sick within one month after getting the booster will not be considered "breakthrough".

***

This post just scratches the surface of all the issues that arise when we don't have RCTs to evaluate medical interventions. These observational studies are small, do not have randomly selected participants, do not have a placebo/holdout arm, and are not blinded. As a result, any observed outcome could have been caused by a host of factors, including unknown or unmeasured variables. This forces data analysts to make assumptions. The quality of the analysis depends heavily on the quality of these assumptions.