Big Data, Plainly Spoken (aka Numbers Rule Your World)

Ethical dilemmas poll: results post

Update on 8-29-2023

Last week, I put up a poll that asks users what they would do if faced with some ethical dilemmas that arise on the business side of data science. I'm presenting the first set of results here. I'm guessing the responses have stabilized but will provide further updates below should the trends shift.

Case Study 1

Is it acceptable for BestHotels.com to present fake statistics to influence users?

A. Yes (8%)

B. No (87%)

C. I don't care (5%)

 

Case Study 2

Has AnyVoice done enough to mitigate such scams?

A. Yes (18%)

B. No (73%)

C. I don't care (9%)

 

Case Study 3

Should ZT&C launch the downgrade campaign?

A. Yes (27%)

B. No (42%)

C. I don't care (31%)

 

Case Study 4

Should banks be allowed to sell such customer data?

A. Yes (8%)

B. No (92%)

C. I don't care (0%)

 

Case Study 5

A. Yes (66%)

B. No (30%)

C. I don't care (2%)

 

***

What do I make of these results?

First the limitations. They reflect the biases of my blog's readers. Also, these issues are worthy of a substantive debate that goes beyond a poll. I chose scenarios that I think are not clear-cut, allowing arguments from either side; the poll format forces readers to pick a side, and it appears on average each question (except one) has a definitely more popular position.

The most lop-sided response came for Scenario #4 in which data scientists purchase bank account data to develop "personalized" pricing. Over 90% of respondents don't want banks to sell their data for this purpose. This is also the only scenario in which almost everyone had an opinion!

People in the business already know that this scenario has already happened, and I am not aware that any American government agency has bothered to ask citizens whether they condone this practice. One agency did intervene to stop a merger between Visa and one of the key vendors of such data but that action isn't a pushback against selling bank account data.

Food for thought: can these data science models be justified because they advance social justice? One could argue that the pricing tool would hike prices on those who can afford them while lowering prices on those who can't.

***

Scenario #3 is also an example of personalized pricing: here, the data scientists create models that determine whether broadband subscribers are paying too little or too much. The responses are roughly split in three: respondents who have no opinion; those who want to notify premium subscribers that they can save money by downgrading their plans; and those who think the company should let the subscribers stay in their plans.

In this case, the business has all the usage data it needs to build the model, unlike in Scenario #4 when the best data for the job must be acquired from third parties. In Scenario #4, the business adjusts pricing for everyone while the overall change leaves the business with more revenues. In Scenario #3, the business definitely takes the action that increases its revenues, and is debating the other action that suppresses revenues.

***

Here is the link to the poll.

 

Ethical dilemmas in data science: an update

In 2016, I posted about ethical dilemmas in the practice of data science, based on a talk I gave at the time. That post continues to be viewed each year so some readers must have found it useful.

The post covered several realistic case studies of ethical dilemmas I have seen in industry. Since ethics is subjective, I don't recommend any course of action. These case studies were designed to be useful in courses on data science ethics. It encourages students to debate both sides of the dilemma, and perhaps for the class to vote on a course of action.

Today's post is an update to the 2016 post. I present a set of new case studies. After each case study, I have activated a poll. You can all vote for one of the options, I will post results here once we have a good sample size.

***

***

***

***

***

***

Thank you for participating!

To see the original set of case studies, click here.

P.S. [8-21-2023] A twitter reader pointed out that the last question should be phrased as "Should" not "Does". I definitely meant "should". The question isn't about what the laws currently say but what we think they should say.

Is it worth it to flout parking rules?

Significance magazine (link) has a fun article about parking in front of fire hydrants in New York City. It's great data journalism.

The researchers conducted a "field experiment" in which they observed "six routes through Manhattan streets" on July 11, 2022 from 1-5 pm, during which they observed 138 violations. They then merged the hand-collected data with NYC's open data recording all tickets, matching license-plate numbers to learn that only 7 of those drivers have been served tickets.

Based on that, they deduced that most drivers who park in front of hydrants get away with the violation. They computed a detection rate of 5% (= 7/138). During that period of time, the traffic cops issued 423 tickets, and therefore, they deduced that the total violations to be 423/5% = 8,460. This number includes both the 423 tickets and a further 8,037 instances for which no tickets were issued.

They now model the cat-and-mouse game between the rule-bending drivers and the cops as an "exponential game". A timer is started each time some driver parks at a hydrant. The game is to see whether the cop shows up to flag the violation first, or the driver returns to pick up the car first. The model assumes that the time until the next check is exponentially distributed. An exponential distribution is where most values are small, but a few values can be very large. Such a distribution is fully described by its average value, which is estimated to be 6 hours. On average, a cop checks a given hydrant once every 6 hours. (That's the average interval, but each interval has a different value.)

They apply the same exponential model structure for the rule-bending drivers. In this case, the time until the driver returns to a given hydrant - i.e. the duration of violation -  is exponentially distributed, with a mean of 18 minutes (0.3 hours). Comparing the two exponential distributions, we can already see that drivers who leave their cars for a short period of time (say under 60 minutes) have a low chance of getting ticketed but if they park their cars illegally for hours at a time, their risk of detection is much higher. The authors did these calculations: the probability of detection is 16% for one hour, and increases to 99% for 24 hours.

Given the above model - and we assume also that the cops and the drivers act independently, the probability that the cop arrives before the driver returns is the key statistic we care about. That's the chance of the driver receiving a ticket, which as we know from above, is 5% based on the observed data. A consequence of the exponential game model is that this number is the average time until a cop arrives divided by the sum of the average times of a cop arriving and of the driver returning. Thus, 0.05 = 0.3/(0.3+6). [This verifies that the above calculation is correct.]

***

This type of model has been deployed to analyze gambling. If one buys lottery tickets every day, what is the chance that one hits the jackpot before one goes broke?

***

What's left unexplained is how they got the 6-hour estimate for the time until detection. This is a clever bit of work. Remember the researchers matched a subset of the observed violations to actual tickets issued. For these violations, the researchers determined the duration between observing the violation and issuing the ticket to be 6 hours on average.

But... that duration is counted from the moment the researchers found the illegally-parked car, not from the moment the driver parked illegally. What's unknown is the passage of time before the researchers documented the violation.

Conveniently, that unknown duration can be ignored. This convenience comes with using the exponential model. In the exponential world, the future does not depend on the past. Let's say the car has been illegally parked for 10 minutes already. How much time will elapse before a cop shows up? The answer is the same as before: it's exponentially distributed with average of 6 hours.

***

Based on the math above, the researchers found that illegally parking makes sense. The penalty for getting caught is $115. Returning within 5 minutes limits the expected penalty to under $2, which is similar to the cost of feeding a parking meter.

 

P.S. [7/10/2023] Corrected "flout" in the title of the post. Thanks to the reader who alerted me to this.

Were the U.S. polls accurate?

Andrew chimes in to defend the polling profession (link). He's responding to a piece in the London Review of Books that claims that the polls for the U.S. 2022 mid-term elections predicted a red wave that did not materialize.

He pointed to the Economist model that is an aggregation of polls that did not insert punditry type assumptions. This model is different from, say, Nate Silver's models for FiveThirtyEight, which incorporates some subjective judgement.

The poll aggregation model did remarkably well in predicting the overall outcomes in both the House and the Senate. In the House, the model predicted 225 Republican seats (208-244) compared to the eventual outcome of 222. In the Senate, the model predicted 51 Republican seats (46-55) compared to the eventual outcome of 49.

The poll aggregation model did not predict a red wave, as the interval estimates of the number of seats for both parties substantially overlap. In other words, it predicted a close election, which it turned out to be.

***
Andrew speculated that the LRB writer thought the polls predicted a red wave because he was "sticking to an existing narrative." The journalist regurgitates the "conventional wisdom" which happens to be wrong.

Andrew distinguishes between poll aggregation and poll forecasting, the latter combining poll data with punditry. He also focuses on the aggregate outcome (number of races predicted correctly), not the outcomes of individual races (e.g., did the polls get in right in state X?).

***

Andrew's post triggered a few thoughts of mine.

I dislike the conventional metrics of accuracy that treat every race as equally hard to predict. Given that most races in all U.S. elections are uncompetitive - as in, even a chimpanzee can predict the winner - a serious metric of accuracy should only consider competitive races. This issue is similar to that of grade inflation in education: the nominal scale (A-F) is misleading when all grades are compressed into the A-B range.

Worse than sticking to an existing narrative, the journalist is reinforcing a false narrative. I see the world as comprising two types of people in their relationship to data: data-first versus story-first (link). People who are story-first starts with the story and finds data that support the story. In the case of the 2022 elections, the Republican mouthpieces were everywhere pushing the narrative of the red wave. The polling data were an inconvenience that didn't matter.

Democrats can also be story-first. In fact, some Democrats might have allowed the false narrative to persist in the hope that fearing the red wave would drive their supporters to the polls.

When I came up with the data-first/story-first framework, I wanted something that is non-judgmental. Story-first is not always wrong; indeed, the data can be consistent with the story. In my experience, story-first is the default human instinct, even though being a data person, I wish the opposite. I'm definitely data-first but data-first isn't right for every decision either, especially when the data are inadequate, contradictory, or inappropriate.

The recent U.S. elections have been so close that it's virtually impossible to predict the result accurately. Statistical models are calibrated to say exactly that. The interval estimates shown above overlap substantially. This is a situation in which the statistician would be unwilling to put a strong bet on either side. For example, in a Bayesian sense, the Republicans were found to win control of the Senate in about 60% of simulations. This means the Democrats won control in the other 40% of simulations. Real life is just one simulation!

The model hitting the right ranges of results was a matter of skill in collecting and analyzing data. The model coming quite close to real life was a matter of luck. People find it frustrating when statisticians don't give a clear-cut answer. The real issue is not the statistics, it's the difference between story-first and data-first thinking. Story-first thinking is typically black and white; data-first thinking often involves shades of gray - as Andrew likes to say, we "embrace uncertainty".

 

P.S. [4/28/2023] Reader Mark P. has some further thoughts on this topic. His point about the inutility of election forecasting reminds me of this past post.

Strange PR for statistics

The University of Waterloo put out a press release to tout an extracurricular activity of a statistics professor. He devised a way to game a free prizes giveaway by the Canadian coffee chain Tim Hortons. (More coverage: here, here)

It's a bit weird because statistics is mostly absent from the strategy.

Here's the game. You buy a coffee and you get a chance to play a lucky draw for prizes. The advertised chance of winning a prize is supposedly 10 percent. Since the pandemic, all this happens online. There is a prize given out every 0.1 second.

The article described two methods of gaming this. The first method exploits the following operational detail: if there are no players in any interval, the prize is re-allocated to a future interval picked at random. Thus, as the day wears on, there are more and more intervals in which more than one prizes would be given away. So, the professor chose a time in the wee hours to play, and said he achieved an almost perfect win rate. He played 96 times, winning 94 prizes.

For this method, there were no data collected, so there could't be any statistical analysis. The article cited a one-time result and it's unclear how repeatable this strategy is. The win rate of 98% is based on a single sample. Perhaps the only analytical question is the best time to play. I'm not sure it's possible to game this precisely.

Also, to play 96 times, the professor had to have purchased 96 coffees. So he got 94 coffees paid for. Another amusing tidbit is that as a British guy, he usually drinks tea, not coffee. So he actually purchased 96 coffees that he wouldn't have otherwise... money spent to create an example for his intro stats class. (Actually, he won 67 coffees and 27 donuts. Assuming donuts are more expensive than coffees, the situation was a bit better than I portrayed. That said, in the crazy inflationary world we live in, a cappuccino in NYC now costs over $6 while a donut typically costs less.)

***
Then apparently, Tim Hortons changed the game a bit, or other people started to play the game the same way, so the professor had to change his strategy.

The second method has marginally a little bit of statistics in it. The coffee chain started to put up real-time statistics of win rates so at least now he could collect data. He didn't want to break the no scraping rule so he had students manually collect the data. As a result, the dataset has "holes". Nevertheless, the general strategy remains the same as the first one - finding a time interval that has the highest win rate.

It sounded like this was just a simple curve fitting exercise, interpolating between the observed time points. What surprised me was the precision with which he stated his results:

He found that the statistically best time to play Roll Up To Win was 3:16 a.m. ET. When he played his rolls at 3:16 a.m, he had an 80 per cent win rate... The worst time of day? 11:46 a.m. ET

Is this supposed to mean that 3:15 a.m. is not good enough? Does the statistical lesson also include prediction error bars?

***

It's also mildly amusing for them to admit that these methods can only win people free coffees - apparently there are bigger prizes, including a free car, but since those prizes are rare, they didn't have enough data to provide any strategy.

***

While this is a fun nerdy thing to do, the people involved in the press release seem to have a warped understanding of businesses. They seemed to think Tim Hortons's marketing team would be alarmed by such gaming, and would actively want to thwart them. In reality, the budget for the game had been set, and the number of prizes given out was fixed.

The strategy of playing late in the game late at night meant that the professor would have been winning surplus prizes that no one else claimed during waking hours. Plus, he was winning only free coffees, not the free car. So, I suspect Tim Hortons's staff weren't losing sleep over this.

P.S.[4/10/23] Corrected spelling of the coffee chain's name in a few places.

Strategies for validating observational studies 1/2

As mentioned before, the team who did the research tying Covid-19 vaccinations to car accidents convinced themselves that they have found a real effect. They attempted a variety of confirmatory analyses. In the prior post, I described how they conducted variants of the basic analysis by analyzing subgroups and by modifying the outcome metrics.

It's instructive to look at several other validation strategies and discuss their strengths and weaknesses.

***

Every observational study delivers the following table, usually called Table 1 because it's the first table in the paper:

Table 1 compares the characteristics of the treatment group and the control group. In the traffic accidents study, the two comparison groups are the vaccinated and the unvaccinated residents of Ontario (after exclusions). For a randomized clinical trial, in which vaccination status would be randomly assigned, the two columns of Table 1 should look statistically the same. For an observational study, in which vaccination status is self-selected (or influenced by biases such as public-health priorities), the differences between the two columns provide hints as to the magnitude and direction of biases.

The first thing to notice is vaccinated residents outnumbered unvaccinated about 5 times. The features are divided into three groups: demographics, prior medical conditions (e.g. diagnoses of cancer in the past year), and something labeled "general". The "general" collection includes clinical contacts, hospital admissions, and emergency room visits in the prior year. These are important variables because prior health status is a likely source of self-selection bias.

The researchers have this to say about Table 1:

The 2 groups spanned a diverse range of demographics, with comparable general health care utilization (Table 1). The largest relative differences were that those who had not received a COVID vaccine were more likely to be younger, living in a rural area, and below the middle socioeconomic quintile. Those who had not received a vaccine also were more likely to have a diagnosis of alcohol misuse or depression and less likely to have a diagnosis of sleep apnea, diabetes, cancer, or dementia. About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

This language is highly typical of how researchers opine on the likely sources of bias, i.e. those characteristics that differ between the comparison groups. Now pay attention to the last sentence: About 4% had a past COVID diagnosis, with no major imbalance between the 2 groups.

The actual data from Table 1 show 4.1% of vaccinated people had a documented Covid-19 infection in the past year, compared to 3.5% of unvaccinated. That's a relative ratio of 1.17. Given the large sample size (millions of people), a 0.6% difference in infection rates is highly significant! If we randomly selected 4/5 of the population to be vaccinated, the difference in infection rates between them and the unvaccinated would typically be much smaller than 0.6%.

So the claim of no major imbalance on Covid-19 prior infection is incorrect. It is also inconsistent because in the main analysis, based on 95% confidence intervals on relative risk, they declared they have found a significant correlation between vaccination status and traffic accidents. The same method would have found the difference between 3.5% and 4.1% prior infection rate to be highly significant. Indeed, if we apply the same method to all rows of Table 1, quite a few would also be found significantly different.

The prior-year emergency visits metric is of particular interest because the primary outcome being analyzed in this study was "emergency visits for individuals injured in traffic crashes". The unvaccinated group had a much higher prior rate of 26% compared to 20% for vaccinated, a relative ratio of almost 1.3, clearly significant. The analysts did not adjust for any of the variables in the general collection in their logistic regression, perhaps because they thought the two groups have "comparable general health care utilization," which as I just explained, is incorrect.

It's ok - in fact, expected - that the unvaccinated group would differ from the vaccinated group on some metrics. It's the opposite that should raise doubt. If the two columns are effectively equivalent, then the two comparison groups resemble those found in randomized trials. If we believe that, we're claiming that obvious sources of bias due to self-selection and public-health policy have not resulted in observed bias.

The biggest weakness of Table 1 analysis is missing variables. The comparisons are made only on the measured covariates. In the case of Covid-19, the measured covariates are whatever can be found in large government databases, collected for purposes other than conducting a study of traffic accidents and Covid-19 vaccination. The researchers said: "The available databases lacked information on driver skill, functional status, personality traits, traffic infractions, political affiliation, and self-identified ethnicity." Several of these variables are clearly pivotal for our present study.

In addition, Table 1 analysis addresses only biases acting independently. Some biases act in complex ways. When I analyzed prior emergency visits, the observed data appeared to indicate a healthy user bias in the vaccinated group. But this factor interacts with the age bias, which was also present. The vaccinated group was quite markedly older than the unvaccinated group. If older people are more likely to need emergency medical service than younger, then the emergency visit bias is even more pronounced since younger, unvaccinated people have a lower baseline emergency room usage than the older, vaccinated group. If younger people are more likely to need emergency medical service older, then age could explain some of the observed bias in emergency room usage. In other words, these factors should be interpreted simultaneously. In fact, any adjustment in the regression model should contain interaction effects, but this is rarely done.

***

The researchers also used "positive controls" and "negative controls" to convince themselves they have properly removed bias from this observational dataset. The idea is to process the data in the same way but substitute other metrics instead of the traffic accidents outcome. These other metrics are chosen because the researchers are 100% sure of the direction of the effect: for positive controls, they claim they knew ex-ante that the alternate outcomes are definitely affected by Covid-19 vaccination status while for negative controls, they claim they knew ex-ante that the alternate outcomes are definitely independent of Covid-19 vaccination status.

For negative control, they presented an analysis of constipation.

There were 2,985 cases of constipation during the month of August in the study population, and the relative risk was 1.03 meaning the unvaccinated group's risk was 3% above that of the vaccinated. This difference is not significant because the confidence interval of 0.94 to 1.14 crosses the 1.00 reference level. In other words, the data provided no evidence that unvaccinated people differ from vaccinated people in terms of experiencing constipation.

This type of validation analysis has many shortcomings. First, it's a single outcome. We don't know if this was pre-specified. Even if pre-specified, it could still show the desired result by chance since there are hypothetically hundreds or thousands of possible negative control candidates. If it's not pre-specified, it's hard to know if other candidates were tried and failed the test.

Perhaps because of reviewer feedback, the research team added several other negative controls, namely, fall, water craft, appendicitis and conjunctivitis. Perhaps because these were not in the original analyses, they did not discuss these other negative controls in the article.

I excerpted the table of results below.

The last column presents the confidence interval of the relative risk, and the second last column from the right shows the point estimate of the relative risk.

The second problem is inconclusiveness. Notice the point estimates of all five negative controls are tightly clustered between 0.90 and 1.14, and every confidence interval except "falls" shows a statistically insignificant difference between the vaccinated and unvaccinated groups. How should one interpret the result of "falls"? The researchers were silent on that one. When a negative control (constipation) shows insignificance, as expected, they were quick to say this finding validated the study conclusion but when a negative control shows a significant difference, it appeared that they explained it away, probably arguing it's a chance observation.

The third problem is over-confidence. How do they know that vaccination could have zero impact on constipation? I imagine this relies on some kind of structural biological argument, such as that mRNA does not act on certain bodily systems. Remember that statistical analyses describe correlations, not causation. We have to rule out ex-ante not just a direct causal relation between vaccination and constipation but any kind of correlation. That's actually a deep assumption, not as simple as it looks.

***

Positive controls are even harder to grapple with because we have to believe ex-ante that vaccination status will certainly result in some given outcome, in this case, "covid pneumonia".

The use of "covid pneumonia" implies the absolute belief that vaccinations reduce covid pneumonia in the analysis population. A few aborted clinical trials and many observational studies do not constitute incontrovertible evidence. Imagine if the data showed no significant difference in covid pneumonia between the unvaccinated and vaccinated. What would be the conclusion? That the methodology is invalidated? That the assumption of causation was incorrect?

These validation analyses using positive or negative controls have limited utility. They produce a situation in which the only possible conclusion is supporting the study - if these analyses do not support the study, we don't know how to interpret them so we throw them out and find another metric to act as control. (As a practical matter, no one would publish the paper if the authors include "failed" controls. See my prior post on the file drawer effect.)

***

The next tactic used for validation is propensity score matching. I'll discuss that in the next post.

Prior posts about this study: 1, 2, 3

 

 

Other numbers from the colonoscopy trial

You think I'm done with the colonoscopy trial - I'm not.

Previously, I discussed the core controversy, which is when investigators learned that the trial outcomes are not what they expect - in fact, colonoscopy effectiveness was far below what the medical profession has believed - they decided to cancel the primary endpoint and tout an "adjusted per protocol" analysis, which was designed with knowledge of the disappointing primary result.

Any per-protocol analysis is a partial lie. It takes a well-designed randomized clinical trial and dissolves the bedrock of random treatment assignment, effectively turning the RCT into an observational study. I call it a lie because people who do PP analysis describe their result as coming from an RCT. The authors of the colonoscopy study are proud that this is the first ever randomized controlled trial for this treatment. This is exactly what happened with all the Covid-19 vaccine trials too: they only analyzed people who took two doses (but if someone gets sick after the first dose, they are ineligible to get the second dose). If the analysis is PP, the study should be classified as an observational study even if the original design is an RCT.

In the last post, I did a deep dive into the two analyses that were presented in the NEJM paper to show why the adjusted PP analysis allowed the research team to print an outcome of 30% improvement on diagnosing colon cancer rather than 18% in an ITT analysis. I find the underlying mechanics rather implausible but as with any observational study, it's anybody's guess as to whether the statistical adjustments helped or hurt.

***

There are a few numbers reported in the study that haven't gotten as much attention in the media.

The most important is the unproven impact on all-cause mortality. Exactly 11% of the invited-to-screen group died within 10 years, and the same proportion died in the usual-care group from any cause. In other words, even if colon cancer was the cause of death on fewer death certificates in the screening group, the same number of death certificates would have been issued in both groups. This is probably because colon cancer kills at an advanced age, and people die from other (related or unrelated) causes if they didn't die from colon cancer. Thus, colonoscopy did nothing to change the life expectancy of the participants.

The researchers did report a large impact on colon cancer specific mortality in their "adjusted PP analysis" (50% compared to 10% in ITT) but they were silent on all-cause mortality. The mortality result was merely stated, and no further details were offered on compliers vs noncompliers, age groups, countries of origin, etc., thus no one can judge whether it makes sense.

Also, note that all rates are cumulative over 10 years of follow-up. Actually, that is also a partial lie. It's a "median" of 10 years of follow-up, meaning that only half of the participants need to have reached year 10. From Figure 1, I can see that about 66% have reached year 10, 85% have reached year 8, etc. The important thing to realize is that the further out in the case curve you go, the smaller the sample size that underlies the numbers (i.e. less reliable).

Another important number is the absolute risk difference. Thirty percent sounds like a large improvement as a relative ratio but it isn't because the baseline risk is very low. Cumulatively over a median 10-year follow-up period, the baseline risk was only about 1.22% (the risk of the usual-care group). Thus, 30% improvement on that is about 0.4%. That means out of 1000 people invited to screen, we can expect about 4 fewer diagnoses of colon cancer over roughly 10 years.

Flip that ratio around and we have what is called Number Needed to Treat (NNT). To prevent 1 colon cancer case, we need to screen 250. Using the ITT result of 18%, the NNT is 455.

A glaring gap is the lack of a cost-benefit analysis. A colonoscopy apparently costs $3000 in the U.S. So, out of 1,000 invitations, perhaps 500 do the screening, which cost $1.5 million, in order to lower diagnoses of colon cancer by 4. Thus, the cost of each avoided colon cancer case (not death, just case) is $375,000.

 

Why no serious researchers conduct "per protocol" analyses

All observational data must be adjusted in order to correct biases. But our science teems with irresponsible statistical adjustments - throwing a bunch of variables into a regression model, and declaring that all known biases have been satisfactorily resolved is all too common.

In the medical literature, publishing one's adjustment model is rarely done. I don't trust any adjusted models unless the researcher discloses the equations, the coefficients and goodness of fit statistics. Even better, the researcher should compare the adjusted to unadjusted results, note how the adjustments affected the outcome, and explain why these adjustments make sense.

Handwaving about the structure of the adjustment model, referencing other papers that supposedly use similar methods, etc. suggest that the researchers lack confidence in their methods.

In this blog post, I demonstrate how to evaluate statistical adjustments, using the recently released, highly controversial results from the colonoscopy clinical trial. (See my related writeup here.) It took a lot of work to put this together, because the disclosure of details in the paper is scarce (this is the norm rather than the exception, unfortunately.)

***

The colonoscopy trial was the first randomized clinical trial (RCT) conducted to measure the effect of colonoscopy screening on risks of diagnoses or deaths from colon cancer. The trial enrolled two groups of participants: those invited to screening, and the "usual care" group which means no screening (as is the practice in Poland, Norway and Sweden, where the trial took place).

For this post, I focus only on risk of colon cancer diagnoses (i.e. cases). That's because the disclosure on mortality outcomes is even worse, making it impossible to judge the adjustment procedure. The trial specifies an intent-to-screen analysis (more generally known as intent-to-treat, ITT), The ITT analysis shows that 0.93% of the invited group were diagnosed during the 10-year follow-up window, compared to 1.13% of the usual-care group. This result is described as a 18% reduction in risk of colon cancer, even though the difference in diagnosis rate is only 0.2%, that is to say, out of 1,000 people invited to colonoscopy, only two fewer cases will be detected relative to usual care.

As discussed in the previous blog, the medical profession was none too happy with this outcome because doctors, esp. American ones, have convinced themselves that colonoscopy could reduce both cases and deaths by 70% or more. Neither did the authors of the paper disclosing the trial results - in the section of the paper in which they discussed the findings, they did not mention the primary endpoint, but only talked about an "adjusted per-protocol" analysis, which moved the reported result from 18% reduction in risk to 31%. Still far from the 70% previously believed, but deemed more palatable.

The media then mis-reported this analysis as "per-protocol" analysis, dropping the all-important word "adjusted". By the end of this post, you'll understand why I say this.

***

Intent-to-treat vs per-protocol is a long-running debate but any serious researcher goes with the former, which is exactly what the designers of the colonoscopy trial did. Let's look at this debate using a marketing example.

Let's say Target is running a Black Friday promotion in which online customers get an extra 30% discount if they type the code SAVE30 when checking out. In order to measure the effectiveness of this promotion, Target data scientists design an A/B test in which shoppers are randomly divided into the exposed group and the non-exposed group upon visiting Target's website. They intend to analyze the results of this experiment by looking at metrics such as proportion of visitors who made a purchase, and average purchase amount, comparing the two groups. This analysis is known as an intent-to-treat (ITT) analysis.

Ouch! The ITT analysis showed that the additional sales generated by exposure to the SAVE30 promotion failed to cover the cost of giving out these extra discounts. Inevitably, there will be at least one business manager who complains that the wrong analysis was done. They argue that we should not include people who did not notice the promotion. Being exposed to the promotion just means Target shows notifications while the customers browse around but some fraction of customers did not pay attention. So, what do they think is the right analysis? We should only analyze those people for whom we know for sure they noticed the promotion - in other words, we only analyze those people who entered the promotion code on checkout (or other variants, such as only those who clicked on a promotion banner). This style of analysis is "per protocol".

Why would all serious researchers shy away from per-protocol analyses? In this example, everyone in the exposed group who subsequently entered the promotion code on checkout is someone who is surely, or almost surely, making a purchase. In the non-exposed group, there is no such thing as entering the promotion code, thus we effectively narrowed the exposed group to almost sure shoppers against the unexposed group that contains the average online shopper. Which group will show better metrics? Will the better results on the exposed group be due to the promotion or inherent differences in the group composition?

***

Back to the colonoscopy trial. In the per-protocol analysis, the data analyst threw out roughly half of the invited-to-screen group because they did not "comply". The reasoning is that they could not have benefited from the screening (read: SAVE30) and thus, they drag down the efficacy. But the "compliers" typically have different characteristics than the "non-compliers", and so per-protocol destroys the randomization aspect of the "randomized" controlled trial. In other words, the complier subset is different from the usual-care group not just in being screened but in unknowable dimensions. We've turned the RCT into an observational study.

People who advocate PP analysis say not to worry - they will throw a bunch of demographic variables into a regression model, and all biases will disappear. I find this quite fascinating. If such adjustment strategies indeed can cure all biases, then why would the community embrace RCTs as the gold standard? It would appear that observational data blended in a mixer known as regression modeling is equally as good as a randomized experiment!

In fact, the colonoscopy controversy clearly shows the opposite. Prior observational studies have consistently shown extremely positive results of 70% or so, and the first time a research team conducts an RCT, the outcome is below 20%. If anything, this proves that carefully analyzed observational studies cannot replace the gold-standard RCT.

***

Even if all biases are cured, and we are able to believe the PP analysis, the real-world results are guaranteed to disappoint.

Let's go back to the Target example. Let's say the PP analysis showed that customers who used the SAVE30 promotion were 80% more likely to complete the transaction than the customers who were not shown the promotion. So now, the head of marketing tells everyone if all shoppers are shown the promotion, then the experiment proves that they should see an 80% jump in number of transactions.

Of course, reality bites. The problem is that the 80% incremental sales apply only to the subset who used the promotion code. If only 10% of those exposed used the code during the A/B test, then only about 10% would contribute an 80% jump in sales. The other 90% are exposed but would not be using the code.

If half the trial population elect not to take the colonoscopy screening during the trial, it is likely the case that half the population would make this decision if colonoscopy was rolled out to all. The 30% reduction in colon cancer risk would still only apply to the subset choosing to be screened, and would not make a dent on those who chose to forego it.

***

It looks like I have to defer showing you my analysis till the next post. The plan is to compare the ITT analysis and the "adjusted PP analysis", pointing out how the adjustments affected the outcomes. Come back tomorrow.

(By the way, all Covid-19 vaccine trials were per-protocol analyses as well, even though such analyses are problematic.)

 

 

 

All evidence are not created equal and some evidence are more equal than others

A casualty of the Covid-19 emergency is scientific rigor, and in particular, certain long-established statistical principles in evaluating evidence, and specifically, that the weight of evidence depends on the method of research, and how data are collected and analysed.

The media, and the scientists they employ as talking heads, rarely identify the nature of the evidence behind health policies. On air, they make little or no distinction between results obtained from randomized clinical trials (RCTs), observational studies, surveys, lab experiments, animal experiments, etc. Like a drumbeat, they promote the impression that all evidence are created equal, despite their disparate origins.

***

While such an attitude towards scientific evidence is not new, it became normalized during the pandemic. A great illustration is found in this recent article reporting the results of a clinical trial on colonoscopy (screening) for colon cancer (link).

In the U.S., the standard of care calls for all men above 45 years old to get a colonoscopy every 10 years. Colonoscopies are both expensive and invasive. In some other countries, less invasive alternatives like stool testing are preferred.

American doctors have pointed to strong evidence that colonoscopies work. According to the article:

Past research always showed that colonoscopy could put a huge dent, on the order of 70%, in the incidence and mortality from colon cancer.

In other words, we can reduce colon cancer deaths by 70% - purely by screening, and presumably catching cancer in its early stage. A researcher was quoted saying the profession has expected colon cancer to become "extinct" if every man were to take colonoscopy.

***

Chances are men were not informed that all past evidence about this practice has come from observational studies only.

Observational studies involve comparing the outcomes of people who have taken colonoscopies and those who haven't. However, because people self-select into these procedures, a direct comparison of the two groups is biased. The people who choose to do it are likely to be different from those who don't, beyond the decision to be screened. Analyzing observational data involves finding ways to remove biases.

Take a current example. It is obvious that when comparing the Covid-19 outcomes of vaccinated and unvaccinated people, these two groups are not identical because self-selection introduces biases. There's age bias, political bias, etc. etc. It's challenging enough to enumerate all sources of biases. The hard nut to crack is to learn the magnitude of such bias. Even if we know older people are more likely to be vaccinated, and more likely to suffer more severe consequences from Covid-19, how do we know the degree of bias? If we don't know the size of the bias, how can we tell if we have adjusted too much or too little?

It's like asking the tailor to adjust the length of a pair of pants without telling her the height of the person wearing those pants. It's just too long, the tailor is told.

Back to the observational studies of colonoscopies. Doctors have relied on a series of peer-reviwed observational studies that converge on the conclusion that colonoscopies are extremely effective. What can go wrong?

***

The profession ignored the nature of the evidence, the deficiencies of observational studies.

Finally, some researchers in Norway had the good sense to check these results by running the gold standard of medical evidence, a randomized clinical trial. In these trials, patients are randomly selected to get invitations to take colonoscopies. The random assignment of treatment is an automatic adjustment for known and unknown biases. (Read this previous blog for a primer.)

And... after 10 years of observation, the RCT showed that colonoscopies has essentially zero effect on mortality from colon cancer, and only about 20% reduction in incidence, both metrics coming in far below 70%, the efficacy found in the observational studies. I'll dig deeper into the results in a future post. For this post, I take these findings at face value.

***

What gave me feelings when reading the Stat article is the response from various experts who effectively advocate trashing the RCT result in favor of the observational studies. This reaction is shocking given the RCT's status as the gold standard of medical evidence.

This RCT is not some summer project by a student intern. It includes 85,000 people, with data collected over 10 years. And yet, the article ends with a quote that says "colonoscopy is still king".

The negative reaction reflects the belief that all evidence are created equal, and illustrates the skepticism toward counter-intuitive scientific findings. It's not just in medicine. Anyone who have run experiments in industry has encountered this attitude.

The arguments for rejecting the inconvenient results can be classified into the following types:

(a) Some other related evidence exists that contradicts the current study - e.g. a randomized trial of a related procedure has previously shown better results

The problem with this type of argument is that it says nothing about the current study. A Chinese term for this strategy is "raising East to beat West". This strategy works under the doctrine that all evidence are created equal. The other study is on a different procedure and it is brought into this conversation purely because it obtained the more "acceptable" result. Imagine this: if that trial has produced worse outcomes than the current study, would the expert have mentioned it to the reporter?

If we allow prior experiments to overrule new experiments if the latter's conclusion differs from the former, then what's the point of running the new experiments?

(b) The treatment worked better for a subset of the test subjects - e.g. if we restrict the analysis to the older population, the positive effect of the treatment is higher

Whenever the overall results of an experiment disappoint, the sponsor of the experiment almost always asks for "deep dive" analyses, which typically means, hunting for a subset of the population for which the treatment made a difference. What's wrong with this? Imagine, what if the researcher discovered a subset of patients for which the effect of colonoscopy on cancer diagnosis was not 20% reduction but 5% reduction? This finding will be dismissed and ignored because it adds "nothing new" to the overall conclusion. The only additional findings that can get the attention of the sponsor are those that validate the desired outcome.

Besides, any such deep dive analysis runs into statistical problems. The sample size of the subset is usually too small, harming the reliability of the result. The filtering effect of statistical significance means any such finding is likely to be an outlier for the experiment.

Worse of all, the sponsor may take the positive outcome for the subset, and create the impression that it works for everyone. This actually happens a lot more frequently than one thinks. Take a Covid-19 vaccine study that uses matching to remove biases. While the matching procedure increases our confidence that the two groups being compared are similar, many records are dropped from the analysis because they are unmatchable. For example, because most older people got vaccinated, and many younger people did not, the matched subpopulation has a different age profile than the full population. While the comparison of the matched groups is not biased, the extrapolation from matched groups to the general population is biased. Nevertheless, experts used the matching study to proclaim that the vaccine is effective for all, rather than that the vaccine is effective for the matchable subpopulation.

(c) Our hypotheses are better than your data - e.g. because treatment methods are improving over time, we believe the benefit will increase over time, therefore in five years' time, the outcome will be better

Everyone is entitled to their hypotheses but only those running the experiment has submitted their hypothesis to testing with real-world data. The experts offer no data to support the alternative hypotheses. The experts are also ignoring hypotheses that reinforce the study's findings. Imagine this: what about the possibility that someone would develop a new screening method that might dominate colonoscopies in the next five years?

(d) The data should be analyzed in some other way - e.g. we should use a per-protocol analysis

This reaction makes a mockery of pre-registering the study methods. We face the same problem  of ex-post hunting for a specific outcome. If the per-protocol analysis had yielded similar or "worse" results as the intention-to-treat analysis, the same expert would not have raised this critique.

***

Imagine this: the RCT for colonoscopy has confirmed the profession's longstanding view that the procedure cuts cancer deaths by 70% or more. What would the article look like? Would there be a line-up of experts raising concerns that the RCT outcome might be wrong?

All four types of counter-arguments could still have been raised. There is likely going to be at least one randomized trial of a related procedure that showed much lower efficacy. If the analysts did a deep-dive analysis, at least one subset of the patients probably experienced much lower benefits than the average. Various hypotheses could be proposed that suggest the RCT over-estimated the efficacy. Some other way of analysing the data would have yielded lower efficacy.

Nevertheless, we wouldn't be hearing these arguments.

***

Lost in this debate is the real question raised by the RCT; in fact, it hasn't even been asked. So I'm asking it here: What went wrong in those prior observational studies to have over-estimated colonoscopy's effectiveness to such a large extent? How did the machinery that "adjusts" for biases fail?

 

 

 

 

Know your data 34: the surveillance industry

It feels cathartic to worry about the plight of foreign citizens subject to surveillance by modern-day technologies. Don't forget about what's happening at home. Motherboard describes a monitoring tool that schools are increasingly using to track students who request to leave classrooms (link).

As the author noted, this tool is similar in spirit to the tool that Amazon uses to measure how long their workers spend in toilets.

Students who want to leave the classroom submit a request through this app; teachers then approve the request. Because it's done through the app, the students are tracked - how long have they been away, where did they go, etc.

***

As with all such technologies, we should look at the bigger picture.

• What such technologies are capable of today, and in the near future
• Other than the advertised "benefits" of such technologies, what obvious harmful uses are enabled
• Other than those disclosed publicly, who else have access to the data, and why
• The right to be forgotten

Also, whether the technologies actually solve the problem. In the article, the vendor claims that the app can be used to stop gatherings of students at certain locations. How so? With or without this app, the teacher likely must show up at those locations to break up the gatherings. If without the app, the teacher cannot stop these gatherings; how does the app change that reality? Data provide information - humans still must make decisions and take actions.