Big Data, Plainly Spoken (aka Numbers Rule Your World)

Beneath the surface of recent Google, Elon, FCC news

Several recent news items caught my eye.

First, there's been endless drama coming out of Twitter since Elon Musk acquired it for an ungodly sum. One of his latest bombshells is to promise an email drop that "makes public all internal discussions about the decision to censor @NYPost's story on Hunter Biden's laptop" in order to "restore public trust" (link). It's just one example of how corporate executives can weaponize data.

Second, Wired reported that Google handed over the personally identifiable information (PII) and detailed location data of at least 1,500 people to FBI investigators (link), through "geofence warrants", which is a dragnet. Geofencing is a favorite technique also of marketers for finding targets to deliver their marketing messages: the marketer draws an area on the map, and the data providers hand over contact information of every cellphone that is known to have entered the geofenced area. In this case, the FBI requested information on anyone who entered the area around the Capitol with their cellphones on January 6, 2020.

Third, the FCC blocks several leading Chinese technology manufacturers from the U.S. market (link) in the name of protecting national security. This action is taken based on the "possibility" of "espionage, IP theft, blackail, foreign influence campaigns, and other nefarious activities".

***

These stories are all related to our data. What happens to our data. Who has our data. What can people do with our data. What have they done with our data.

Americans have allowed corporations to be custodians of our private data. This was originally cast as a grand bargain, in which corporations get our data for free while they offer us some service of value for free. Think back to the beginnings of Google and Facebook... scratch that, not the humble beginnings when the entrepreneurs lured us in by offering their services for free without a quid quo pro. It was after these companies went public when they realized they must make money, and eventually, every one of these outfits has become a hoarder of private data, which are sold to the highest bidders. Every new digital product now is conceived as a way to augment the profiles of the users. This includes paid services (e.g. Amazon and any number of subscription services) which also collect the same types of data, and trade them in an obscure market... without the grand bargain.

***

Many - most, perhaps - of us have decided to trust Google, Twitter and similar companies with our data. Some of us prefer to trust our corporations rather than our governments. But can we really trust corporations to act in our best interests?

What is our best interest? It depends. If you're one of those people who entered the Capitol on January 6, your best interest is for Google not to send your location data to the FBI. If you happen to be around the Capitol on January 6 but not involved in the election politics, you don't want investigators to be probing you just because you were in the neighborhood.

If you are one of the anti-government protestors in China, Iran, Russia, etc., your best interest is for Google not to send your location data to the investigators. Now, is it any different if you're protesting against a Western democratic government?

Closer to home, if you happen to live in one of those states that have banned abortion, and also encourage your neighbors to report on you to authorities, your best interest is for Google not to sell your location data to the highest bidder. If your neighbors are pro-life activists, their best interest is for Google to release geofenced location data around known abortion clinics so you can fulfil your civic duty to report on people and doctors who skirted the abortion ban.

Do we really believe that corporations are making these decisions based on our individual desires, or based on what their lawyers think is the best course of action for the corporate interest?

If the government entity is collecting our data, there are constitutional rights that govern what this entity can do. There are Freedom of Information Act (FOIA) provisions that citizens can use to discover if laws have been broken. There are no FOIA requirements that regulate corporate behavior. There is no punishment for companies issuing false statements when not under oath.

***

Elon Musk's behavior provides a glimpse at the answer. His threat to release a load of internal emails relating to Twitter's moderation process is a direct threat to weaponize data. He's not the first corporate executive to do this sort of thing: remember how a Uber executive casually told a reporter that she's being tracked during a time when another Uber executive threatened to dig dirt on journalists who wrote critical articles about Uber (link).

If Twitter will release internal emails to make a political point, it certainly has the means to release user data if an executive finds a reason to do so. This is selective, targeted disclosure of potentially private data. Musk is not releasing all emails about all banned accounts, but emails about one particular political figure that happens to belong to a political party he is on the record of opposing.

There is no way to verify that all emails related to that one situation would be released. No one outside of Twitter has access to this data. If there are any emails that could place Twitter in legal jeopardy, what's the chance that the corporate lawyers would allow them to be released? Would these lawyers play government and redact the documents?

Remember, besides the messages that Twitter users post publicly, there is a whole hidden universe of direct messages which are private conversations between people. Activists in authoritarian countries are known to use platforms like Twitter to hide communications from prying eyes.

Musk's announcement is much more than about Hunter Biden's laptop. It is an admission that private messages are not really private. They are in the hands of the companies which own these platforms. It only takes one CEO or executive with a grudge to weaponize someone's data. (Junior staff can also wage their own small-scale vendettas as many people have access to data as part of their work.)

***

The banning of Chinese tech companies is based on the possibility that the Chinese government may get access to private data from Chinese companies, and the possibility that such data would be used to harm people.

In Twitter's story, the CEO articulated a motive and a plan to selectively release data. He just hasn't pushed the button yet.

By contrast, Google pushed the button when they transferred data to the FBI. It's not anonymized data, either. It's personally identifiable information plus detailed maps of these people's locations. (While the Wired article singled out Google, it's likely that other big tech companies, like Microsoft and Apple, and telecoms companies, may have done something similar.)

 

P.S. [12/4/2022] In the evening of the day I put up this post and the next day, Elon Musk pushed the button and released the data so we now have two instances in which prominent U.S. corporations released data with the express purpose of harming the people who generated the data, plus an allegation that Chinese companies may do similar things.

 

 

Self-checkout takes another hit

I recently wrote about a common misconception about self-checkout technology in supermarkets. Many people experience shorter wait times, and attribute the gain to the technology - when in fact the higher efficiency is merely because the supermarket has many self-checkout counters open compared to human workers. See here for more.

Self-checkout technology has another downside, which has surfaced with the tide of the economic slowdown. Target - one of the top retailers in the U.S. - just reported disappointing business results, blamed on "organized retail crime" (stealing from stores). In this Yahoo article, they explained a major reason why retail crime has increased - self-checkout technology:

our stressed social contract may be capping how far we can push this people-light, technology-heavy model. Last month Wegman's ended its scan-and-go shopping app. Why? Shrinkage, of course.

[Note: Wegman's is another major supermarket chain.]

It's telling how scared these journalists are of the tech industry; I guess they are employed by a tech company (Yahoo) so it's understandable. The above sentences were tagged on at the end of a paragraph, enclosed in parentheses, and drenched in technical jargon when the message can be simply stated as:

self-checkout technology has opened the floodgates to cheaters and thieves who steal from major retailers, causing substantial losses to retailers

This "shrinkage" is very damaging to a company's finances. The retailer paid for the cost of the stolen products but received $0 for them. It's a "total loss". This isn't counting any expenditure the retailers spent on, e.g., more technology to prevent or detect stealing, or efforts to catch thieves.

Is this really surprising? Not if you live in a city with a subway system which - maybe 20 years ago - started replacing humans with ticket machines (e.g., New York CIty). Which of these scenarios result in more fare evasion?

(A) each entrance has staffers in front of the turnstiles
(B) most entrances have no staff on duty since passengers buy tickets from machines

No behavioral economist is needed to figure this one out! In fact, there have been quite a few reports on self-checkout theft, e.g. this one.

We can be sure that Target executives are being marketed more technology - for detecting self-checkout theft. One idea might be to use video surveillance to find acts of stealing, face recognition to identify thieves, and cellphone tracking data to locate suspects.

We already know such a solution doesn't work. Think about what happens if one's iPhone got stolen. If Find My Iphone is turned on, one can literally see where the thief is, and track his/her movements. But...Apple isn't going to send someone to retrieve the phone for us, nor, as far as I know, will the police do so.

 

 

 

Other numbers from the colonoscopy trial

You think I'm done with the colonoscopy trial - I'm not.

Previously, I discussed the core controversy, which is when investigators learned that the trial outcomes are not what they expect - in fact, colonoscopy effectiveness was far below what the medical profession has believed - they decided to cancel the primary endpoint and tout an "adjusted per protocol" analysis, which was designed with knowledge of the disappointing primary result.

Any per-protocol analysis is a partial lie. It takes a well-designed randomized clinical trial and dissolves the bedrock of random treatment assignment, effectively turning the RCT into an observational study. I call it a lie because people who do PP analysis describe their result as coming from an RCT. The authors of the colonoscopy study are proud that this is the first ever randomized controlled trial for this treatment. This is exactly what happened with all the Covid-19 vaccine trials too: they only analyzed people who took two doses (but if someone gets sick after the first dose, they are ineligible to get the second dose). If the analysis is PP, the study should be classified as an observational study even if the original design is an RCT.

In the last post, I did a deep dive into the two analyses that were presented in the NEJM paper to show why the adjusted PP analysis allowed the research team to print an outcome of 30% improvement on diagnosing colon cancer rather than 18% in an ITT analysis. I find the underlying mechanics rather implausible but as with any observational study, it's anybody's guess as to whether the statistical adjustments helped or hurt.

***

There are a few numbers reported in the study that haven't gotten as much attention in the media.

The most important is the unproven impact on all-cause mortality. Exactly 11% of the invited-to-screen group died within 10 years, and the same proportion died in the usual-care group from any cause. In other words, even if colon cancer was the cause of death on fewer death certificates in the screening group, the same number of death certificates would have been issued in both groups. This is probably because colon cancer kills at an advanced age, and people die from other (related or unrelated) causes if they didn't die from colon cancer. Thus, colonoscopy did nothing to change the life expectancy of the participants.

The researchers did report a large impact on colon cancer specific mortality in their "adjusted PP analysis" (50% compared to 10% in ITT) but they were silent on all-cause mortality. The mortality result was merely stated, and no further details were offered on compliers vs noncompliers, age groups, countries of origin, etc., thus no one can judge whether it makes sense.

Also, note that all rates are cumulative over 10 years of follow-up. Actually, that is also a partial lie. It's a "median" of 10 years of follow-up, meaning that only half of the participants need to have reached year 10. From Figure 1, I can see that about 66% have reached year 10, 85% have reached year 8, etc. The important thing to realize is that the further out in the case curve you go, the smaller the sample size that underlies the numbers (i.e. less reliable).

Another important number is the absolute risk difference. Thirty percent sounds like a large improvement as a relative ratio but it isn't because the baseline risk is very low. Cumulatively over a median 10-year follow-up period, the baseline risk was only about 1.22% (the risk of the usual-care group). Thus, 30% improvement on that is about 0.4%. That means out of 1000 people invited to screen, we can expect about 4 fewer diagnoses of colon cancer over roughly 10 years.

Flip that ratio around and we have what is called Number Needed to Treat (NNT). To prevent 1 colon cancer case, we need to screen 250. Using the ITT result of 18%, the NNT is 455.

A glaring gap is the lack of a cost-benefit analysis. A colonoscopy apparently costs $3000 in the U.S. So, out of 1,000 invitations, perhaps 500 do the screening, which cost $1.5 million, in order to lower diagnoses of colon cancer by 4. Thus, the cost of each avoided colon cancer case (not death, just case) is $375,000.

 

With adjustments comes responsibility

Yesterday, I laid out the framework for today's post, in which I compare intent-to-treat (ITT) analysis and per-protocol (PP) analysis of the colonoscopy trial. All the data I cite below are pulled from this paper and its supplement (link).

***

The outcome metric of the clinical trial is the risk of colon cancer (diagnosis) over a 10-year follow-up window (a final analysis will be performed after 15 years). Risk is based on the relative ratio of diagnosis rates between the treatment and usual-care (control) group. In the ITT analysis, the treatment group consists of everyone who's been randomized to receive invitations to colonoscopy while in the PP analysis, the treatment group consists of the "compliers" only, those who responded to the invitation and underwent colonoscopy. Because compliers self-select to accept screening, they cannot be treated as a random subset of the invited group, and thus, the researchers spun up a regression model used to "remove biases".

In the chart below, I present the results of the ITT analysis from the paper, which is pre-specified for this trial. The trial was run in three countries but Poland (64%) and Norway (31%) contributed 95% of the participants.

 

Immediately, two features of this data come to life. Firstly, the case rate in Poland (0.88%) is much lower than that in Norway (1.35%). The relative risk ratio of 35% is higher than the relative ratio between the treatment and control groups (18%). (In other words, variability between countries is larger than variability between treatment and control.) Secondly, the observed reduction in case rate among Norwegian participants was 24% while that in Poland was 16%. Since Poland comprises almost 61% of the overall population, the reported aggregate effectiveness is just north of Poland's value.

The next chart shows what happens when the researchers switched to an "adjusted per-protocol" analysis. They broke up the invited group into compliers and non-compliers. The paper reported diagnosis metrics for Poland, Norway and the aggregate. (They did not disclose mortality metrics, only reporting the relative ratio, and thus no one can validate that part of the analysis. Nor did they break out Sweden's data.)

In Norway, the paper printed the compliers' case rate to be 0.86%, which is 30% below the average of the invited group. (It is not clear from the paper how much of this shift is due to adjustment and how much is due to filtering out noncompliers. It's not even clear whether this rate is computed from data or estimated from a model.) In any case, the treatment group's case rate was marked down from 1.2% to 0.86%, and correspondingly, the effectiveness metric in Norway leaped from 24% to 45%!

In Poland, a completely different picture emerges. The compliers are said to have a case rate of 0.85%, almost the same as the 0.83% aggregate rate of the invited group. This means that the effectiveness metric in Poland has barely shifted compared to the ITT analysis. It's still about 15%.

From this, we learn that the change in analysis strategy embeds the following key assumption:

(A) Compliers are very different from noncompliers in Norway but not in Poland

Such an assumption might be justifiable; I am in no position to judge. Hold on tight, as there are more assumptions to be unearthed.

Not explicitly stated in the paper is the fact that the case rate for non-compliers must have been altered in the other direction: since in Norway, compliers form 40% of the invited group, and noncompliers 60%, we can back out the case rate of the non-compliers - implied by the adjusted per-protocol analysis - which is 1.77%. This observation reveals two other embedded assumptions:

(B) The diagnosis rate of colon cancer for Norwegians who refused to be screened when invited  (1.77%) is higher than that for people not invited to screening and also not screened (1.57%). This requires explanation since people were randomly selected to receive invitiations

(C) The implied gap between the case rates of the compliers and noncompliers in Norway is a whopping ~1%, which is very wide

 

In Poland, two-thirds of those invited did not comply. Using this information, we can back out the case rate of the noncompliers, and it's 0.82%, actually slightly below the aggregate value of 0.83% for the entire invited group. This leads to further observations:

(D) There is almost no difference in diagnosis rates in Poland between those who complied and those who didn't comply with colonoscopy screening. This is particularly jarring considering (C) above.

(E) Colonoscopy is 3 times more effective in Norway than in Poland. In Norway, it brought the case rate down from 1.57% (usual care) to 0.86% (compliers) which is about the same level as compliers in Poland - despite the control group in Poland coming in much lower at 0.99% than the control in Norway.

***

The researchers decided to do an "adjusted per-protocol analysis" which means they did two things: they filtered noncompliers from the treatment group, and they made adjustments using a regression model that includes variables like country of origin, age, etc.

Since the paper does not disclose the model details, we have to look at the model outputs to infer what happened. What happened is listed above as (A)-(E). These are all changes to the data that ultimately caused the estimated effectiveness to shift from 18% to 31%.

***

What I want to show is that current reporting practices for observational studies in medical journals is wholly inadequate. It should be much easier for readers to comprehend what adjustment models are doing. Much more space should be devoted to justifying the effects of adjustments on the conclusions. Any of the above assumptions can make sense, but there is nothing in the paper to back them up.

The models used to adjust data should be disclosed in full. Tables showing covariate balance between groups should be standard, for every pair of subgroups that feature in any of the metrics included in the publication. For example, age is clearly an important confounder in this study, since colon cancer tends to appear in older people but there is nothing in the paper that shows age distribution between the compliers subset and the usual-care group.