Big Data, Plainly Spoken (aka Numbers Rule Your World)

How NYC restaurants manage queues

On the news, I keep hearing that shipped vaccines are not getting into arms quickly enough in the U.S. I'm not sure if this is an accurate narrative as we have limited visibility into those operations. Shipped vaccines may exceed administered vaccine by design. One reason is the fixed capacities of trucks and storage bins.

Others surmise that some people in front of the priority queue are refusing to take the vaccine, causing those doses to go to waste. I don't know if this is true either, but in this post, let's assume this is a problem.

It makes me think about how some popular NYC restaurants (used to) manage their waiting lists, which are priority queues. When you arrive at the restaurant, you put your name down on the list. If a table opens up, the next name is called. If your name is called, and you are not present, you lose your spot.

Here's the key: at some restaurants, if you lose your spot, you start over at the end of the line. If you show up 30 minutes after your name is called, you can't go back to the front of the line.

***

This general idea is useful for managing priorities for vaccine distribution. It is also "fair" in the sense that if someone with high priority decides not to take advantage, then this spot can be passed to the next person in line but now there is a cost to the decision to opt out - the added delay.

Similarly, how do popular NYC restaurant handle no-shows? They may impose a penalty. At the time of reservation, they ask for a credit card number and disclose that a minimum charge will be assessed for no-shows.

A small number of places don't do advance bookings. You have to show up at particular times, and claim a spot. Then, you return when the table is ready. I'm sure the rate of no-shows goes way down. Admittedly, this policy has distractors - people who think their time is too valuable.

Late arrivals? You may end up at the back of the line. Some may allow a short grace period, say 10 minutes.

In all these situations, the restaurants make rules that influence our decision-making.

 

Real-life demonstration of the value of statistical controls

In an email conversation with me, a doctor who's making managerial decisions in a health-care system is concerned about whether we'll be able to see the effect of vaccination in the data on cases. There are at least two sources of variation that could pollute the data:

<1> the emergence of new variants, especially fast-spreading variants of the virus

<2> the natural boom-bust cycle (waves) of the pandemic

If the cases start falling, do we know for sure it is due to the vaccines? Or could it be a natural ebbing of the current wave? If the cases continue to grow, do we know the vaccines are ineffective, or just ineffective for a new variant?

Other combinations are also possible. The vaccines can be effective against the current variant and less effective against the fast-spreading variant, leading to either (a) a net acceleration or (b) a net deceleration of cases, depending on the strength of each factor.

***

Ouch, you say, so we can't know anything.

If we can't, it's because we choose not to. This is why statisticians pleaded that the pharmas be required to continue running the placebo groups in the ongoing vaccine trials. These placebo groups are statistical controls.

If there is a new variant of the virus, it will raise the case counts in the placebo and vaccine groups both. So if we look at the difference between the two groups, the effect of the new variant is cancelled out.

Similarly, if there is a natural ebbing of the pandemic, it will affect both placebo and vaccine groups. When we look at the difference between the groups, this effect will also be self-cancelling.

The placebo control group establishes the baseline used to judge the vaccinated group. Besides, I have mentioned frequently here that we should keep running the placebo groups in order to establish long-term efficacy and safety, and to gain confidence in subgroup analyses.

***

This self-cancellation is valid only if the placebo and vaccine treatments are randomized. This means any participant has equal chance of being in placebo and vaccine groups. If placebo participants are offered the chance to get vaccinated, then the randomized structure of the vaccine trial has been destroyed.

If that happens, we have to analyze the group comparison as an "observational study". We do some form of matching to create quasi-randomized comparison groups. However, there are some risks. For example, the self-selection bias may be so severe that we can't find comparable people. Everyone above 75 may decide to take the vaccine, which leaves us with nobody in that age group who's unvaccinated.

Statistical controls are really valuable, and solve many problems all at once! Unfortunately, in this case, I believe they are destroying the placebo groups, which means we will not obtain those insights, which sadly makes us wander forward somewhat blinded.

(It's not all or none either. Given what we learned about overall efficacy, I believe the placebo group can be reduced in size.)

 

P.S. If the placebo groups are intact, then I hope the pharmas are still passing data to the FDA. Assuming we have a way to detect the new variants effectively, we have direct evidence on whether the vaccines work with the new variants. The scientific value of the placebo groups goes way beyond the pre-planned analyses!

 

Know your data 28: Data breaches and tech contradictions

The Verge reports yet another data breach. This one likely affects few people but made the news because the "victim" - Ubiquiti - ironically sells technology that is designed to protect user privacy.

The notice that Ubiquiti sent to users reveals a host of contradictions in the tech industry that we have yet to confront head-on.

We recently became aware of unauthorized access to certain of our information technology systems hosted by a third party cloud provider.

It is widely believed that one of the benefits of "cloud services" is security. Anyone who has experience with accessing both (old-school) corporate networks and cloud services appreciates that cloud access is a lot easier, more convenient. More convenience is usually associated with less security. Installing 10 locks on your door is more secure than having just one lock but it slows you down each time you enter and exit.

We are not currently aware of evidence of access to any databases that host user data, but we cannot be certain that user data has not been exposed.

The tech company professed it doesn't even know what the perpetrator did after gaining unauthorized access. This claim implies that it could not trace the actions of a trespasser, even after his/her identity is revealed. If true, this itself is a security weakness.

At the same time, the tech industry is selling our identities and actions to marketers, not only mouse movements, clicks and purchases but also moods, personalities, and inclinations - the user profile is the product, we're often told, and that's why they can gift us services for free (or below cost). Those two things can't both be true! Either they log precise details of each user or they don't.

This [potentially exposed] data may include your name, email address, and the one-way encrypted password to your account... The data may also include your address and phone number if you have provided that to us.

I've always wondered why some companies require you to use a valid email address as your user name. It is not convenience since the email address almost surely takes longer to type than a pseudonym. It is certainly not security. And I'd like to draw your attention to the last sentence. For this vendor, it appears that some users volunteer their addresses and phone numbers.

Finally, we recommend that you enable two-factor authentication on your Ubiquiti accounts if you have not already done so.

Which users will have phone numbers on file? Those who set up two-factor authentication! Two-factor authentication is a great example of contradictions inherent in modern tech. It is billed as a security measure that prevents unauthorized access and protects user privacy; and yet, by activating it, users send even more private data to the vendor. Some companies have already been caught having harvested these phone numbers for marketing purposes. Unless the company has very strong data governance, the engineers and data teams accessing user data may not know that the phone numbers were obtained from two-factor authentication.

 

I'd let Ubiquiti have the last word. Here's the end of the notice:

We take the security of your information very seriously and appreciate your continued trust.

***

 

It's been a while since I wrote about data practices. I have a whole series of blog posts called "Know your data". You can still find them here.

 

 

One-dose vaccine strategy elevates PR over science

I fear that the U.K. policy of one-dose vaccines will backfire, and cause the pandemic to continue longer than necessary. Here are several reasons why.

Partial protection provides a convenient excuse for vaccinated people to do away with inconvenient mitigation measures

Assuming that the trial results hold, people receiving two vaccine shots have a very low chance of getting infected, and thus, it is acceptable that they return to normal living after vaccination. (I myself would not, since I think two doses are closer to 70 percent effective in reality, and we don't know that the vaccine stops asymptomatic spread, and so continuing to reduce contacts is advisable. Yet, under this scenario, one should not begrudge people who decide to return to normalcy.)

If people are given partial protection through just the first shot, it is irresponsible to return to normal living. Even the vaccinated must continue to wear masks, maintain social distancing, and stay at home as much as possible for the foreseeable future. Even now, it has proven impossible to enforce mitigation measures - at least not in the U.S. where I live. Vaccination presents a convenient justification for non-compliance.

This behavioral problem alone should have doomed the one-dose lobby.

One-dose treatment solves a problem that does not exist

The first weeks of vaccination has shown that the problem is demand not catching up to supply, not the other way round. We have more doses shipped than doses that are going into arms. If people don't like your cakes, you can't solve this problem by selling half-cakes!

The high efficacy of single doses is wishful thinking

Anyone who believes that the efficacy of the first dose is 90% lives in a fantasy world. Such a conclusion requires cherry-picking the best-case scenario, ignoring results that don't fit the theory, ignoring the margin of error, assuming that the second shot is essentially worthless, and assuming that the first shot is effective for the pertinent period of time without any boost from the second shot. The last two assumptions expose the one-dose strategy as a tautology - a conclusion that stems directly from one's assumption.

In a prior post, I explained how any guess of single-dose efficacy requires an assumption of (a) the efficacy of the second dose and (b) the duration of protection of the first dose, for neither of which do we have direct evidence coming from the vaccine trials.

It's a jigsaw puzzle, not a Barbie doll

People who propose the one-dose strategy are treating the clinical trial results like a Barbie doll. They are switching the green dress with the red skirt, or the high heels with flats. In fact, the toy is a jigsaw puzzle, in which each piece has its exact place, and cannot be haphazardly substituted.

For example, the headline vaccine efficacy numbers of 95% - recall the decimal obsession by Moderna - is based on a case definition in which they only count cases that occur at least 7 to 14 days after the second shot. You simply can't talk about that number in the same breath as efficacy for the first shot which requires counting cases after the first shot and before the second shot. They are not comparable numbers.

Use the scientific method properly

What supporters of the one-dose strategy should do is to lobby the FDA and/or pharmas to immediately start new clinical trials to test the one-dose strategy against the two-dose treatment.

From a scientific perspective, the biggest problem is post-hoc theorizing. Let's use stock market performance as an example. Do you feel like the business press are having it both ways? If stock prices went down yesterday, it's because the unemployment numbers were terrible. But if stock prices went up yesterday, it's because the terrible unemployment numbers were not as bad as imagined, or because businesses expected the job market to start improving, or because executives shrugged off the terrible unemployment numbers. They end up fitting a story to the observed data, as opposed to using the observed data to confirm a hypothesis.

Given the setup of the vaccine trials, before seeing these results, no one would have argued for one dose. The argument emerges after one sees the particular outcome from the trial. Trouble is, if the trial were run many times, would we have seen the same efficacy curve? (The variance across different trials already tells us the answer is NO.)

If the front part of the curve looks less favorable (say 30%), these same people would say we can't draw inferences about one dose (as opposed to one dose isn't going to work). In other words, this analysis contains an optimism bias, as does any theorizing after peering at the data.

***

Despite what I said above, I rate it as likely that certain countries will eventually follow the U.K. example and allow one-dose treatment. (As of now, the FDA has sensibly decided not to go down this path.)

Politicians score easy PR points by claiming they are helping "more people" when in reality, such a policy will lengthen the public-health crisis. Previously, I already noted that the PR agency of the UK government should be given an award.

 

P.S. Sadly, it appears that the incoming Biden administration is intent on elevating PR over science, as news has emerged that they will "release all available vaccines" and "not withhold second doses". This is euphemism for giving out one dose to everyone (as opposed to reserving the second doses for those who have gotten the first dose). For those "scientists" who are jumping behind this policy, I'd request that they answer the questions listed on my previous post. I can be persuaded that the one-dose strategy will lead to a better outcome but I'm having trouble understanding what set of assumptions lead to such a policy.

Added to the problems listed above, I think this latest news may persuade some of us to become vaccine skeptics. Those of us who believe the data show the vaccine needs two doses to work, and who do not get assurance that the second dose will be available in the right time frame, may decide to opt out of the vaccine.

 

P.P.S. In justifying this misguided policy, people are going to say "don't let the perfect be the enemy of the good". This is the cue that they are about to ignore the science. No one ever said the vaccine trial results are "perfect"; in fact, many of us believe they are less than perfect. I have written frequently about the shortcuts that were taken. We are not letting the perfect be the enemy of the good; we are opposing letting the unknown replace the good enough.

Other than some handwaving, I have not heard any description of what "the good" entails. If you can't quantify it, it's not science. If we are not going to argue about the science, then we must debate the assumptions, and evidence supporting such assumptions. Let's go!