Big Data, Plainly Spoken (aka Numbers Rule Your World)

How the U.K. justified its vaccination strategy

You have read my arguments against one-dose vaccine schedules. What is the argument on the other side?

The only technical justification I can find on how the U.K. justified its decision to deviate from the vaccine dosage schedule that has tested in the clinical trials is this document issued by the Joint Committee on Vaccination and Immunisation, citing an analysis by Public Health England (link, Appendix A).

The entire argument hinges on the claim that "short term vaccine efficacy from the first dose of Pfizer-BioNTech vaccine is calculated at around 90%".

Wait - if you've been reading my prior posts, you'd ask: wasn't the vaccine efficacy around 50% right before the second shot? I've been showing this chart, which is derived from the cumulative-cases chart published by Pfizer:

The British experts acknowledged this much: "Published efficacy between dose 1 and 2 of the Pfizer-BioNTech vaccine was 52.4% (95% CI 29.5 to 68.4%)."

So, the question is how did they make magic? How did the 50% move up to 90%? Notice that this grade inflation moved the value way above the upper bound of the 95% confidence interval.

***

The short answer is they cherry-picked the data. They re-computed the efficacy by counting only cases that occurred between Day 15 and 21. This has the effect of pushing up the VE because most of the cases on the vaccinated arm occurred on the front part of this curve.

Deviating from the results of the vaccine trial is to allow assumptions and gut feelings to modify the science. I am not a data purist, and believe there are some situations in which incorporating assumptions and guts improve the analysis. I want you to decide for yourself whether this re-calculation is justified. So I'll provide some notes on the issues you should be thinking about.

A key claim is that any vaccine takes time to generate an immune response and so it's "unfair" to start counting from the beginning. Note that this modification literally turns positive cases into negatives because the number of participants in the trial (the denominator) is not changed.

Because vaccine efficacy is computed from comparing the case rate of the vaccinated arm with that of the placebo arm, the trial design has a built-in control for exactly the problem being raised. If the vaccine were completely useless until day 10, then we'd have counted about the same number of cases up to day 10 on either arm.

How do they know whether it takes 5 or 7 or 10 or 15 days for the first dose to take effect? How did they draw the line at Day 15?

From the document: "Figure 3 [the cumulative cases chart] clearly shows that from approximately 10 days after the first dose the cumulative incidence in the vaccine and Placebo groups diverge. It would therefore be appropriate to calculate the VE of a single dose in a period after this 10 days."

Day 10 slipped to Day 15 in the next sentence: "A reasonable interval to use for post first dose VE would therefore be from >14 days to the time of the second dose..."

Also worth considering is the tiny sample size underlying an analysis at Day 21. Recall that the trial was stopped when a certain number of prespecified cases was reached. That minimum was not reached by Day 21. This leads to hilariously big error bars. The error bar around that 90% number is 52% to 97%. In plain language, the data was consistent with any VE between those two bounds. Think about that. (By contrast, the proper analysis that used all of the data and placed VE at 95% with median two-month observation yielded an error bar of 90% to 98%.)

***

As a reminder, the entire argument is based on the supposition that with just one dose, the "short-term efficacy" is 90%. What does "short-term" mean?

That 90% number is computed based on what happened between days 15 and 21 in the trial. Thus, short term is 7 days. Beyond Day 21, everyone in the trial received a second shot so it is no longer possible to separate the effects of the first and second doses.

Accepting the first shot really reduces the case rate by 90%, and the sum of two shots reduces the case rate by 95%, we know for sure that the 5% difference between those two values is not statistically significant. (The error bar for the 90% estimate is roughly 50% to 100%.)

In other words, if the above were true, then the second dose would not generate a statistically significant increase in efficacy above that afforded by the first dose. In other, other words, the second dose is useless.

Believing the U.K.'s calculation of first-dose efficacy is tantamount to believing that the second dose is useless. If the second dose is useless, no one, including me, could object to giving twice the people the first shot. The question is whether that strategy works if the first shot is 50% effective - which is the "published efficacy".

***

Finally, what is the efficacy of the first dose after day 21?

The U.K. experts now make this argument: they claim that the second shot would have no effect on the first 7 days, therefore one can look at Days 21-28 as if the second shot were not administered. Thus, the period of protection of the first dose is not just the 7 days (Day 15-21) but 14 days (Day 15-28).

How should we think about the effect of the second dose? In a simplistic additive model, we assume the two doses have independent effects which can be summed to get the overall effect. Most vaccine researchers believe that the second "booster" shot reinforces the work of the first shot. In this case, a model with an interaction effect is more appropriate. This means that the combined effect of two shots can be larger than the sum of the individual effects.

The interaction effect may be even more complex. The effect of the first dose may be waning, while the second dose reverses the decline in immune response.

In summary, it appears that two further assumptions have been made implicitly. First, that the effect of the two doses are independent and additive. Second, beyond day 28, the VE will hold steady. This implies that the second dose has no material effect for about 12 weeks (which is claimed as the maximum recommended delay between doses).

 

P.S. [1/27/2021] It seems like after I wrote this post, a second appendix (B) was added to the UK Government page. This explains how they took the vaccine efficacies and projected averted deaths. On a quick glance, I don't see any details of the mathematics. In particular, I like to know if this model incorporates lockdowns and personal measures, and how vaccinated people will behave. As indicated above, if the first dose of Pfizer is assumed to be 90 percent efficacious, which is statistically indistinguishable from 95%, I can't imagine any model would prefer two doses.

 

 

Actions have consequences: the messy aftermath of one-dose Pfizer

Credit: Johannes Krupinski

I spent a good chunk of my career designing, running and analyzing industrial experiments, and as policymakers stray further away from the underlying science, I have a bad feeling about the rollout of the Covid-19 vaccines

At issue is a push to deviate from the treatment protocol that produced the highly promising clinical trial results. During the clinical trial (by Pfizer), the vaccinated participants were given one dose of the vaccine on day 1, and a second dose on day 21, the vaccine was then given one week to take effect, so that the 95% efficacy was computed by counting cases from day 28 (i.e. nullifying cases between day 1 and 28). Moderna and AstraZeneca had a similar treatment protocol, all specifying two doses, but the number of days separating the doses, and the waiting period for the effect to take hold depended on the protocol.

When the outcome of a carefully-designed experiment returns positively, the straightforward next step is to roll out the treatment "per protocol". Any deviation from the protocol requires assumptions (coming from intuition, experience, and gut feelings) modifying the science. The point of running a randomized controlled trial is to follow the science rather than one's guts.

***
The specific alterations to the treatment protocol are as follows:

• the elimination of the second dose
• extending the timing of the second dose (if a supply shortage were to materialize, the delay of the second dose may become infinite, folding this into the first item)
• the size of the first dose (in the case of the AstraZeneca trial, they had initially trumpeted a half-dose followed by full-dose treatment, which has been quietly forgotten after the U.K. government decided to authorize a one-dose treatment. Does anyone know if the single dose is a half dose or a full dose?)
• mixing and matching different vaccines (such as Moderna and Pfizer)
• the success metric now based on (very small?) immunological samples and not based on cases confirmed by PCR testing

So far, the U.K. is the only country that adopts most of these as official policy while the U.S. and others currently claim they allow them for "exceptional cases". The U.S. announced that second doses are no longer reserved for those who have taken their first shots, increasing the chance that people would not receive their second doses at the prescribed time, if ever.

***

A fundamental best practice of running statistical experiments on random samples of a population is that once the winning formula is rolled out to the entire population, the scientists should look at the real-world data and confirm that the experimental results hold.

This post-market validation is hard even if properly done. That's because on rollout, everyone is eligible for the treatment, and those who have received the vaccine up to the time of analysis do not form a random sample of the entire population. So any difference between the vaccinated and unvaccinated groups may not be a pure effect of vaccination. (This difference is why we conduct randomzied controlled trials, which allow scientists to isolate causes.)

The action of the U.K. government (and others who may follow suit) has severely hampered any post-market validation. It is almost impossible to compare real-world evidence with the experimental result, because most people are not even getting the scientifically-proven treatment per protocol!

Even if the rollout was perfect, the real-world outcomes are likely to deviate from the experimental findings. Statistical science gives us confidence that any such deviation is immaterial.

With the altered protocol, the real-world data come from a variety of dosing schedules - some with one dose, some with two, some with two doses 21 days apart, some with two doses 10 weeks apart, etc.

What happens when the aggregate real-world outcome falls below expectation? How will scientists figure out what are the reasons for the under-performance?

First, the scientists can't even say what the expected performance is because the experiment yielded data on just one dosing schedule, which could be the least used in the rollout.

Second, there may be several simultaneous changes to the dosing schedule. For example, one might be comparing someone with 1 dose of Pfizer on day 1 with another who got 1 dose of Pfizer on day 1 and 1 dose of Moderna on day 36. Is the difference of outcomes due to mixing and matching, or number of doses, or the gap between doses? How do we learn which factors are contributors, which are not, and the relative importance of these factors?

Third, many post-market validation studies are doomed by focusing exclusively on factors that the investigators can control, such as the factors listed above. It's possible that the policymakers are correct - that none of the dosing changes would affect the vaccine efficacy. In that case, an observed difference in outcomes is due to other factors, such as viral escape by new variants, as-yet unclear duration of protection, compliance to mitigation measures after vaccination, which people choose to take the vaccine, etc. Notably, these are factors outside the control of scientists.

Why do we need answers for those questions? If the rollout does not meet expectations, we need to correct the course. To rectify the problems, we need to know what they are. If we can't even diagnose the potential problems, as will be the case here, we will be swinging in the dark.

***

Actions have consequences. The decision to deviate from the treatment protocols that delivered the promising vaccine trial results makes it very challenging, if not impossible, to measure the impact of these vaccines. One of the key lessons of managing this pandemic so far is that good data drive good decisions, and bad data drive bad decisions. Unfortunately, policymakers have signed up for bad data, so no one should be surprised if future policies turn out badly.

 

P.S. On Twitter, some are offering this UK government document as evidence that the scientists have a plan for post-market validation. I recommend reading this document to learn about what appropriate validation steps would have been in normal circumstances when the rollout of the treatment is per protocol. In the vaccine efficacy section, it addresses unanswered questions from the trials such as the duration of protection, subgroup analyses, other outcome metrics, etc. I didn't find anything in that document that estimates how deviations from the treatment protocol affect outcomes.

The business manager's guide to the one-dose, two-dose vaccine debate

The current debate about whether the Pfizer and Moderna vaccine trial results can be used to justify a single-dose treatment reminds me of something I've encountered frequently in the business world. This different context might help some of you grasp the key issues in the debate.

Assume we are running a food box subscription company, e.g. Blue Apron. We like to hold on to as many subscribers as possible. When a subscription comes up for renewal, our marketing team sends a sequence of two emails to entice them to renew. The success metric is lower number of deactivations within 24 days of the renewal date. The first email goes out on the renewal date, with the second email delivered one week later. As the following response curve shows, about half of the eventual cancellations happen within three days after the first email, and deactivations slowed to a crawl soon after the second email is delivered.

1

We ran a "holdout" experiment (a randomized controlled test), with some percentage of customers withheld from the email campaign completely. As shown below, if the customers do not receive emails (red line), then the total number of deactivations is higher.

The data scientists who analyzed this experiment concluded that the two-touch email campaign reduced deactivations from 20% to 12% by Day 24. They also offered the following chart of relative efficiency, which is based on the ratio of the deactivation rates:

The emails suppressed the deactivation rate by 40% by the 24th day after the renewal date. The effect accumulated over time. 

The data scientists' opinion was immediately buried by boisterous chatter from both sides.

***

On one side is the finance team. Pointing to the deactivation curve in the first chart, the CFO pushed to eliminate the second email. The first email did all the heavy lifting, and we save about half the campaign cost by removing the second email. [By the way, this is analogous to skipping the second dose of the Pfizer or Moderna vaccine.]

On the other side is the marketing team. The CMO alleges that the data scientists have undervalued the email campaign. People don't always read emails the day they are sent, so it's unfair to judge their emails from that day. Since it takes five days for people to take action, the test should be re-evaluated from Day 6.

As the chart shows, by starting the analysis on Day 6 rather than Day 0, the relative efficiency is now 50%, which is 10% higher than reported by the data science team. [This is analogous to measuring vaccine efficacy from 7-14 days after the second dose, compared to measuring VE from the day after the first dose.]

***

Rebuttal time. 

The marketers object vehemently to the finance team's proposal to trim the campaign to one email. By design, the first email sets the stage, and the second email closes the sale. If the second email were dropped, we would lose all the benefits beyond day 5.

Both sides have in fact speculated on how the deactivation curve would appear should the second email be cut. The marketers attribute everything beyond day 7 to the second email while the finance team attributes nothing to it.

When the scientists were able to speak up, they confess that they don't have any data to support one side or the other. Most likely, the second email has a partial impact, neither full nor zero. This means the true curve for a single email lies between the two blue lines on the above chart. The scientists suggest looking at intermediate metrics, such as how many times and when users clicked on or interacted with each email, which offer circumstantial evidence if subscription decisions beyond day 7 are partly driven by the second email. [This is the debate about one or two doses of the vaccine.]

It's the financial analysts' turn to play opposition, and they dislike leaving the first 5 days out of the evaluation. It's misleading to claim that the emails reduced deactivation rate by 50%. It's only 50% if applied to the list of subscribers starting on Day 6. But about half of the customer attrition happens before Day 6. An improvement of this new metric could not touch any of the deactivations prior to Day 6, by definition. [In the same way, the frequently reported 94% vaccine efficacy ignores any case that arises between the two shots, and only counts cases starting 7-14 days after the second shot. We cannot measure the impact of the vaccine by applying 94% to the number of vaccinated people.]

In the example I concocted, most cancellations occur within the first 7 days and so a metric that starts counting from day 6 ensures that our actions miss the bulk of our target audience, and focus on a minority. 

***

The CEO calls time-out. S/he asks the data scientists why the experiment did not contain a third test cell that sends customers just the first email and not the second. The scientists defer to the business teams. They say the third cell was dropped from the test design for expediency. Adding a third cell would have extended the test for another month, and we simply could not afford the wait. They took this action because in the last metric review meeting, the CEO was pounding the table asking for warp-speed actions to avert the declining subscriber numbers.

P.S. I have simplified the test design for these marketing experiments. A so-called "drip" campaign typically contains multiple touch points across multiple channels, which means each experiment involves many test cells. The challenge of these designs is to arrange the test cells so that we can get a clean read of whether a specific touch point on a specific channel affects the performance metric.

[1/22/2021: Early signs coming from Israel, where the vaccination campaign has gone the fastest, appear to support my opposition to single-doses. The Guardian has a write-up about their disappointment. Now, because these analyses are performed using observed data, not coming from a randomized experiment, there is always the possibility that the number of doses is not the reason for the continued rise in cases.

Vaccines free you from the herd

source: Regione Umbria

The vaccination campaign in the U.S. got off to a slow start. One culprit is the initial obsession over "herd immunity". I wrote back in October that the focus on herd immunity is misguided:

Herd immunity draws attention to non-cooperative behavior away from the scientific achievements of vaccines. The people who stand to benefit from herd immunity don't even have immunity, as they will get sick if they come in contact with the virus. That's why I don't like the term herd immunity.

This commentary applies even if herd immunity is gained through mass vaccination. The only people to benefit from "herd immunity" are the 25% who are not immunized. A fraction of these may have conditions that make it risky to take a vaccine but most of this group will comprise of vaccine skeptics. Despite being unvaccinated, they are expected to be protected through non-exposure when a sufficient proportion of their fellow citizens has received their shots.

By contrast, someone who is vaccinated is protected. This is a direct personal benefit stemming from a personal decision. The argument for vaccination is simple: protect yourself, so you can return to normal living. You can protect yourself, regardless of what the herd does. You don't vaccinate to attain herd immunity, you vaccinate to free yourself from the herd behavior!

***

The focus on "herd immunity" reflects a technocratic bias. The discovery of herd immunity tied to a critical level of immunization is one of the shiny objects of epidemiological modeling. The mathematics is quite beautiful and simple. It's great pedagogical material. Herd immunity is also a natural goal for public health policy - the goal is better stated as eradication of the virus. If one achieves "herd immunity," the virus has been defeated.

It's not a step function - we don't need to reach the herd immunity level for the pandemic to slow down. Every vaccination cuts some transmission chains so the benefit accumulates as more people get shots.

Telling people to vaccinate for the sake of herd immunity is like telling them to do something for the benefit of others. Unfortunately, many Americans don't do things for others, as we have observed. In this case, the others are mostly people who will not reciprocate.

Tell people to vaccinate in order to sweep away the fear, and return to normal.

 

[P.S. 1/20/21 A reader rightly pointed out to me that even after vaccination, one should still wear masks, keep a distance, etc. since the vaccine is not 100% effective. I made a comment about this in a previous post - here. I said I would personally continue to take precautions, but would understand if others decide not to after vaccination. I'm making a suggestion to public health officials who believe the real-world vaccine efficacy of two doses is 90 percent or thereabouts. If it turns out that VE is lower than expected, we should be worried about too many of us returning to normal too soon.]

[P.S. 1/27/21 The Atlantic just published a piece that makes a similar point, that it may be counterproductive to tell people nothing can change even after getting vaccinated. The article also exposes a lingering problem - why are we still relying on survey responses to project how people will behave when the tech industry has all the surveillance data we need to know how people are behaving? Another important point is that the argument to allow returning to normal is founded on the belief that these vaccines are highly effective, which can be invalidated by real-world evidence.]

 

 

Record deaths every day

The situation in the U.S. reminds me of this post from the time when Italy was slammed by the pandemic's first wave. Every day brings news of a new record of daily deaths.

Deaths being a lagging indicator, they don't tell us about the current level of alarm but what was happening weeks ago. Thus, record deaths are not news. Record deaths follow from record hospitalizations, which follow from record cases.

Deaths from coronavirus is an example of what I've been calling "statistical gravity". People who decided to travel and assemble during the holidays set off a chain of events: cases rise, hospitalizations rise and inevitably deaths rise.

Here is a picture of hospitalizations in the U.S. since September (from the valuable CovidTracking Project):

To establish the cadence, we can look for the inflection point, in mid October, when the cases started to surge in this current wave. This maps to the inflection point in the curve of deaths (see below) in early November, so the time lag between hospitalizations and deaths may be around 2 weeks.

With a two-week lag, the deaths we're witnessing now are correlated with the hospitalizations from two weeks ago (post Christmas). The deaths in the next two weeks are predictable by looking at what happened to hospitalizations post Christmas. In the first chart, we learned that hospitalizations went up by 10 percent in about two weeks so we expect deaths to rise by another 10 percent in the next two weeks.

Yes, the U.S. death tolls will be breaking records for at least two more weeks. The numbers will likely continue rising beyond those two weeks, as a similar analysis looking at the growth in cases will show. When will deaths decline? Watch when cases stop growing.

 

 

How NYC restaurants manage queues

On the news, I keep hearing that shipped vaccines are not getting into arms quickly enough in the U.S. I'm not sure if this is an accurate narrative as we have limited visibility into those operations. Shipped vaccines may exceed administered vaccine by design. One reason is the fixed capacities of trucks and storage bins.

Others surmise that some people in front of the priority queue are refusing to take the vaccine, causing those doses to go to waste. I don't know if this is true either, but in this post, let's assume this is a problem.

It makes me think about how some popular NYC restaurants (used to) manage their waiting lists, which are priority queues. When you arrive at the restaurant, you put your name down on the list. If a table opens up, the next name is called. If your name is called, and you are not present, you lose your spot.

Here's the key: at some restaurants, if you lose your spot, you start over at the end of the line. If you show up 30 minutes after your name is called, you can't go back to the front of the line.

***

This general idea is useful for managing priorities for vaccine distribution. It is also "fair" in the sense that if someone with high priority decides not to take advantage, then this spot can be passed to the next person in line but now there is a cost to the decision to opt out - the added delay.

Similarly, how do popular NYC restaurant handle no-shows? They may impose a penalty. At the time of reservation, they ask for a credit card number and disclose that a minimum charge will be assessed for no-shows.

A small number of places don't do advance bookings. You have to show up at particular times, and claim a spot. Then, you return when the table is ready. I'm sure the rate of no-shows goes way down. Admittedly, this policy has distractors - people who think their time is too valuable.

Late arrivals? You may end up at the back of the line. Some may allow a short grace period, say 10 minutes.

In all these situations, the restaurants make rules that influence our decision-making.

 

Real-life demonstration of the value of statistical controls

In an email conversation with me, a doctor who's making managerial decisions in a health-care system is concerned about whether we'll be able to see the effect of vaccination in the data on cases. There are at least two sources of variation that could pollute the data:

<1> the emergence of new variants, especially fast-spreading variants of the virus

<2> the natural boom-bust cycle (waves) of the pandemic

If the cases start falling, do we know for sure it is due to the vaccines? Or could it be a natural ebbing of the current wave? If the cases continue to grow, do we know the vaccines are ineffective, or just ineffective for a new variant?

Other combinations are also possible. The vaccines can be effective against the current variant and less effective against the fast-spreading variant, leading to either (a) a net acceleration or (b) a net deceleration of cases, depending on the strength of each factor.

***

Ouch, you say, so we can't know anything.

If we can't, it's because we choose not to. This is why statisticians pleaded that the pharmas be required to continue running the placebo groups in the ongoing vaccine trials. These placebo groups are statistical controls.

If there is a new variant of the virus, it will raise the case counts in the placebo and vaccine groups both. So if we look at the difference between the two groups, the effect of the new variant is cancelled out.

Similarly, if there is a natural ebbing of the pandemic, it will affect both placebo and vaccine groups. When we look at the difference between the groups, this effect will also be self-cancelling.

The placebo control group establishes the baseline used to judge the vaccinated group. Besides, I have mentioned frequently here that we should keep running the placebo groups in order to establish long-term efficacy and safety, and to gain confidence in subgroup analyses.

***

This self-cancellation is valid only if the placebo and vaccine treatments are randomized. This means any participant has equal chance of being in placebo and vaccine groups. If placebo participants are offered the chance to get vaccinated, then the randomized structure of the vaccine trial has been destroyed.

If that happens, we have to analyze the group comparison as an "observational study". We do some form of matching to create quasi-randomized comparison groups. However, there are some risks. For example, the self-selection bias may be so severe that we can't find comparable people. Everyone above 75 may decide to take the vaccine, which leaves us with nobody in that age group who's unvaccinated.

Statistical controls are really valuable, and solve many problems all at once! Unfortunately, in this case, I believe they are destroying the placebo groups, which means we will not obtain those insights, which sadly makes us wander forward somewhat blinded.

(It's not all or none either. Given what we learned about overall efficacy, I believe the placebo group can be reduced in size.)

 

P.S. If the placebo groups are intact, then I hope the pharmas are still passing data to the FDA. Assuming we have a way to detect the new variants effectively, we have direct evidence on whether the vaccines work with the new variants. The scientific value of the placebo groups goes way beyond the pre-planned analyses!

 

Know your data 28: Data breaches and tech contradictions

The Verge reports yet another data breach. This one likely affects few people but made the news because the "victim" - Ubiquiti - ironically sells technology that is designed to protect user privacy.

The notice that Ubiquiti sent to users reveals a host of contradictions in the tech industry that we have yet to confront head-on.

We recently became aware of unauthorized access to certain of our information technology systems hosted by a third party cloud provider.

It is widely believed that one of the benefits of "cloud services" is security. Anyone who has experience with accessing both (old-school) corporate networks and cloud services appreciates that cloud access is a lot easier, more convenient. More convenience is usually associated with less security. Installing 10 locks on your door is more secure than having just one lock but it slows you down each time you enter and exit.

We are not currently aware of evidence of access to any databases that host user data, but we cannot be certain that user data has not been exposed.

The tech company professed it doesn't even know what the perpetrator did after gaining unauthorized access. This claim implies that it could not trace the actions of a trespasser, even after his/her identity is revealed. If true, this itself is a security weakness.

At the same time, the tech industry is selling our identities and actions to marketers, not only mouse movements, clicks and purchases but also moods, personalities, and inclinations - the user profile is the product, we're often told, and that's why they can gift us services for free (or below cost). Those two things can't both be true! Either they log precise details of each user or they don't.

This [potentially exposed] data may include your name, email address, and the one-way encrypted password to your account... The data may also include your address and phone number if you have provided that to us.

I've always wondered why some companies require you to use a valid email address as your user name. It is not convenience since the email address almost surely takes longer to type than a pseudonym. It is certainly not security. And I'd like to draw your attention to the last sentence. For this vendor, it appears that some users volunteer their addresses and phone numbers.

Finally, we recommend that you enable two-factor authentication on your Ubiquiti accounts if you have not already done so.

Which users will have phone numbers on file? Those who set up two-factor authentication! Two-factor authentication is a great example of contradictions inherent in modern tech. It is billed as a security measure that prevents unauthorized access and protects user privacy; and yet, by activating it, users send even more private data to the vendor. Some companies have already been caught having harvested these phone numbers for marketing purposes. Unless the company has very strong data governance, the engineers and data teams accessing user data may not know that the phone numbers were obtained from two-factor authentication.

 

I'd let Ubiquiti have the last word. Here's the end of the notice:

We take the security of your information very seriously and appreciate your continued trust.

***

 

It's been a while since I wrote about data practices. I have a whole series of blog posts called "Know your data". You can still find them here.

 

 

One-dose vaccine strategy elevates PR over science

I fear that the U.K. policy of one-dose vaccines will backfire, and cause the pandemic to continue longer than necessary. Here are several reasons why.

Partial protection provides a convenient excuse for vaccinated people to do away with inconvenient mitigation measures

Assuming that the trial results hold, people receiving two vaccine shots have a very low chance of getting infected, and thus, it is acceptable that they return to normal living after vaccination. (I myself would not, since I think two doses are closer to 70 percent effective in reality, and we don't know that the vaccine stops asymptomatic spread, and so continuing to reduce contacts is advisable. Yet, under this scenario, one should not begrudge people who decide to return to normalcy.)

If people are given partial protection through just the first shot, it is irresponsible to return to normal living. Even the vaccinated must continue to wear masks, maintain social distancing, and stay at home as much as possible for the foreseeable future. Even now, it has proven impossible to enforce mitigation measures - at least not in the U.S. where I live. Vaccination presents a convenient justification for non-compliance.

This behavioral problem alone should have doomed the one-dose lobby.

One-dose treatment solves a problem that does not exist

The first weeks of vaccination has shown that the problem is demand not catching up to supply, not the other way round. We have more doses shipped than doses that are going into arms. If people don't like your cakes, you can't solve this problem by selling half-cakes!

The high efficacy of single doses is wishful thinking

Anyone who believes that the efficacy of the first dose is 90% lives in a fantasy world. Such a conclusion requires cherry-picking the best-case scenario, ignoring results that don't fit the theory, ignoring the margin of error, assuming that the second shot is essentially worthless, and assuming that the first shot is effective for the pertinent period of time without any boost from the second shot. The last two assumptions expose the one-dose strategy as a tautology - a conclusion that stems directly from one's assumption.

In a prior post, I explained how any guess of single-dose efficacy requires an assumption of (a) the efficacy of the second dose and (b) the duration of protection of the first dose, for neither of which do we have direct evidence coming from the vaccine trials.

It's a jigsaw puzzle, not a Barbie doll

People who propose the one-dose strategy are treating the clinical trial results like a Barbie doll. They are switching the green dress with the red skirt, or the high heels with flats. In fact, the toy is a jigsaw puzzle, in which each piece has its exact place, and cannot be haphazardly substituted.

For example, the headline vaccine efficacy numbers of 95% - recall the decimal obsession by Moderna - is based on a case definition in which they only count cases that occur at least 7 to 14 days after the second shot. You simply can't talk about that number in the same breath as efficacy for the first shot which requires counting cases after the first shot and before the second shot. They are not comparable numbers.

Use the scientific method properly

What supporters of the one-dose strategy should do is to lobby the FDA and/or pharmas to immediately start new clinical trials to test the one-dose strategy against the two-dose treatment.

From a scientific perspective, the biggest problem is post-hoc theorizing. Let's use stock market performance as an example. Do you feel like the business press are having it both ways? If stock prices went down yesterday, it's because the unemployment numbers were terrible. But if stock prices went up yesterday, it's because the terrible unemployment numbers were not as bad as imagined, or because businesses expected the job market to start improving, or because executives shrugged off the terrible unemployment numbers. They end up fitting a story to the observed data, as opposed to using the observed data to confirm a hypothesis.

Given the setup of the vaccine trials, before seeing these results, no one would have argued for one dose. The argument emerges after one sees the particular outcome from the trial. Trouble is, if the trial were run many times, would we have seen the same efficacy curve? (The variance across different trials already tells us the answer is NO.)

If the front part of the curve looks less favorable (say 30%), these same people would say we can't draw inferences about one dose (as opposed to one dose isn't going to work). In other words, this analysis contains an optimism bias, as does any theorizing after peering at the data.

***

Despite what I said above, I rate it as likely that certain countries will eventually follow the U.K. example and allow one-dose treatment. (As of now, the FDA has sensibly decided not to go down this path.)

Politicians score easy PR points by claiming they are helping "more people" when in reality, such a policy will lengthen the public-health crisis. Previously, I already noted that the PR agency of the UK government should be given an award.

 

P.S. Sadly, it appears that the incoming Biden administration is intent on elevating PR over science, as news has emerged that they will "release all available vaccines" and "not withhold second doses". This is euphemism for giving out one dose to everyone (as opposed to reserving the second doses for those who have gotten the first dose). For those "scientists" who are jumping behind this policy, I'd request that they answer the questions listed on my previous post. I can be persuaded that the one-dose strategy will lead to a better outcome but I'm having trouble understanding what set of assumptions lead to such a policy.

Added to the problems listed above, I think this latest news may persuade some of us to become vaccine skeptics. Those of us who believe the data show the vaccine needs two doses to work, and who do not get assurance that the second dose will be available in the right time frame, may decide to opt out of the vaccine.

 

P.P.S. In justifying this misguided policy, people are going to say "don't let the perfect be the enemy of the good". This is the cue that they are about to ignore the science. No one ever said the vaccine trial results are "perfect"; in fact, many of us believe they are less than perfect. I have written frequently about the shortcuts that were taken. We are not letting the perfect be the enemy of the good; we are opposing letting the unknown replace the good enough.

Other than some handwaving, I have not heard any description of what "the good" entails. If you can't quantify it, it's not science. If we are not going to argue about the science, then we must debate the assumptions, and evidence supporting such assumptions. Let's go!

Lab science and statistical science

Since new variants of the novel coronavirus have emerged, there is an expectation that we will soon learn whether the recently-approved vaccines will be effective against these new variants. The public has already been led astray, sometimes by scientists, about what constitutes evidence that these vaccines can stop the new variants.

With no proof, some scientists have already issued predictions that the vaccines would work. These arguments are purely on theoretical grounds. These scientists might think they are doing a public good but instead, these premature pronouncements hurt public confidence because they confirm people's suspicions that scientists are cutting corners.

We are starting to get some lab studies that are being touted as proof of efficacy. These lab studies are just the first step in answering the question but by no means do they provide a conclusion.

Think about this: if lab studies are sufficient to establish vaccine efficacy, then we would never have run a single clinical trial. We could have declared months ago based on lab studies that the Pfizer/Biontech, the Moderna, J&J, etc. all work. Clinical trials provide real-world conditions, real-world metrics, much larger sample sizes, and representative samples.

Lab studies are necessary but not sufficient to get to the answer. There should be follow-on clinical trials which embody statistical science to measure the efficacy. It is neither too hard nor too expensive to organize such trials - the vaccine is Big Business, imagine needing to give two shots to at least 75% of the world's population!

Separately, we will get some early signals from the U.K. as they roll out vaccinations. The case rate should start coming down massively if the Pfizer vaccine would cut the case rate by 95% (frequently claimed by scientists) despite the emergence of the new variant. Real-world evidence doesn't substitute clinical studies because if the case rate didn't fall drastically as expected, it is not clear that the new variant is the reason.

For those observing, we are seeing science in real time. Step 1 is the theory. Step 2 is the lab tests. Step 3 is clinical trial evidence. Step 4 is real-world impact.