Big Data, Plainly Spoken (aka Numbers Rule Your World)

U.S. regulators asleep at the wheel

My friend Mark Palko has been on Tesla's case for a while. He just commented on the latest fatal accident involving a Tesla vehicle. Amusingly, at this point in time, we are supposed to believe that a ghost crashed the car killing two passengers - as we're being told neither a human driver nor Tesla's software was directing the car.

Mark rightfully laid the blame on regulators. It's pretty clear that U.S. regulators put business profits above citizens' interests. We learned how Boeing was allowed to inspect themselves. The Europeans, not U.S. regulators, are reigning in the likes of Google and Facebook. Tesla's case is particularly telling.

Observation #1

U.S. regulators are allowing Tesla to lie about its "self-driving" technology. The "Auto-pilot" software does not auto-pilot - in fact, drivers are told to have their hands on the wheel at all times. The "full self-driving" mode is not what researchers in the field mean by full self-driving. Tesla customers are used as test subjects for an immature technology. The regulators should have stepped in to stop the false advertising but they have been absent.

Observation #2

After an accident happens, it appears that regulators rely on Tesla to report on itself. It seems that only Tesla has data on whether Auto-pilot or other features were turned on or off, whether the driver's hands were on the wheels, etc. Imagine the Chauvin trial, and the only video for the incident came from the police's bodycams. How do we know the footage hasn't been edited, and that the best perspectives were submitted?

Observation #3

The entire conceit of requiring the drivers to have their hands on the wheel at all times while using "Auto-pilot" not only makes a mockery of the term "auto-pilot" but also creates a liability-free zone for Tesla. If the driver's hands were determined not to be on wheel (and the only source of this information is Tesla), any accident is blamed on the driver. In past accidents, they even faulted drivers for momentarily not touching the wheel. If the driver had hands on the wheel, the accident is also not caused by Tesla because this accident could not have been prevented.

Observation #4

Some Tesla fans argue that everyone should know "Auto-pilot" is false advertising, and so any dead or injured Tesla customers have themselves to blame since they chose to use the feature. Using this logic, anyone who died from opioids deserves to die since they chose to ingest.

 

 

 

 

The cheerleaders of herd immunity

In epidemiology, herd immunity is the end of a pandemic. But epidemiological models do not provide a magic number of vaccinations beyond which the pandemic is certain to cease.

The messaging around herd immunity has gotten very sloppy lately.

On the one side, we have the CDC and epidemiologists screaming about vaccine skepticism, exhorting Americans to get vaccinated in search of herd immunity. On the other side, we have pharmas (Moderna and Pfizer) asserting that every vaccinated person will need booster shots, perhaps every year. If that is true, Covid-19 has become an endemic disease that does not go away: there is no herd immunity.

You either believe Covid-19 is not endemic, and will end through herd immunity, or you believe it is endemic, requiring seasonal vaccination. Not both.

***

In the anti-lockdown communities, a calculation has been circulating around that claims to track our progress toward herd immunity. This calculation takes the number of reported cases and the number of vaccinations, add them together and call that progress toward herd immunity.

Such an analysis has serious holes, and you should know about them.

First, it assumes that the second shot is useless because anyone who has at least one shot is counted toward herd immunity. If the second shot is required, plus the infamous 14-day waiting period after the second shot, then the calculation is much too optimistic.

Second, it assumes the level of vaccination required to achieve herd immunity does not depend on the efficacy of the vaccine. On the contrary, the lower the vaccine's efficacy, the more people must take it for the country to attain herd immunity.

Third, it assumes we know what the target level of immunization is. Herd immunity is an abstract construct dropping out of a mathematical model with lots of assumptions (e.g. R0 which is not directly measurable). You don't stop vaccinating people when some level is reached; you keep vaccinating more people until the cases disappear completely. At that time, the remaining people may not need to be immunized because the virus would not find enough hosts to keep spreading.

Fourth, it assumes that only uninfected people are getting vaccinated. The spreadsheet just adds up the two columns and double counts anyone who was infected in the past and received a shot.

Fifth, it assumes every reported case has ever-lasting immunity without getting vaccinated. Most infected people have some degree of immunity but we do not know how long natural immunity lasts.

Sixth, it assumes every positive test result is a true positive. People with false-positive results were not infected, and do not have immunity.

Seventh, it assumes people who got sick but recovered without getting tested (e.g. mild or asymptomatic cases) cannot have antibodies. Some of these people probably do have immunity. As pointed out by a reader, the calculation does deal with unreported cases - by inflating the case counts by a factor of 4. This is an assumption that only 25% of the infections are counted as cases. I have no idea how this factor can be estimated since the U.S. does not have a random testing program in place. Such an assumption makees assumption #4 above much worse. Many people are counted once as unreported cases and then again as vaccinated persons.

***

No matter how we slice it, at this time, vaccinations are a fraction of the number of confirmed cases in the U.S. So, much of the progress toward herd immunity has been achieved through natural infections. As I pointed out a year ago, the path to herd immunity is paved with body bags. If we believe the calculation, then the U.S. has sacrificed over half a million of our fellow citizens in the name of herd immunity. Is this the lesson of the Covid-19 pandemic? In the next pandemic, will we sacrifice another half million, one million, or more of our fellow citizens?

***

Here are a few other aspects of herd immunity that you're not hearing from the media.

Herd immunity is not an inevitable outcome of the mathematical model. It is a possible outcome if certain conditions are met. The vaccines must be effective enough, the proportion of people taking the vaccine must be high enough while the virus must not be too infectious.

Herd immunity if achieved at a level of vaccination close to 100% is practically meaningless.

It's also time to direct your attention back to the old post, in which I argue that vaccination is not about protecting others (i.e. herd immunity) -- it's about protecting yourself. Assume 80% must take the shot to get herd immunity. Those 80% protect themselves because their cells are producing antibodies against the coronavirus. The other 20% are not infected because the mathematical model predicts that they will not be exposed to the virus, but if they do get in contact with the virus, they will get infected because they don't have antibodies.

The messaging on vaccinations is confusing and misguided. Vaccinations first and foremost give one protection. Herd immunity is a by-product, far less important.

 

What to think about vaccines today, including why vaccines protect you against blood clots

Georgette's comment to my other post prompted me to write a new post on current Covid-19 topics.

Vaccines protect you against blood clots

They didn't say it out loud but they imply it. Keep reading...

The FDA's pause on the Johnson & Johnson vaccine is a significant event. The situation parallels the controversy over many European countries halting the distribution of the Astrazeneca vaccine. My take is the same as before. The only unknown is whether the FDA will issue an additional warning about rare blood clots that affect certain subgroups of people. They are likely to keep recommending the vaccine to most of us.

The circumstantial evidence related to blood clots is accumulating. Both J&J and Astrazeneca vaccines use the adenovirus technology. These blood clots are happening mostly to young women, and are paired with other conditions (low platelets), which makes natural cause being rather unlikely. The diagnoses and demographics of the affected people are similar across both vaccines, and, though rare, the same condition has popped up in numerous countries.

The cost-benefit analysis for the young female demographic is completely different from the average person. The benefit of the vaccine is much lower, given the tiny mortality rate from Covid-19 (not counting the benefit accrued to others). Meanwhile, the cost of serious illness and death from rare blood clots is relatively higher because the baseline risk of this condition is minuscule for that age group. A preliminary estimate by the European Medicines Agency (EMA) found that for the younger population, the reported cases of those rare conditions were 5 - 12 times higher than expected.

Doctors are justifiably alarmed, given the "do no harm" principle. Side effects have always been the challenge of developing vaccines. Injecting otherwise healthy people with a substance and causing harm is not accepted, except in a pandemic. The risk to most people is still low but it doesn't then follow that the risk is low to all.

The statistic of the moment says the overall incidence of blood clots is far, far lower than expected among those who were vaccinated. Therefore, they say, the vaccine does not cause blood clots. The gaping hole in this argument is that the data present strong evidence that these vaccines protect people against blood clots! If the vaccines have nothing to do with blood clots, then we expect similar incidence among the vaccinated as among the unvaccinated, and both rates being similar to the expected value. Instead, they tell us blood clots are far less common than expected among those who got the Astrazeneca shots.

I support the FDA's pause decision. It builds confidence that the agency is keeping an eye on safety as there are plenty of examples of medicines that have side effects that are not found during a clinical trial. I hope they look beyond case counts, and review other information as well. Recall my post on the law of small numbers. 

Fully vaccinated people does not have a 5% chance of catching Covid19

Your chance of catching Covid-19 after getting fully vaccinated is not 5%. It cannot be that high. If you review the data from all the vaccine trials, the typical case rate on the placebo arms is 2-3 percent. That means, of those participants who were given placebo shots during the trials, 97-98 out of 100 did not get symptomatic, confirmed Covid-19. If the chance of getting infected after 2 vaccine shots was 5%, then the vaccine would have negative efficacy (worse than placebo)!

The source of this confusion reflects a misunderstanding of a vaccine trial. Vaccine trials are different from the usual clinical trials of drug treatments. In a cancer drug trial, every participant has cancer. In a vaccine trial, we don't know who has been exposed to the coronavirus. In a short trial, with only half the participants reaching 8 weeks of observation, most of the uninfected probably have not been exposed yet.

Giving a probability of getting infected without stating a time window is also useless. If the 5% number came from the trial results, then the right way to describe it is 5% within two months (which, if you think about it, is disturbingly high).

Vaccines and the cargo cult

There is a cargo cult developing around vaccine science. Vaccines have become the single and only explanation for positive trends while not enough vaccines is the single and only explanation for negative trends.

For example, the U.K. and Israeli governments credit their fast vaccine rollouts as the reason for plunging cases. To do proper science, we must take into account all other factors, and in particular, the impact of national lockdowns - which coincided with the vaccination campaign - and the reversal of the holiday spike.

The vaccines-only narrative lends support to lockdown skepticism, mask skepticism, and social-distancing skepticism. Those brands of skepticism also weaken our national fight against the coronavirus.

All vaccines are the same

Vaccines are not all the same. To believe that a VE 60% vaccine is the same as a VE 90% vaccine is to believe that the scale for measuring VE is severely flawed. To believe that vaccines using different technologies and formulations are all the same is to believe that formulations and technologies are window-dressing. Even if all vaccines were identical, people still have preferences: some go for a black car, others a red car - even when it's the same car.

 

 

In science, truth matters

When I turn on the media and listen to the media hyperventilating about Covid, it's no different from turning on CNBC, listening to Jim Cramer etc. lecturing us on why the stock market went up or down yesterday. If cases went down, it's because of the vaccine. If cases went up, it's the variants. We can write down all the usual suspects: vaccine, lockdowns, variants, holidays, travel, people behaving badly, masks, schools, treatments, parties, dining outdoors, dining indoors, etc. Depending on what happened yesterday, the host picks one from the box. Medical science reporting has descended to the level of stock price watching.

So, I was immediately attracted to Andrew Gelman's recent account of science as "a harbor clotted with sunken vessels," in which he lamented the current state of scientific publications that elevates headline-grabbers, failing to self-correct when such studies subsequently prove to be bogus.

The problem is much broader, and it's about to hit medical research in a huge way, once the pandemic is in the rear-view mirror. The emergency situation has given cover to an ocean of pre-prints and warp-sped refereeing, in which headline-grabbers have floated to the top, without a clear path for future researchers to navigate through this sea of debris. Many - if not most - of these studies will eventually be debunked.

***

In his post, Gelman cited the example of research on the effect of neuropeptide oxytocin (OT) on trust between humans. A seminal paper published in 2005 claimed to have found an association between the two. The original experiment failed to replicate, and by now, it should be clear that the association has not been proven. And yet, Prof. Gelman found that (a) the citations of the original paper are still multiple times larger than those of the replication studies; and (b) the original paper continues to get new citations. He just did a casual Google search so we don't know if those who cited the original research also cited the refutations - I doubt it.

If science advances by cleansing bad ideas as better ones emerge, this situation is not putting it in a good light. This situation is due to a host of reasons. One can be confirmation bias - the researchers are looking for anything that supports an assumption of theirs, so they can move on with the rest of their study. Another reason is professional misconduct, or socialized behavior of a subculture.

For me, the problem is also rooted in the practice of scientific publication. It's too damn hard to find those follow-up studies! The following is a diagram of a scientific paper which links to past papers through the device of "references".

By definition, a reference is a link to the past. So a replication study, or for that matter, any kind of follow-on work, will not be found in the references, nor in the original paper. In this system, it's a chore to discover future papers that reference a given paper. Modern search engines attempt to solve this problem by the device of "citations". From the perspective of the cited paper, a citation is a link to a future paper that lists it in the references.

Unfortunately, the citation device is extremely noisy. Most of the papers that cite the original paper do not evaluate the merits of the paper. Citations are tracked as a means of measuring the impact of a given paper - and this has led to gaming. Citations are given to appease referees, or to elevate colleagues, or any number of reasons.

Buried deep inside a long list of citations may be a few replication or follow-up studies. These are future studies that directly address the validity of the original paper.

Adding to the mess, one expects follow-up studies to be published long after the original publication. Citations that are not follow-up studies are both more numerous and also sooner to arrive than follow-up studies.

Researchers may then rely on meta-analyses or survey studies. If they exist, these studies are easy to locate. But they are published far, far, far into the future relative to the original study. They also require the existence of a prior body of follow-up or replication studies.

The end result is that follow-up studies take a lot of effort to find.

Nevertheless, if there is a will, there is a way. Follow-up studies can be marked or registered. And they could be automatically attached by search engines to the original paper, just like citations, but as a separate tag.

Why should we care? Because in science, truth matters.

***

This timing problem affects other fields as well. I published a letter to the editor once about a New York Times Magazine article. But if you read that original article, you're not going to find my letter. There are no links to the future, only to the past.